jarredhj0214 commented on code in PR #13550:
URL: https://github.com/apache/gravitino/pull/13550#discussion_r4225895651
##########
catalogs/hive-metastore3-libs/src/main/java/org/apache/gravitino/hive/client/hive3/HiveShimV3.java:
##########
@@ -158,6 +159,13 @@ public HiveTable getTable(String catalogName, String
databaseName, String tableN
if (TableType.VIRTUAL_VIEW.name().equalsIgnoreCase(tb.getTableType())) {
return HiveTableConverter.fromHiveTable(tb);
}
+ if (hasDerivedColumnTypes(tb)) {
+ List<FieldSchema> resolvedColumns =
+ invoke(
+ ExceptionTarget.table(tableName),
+ () -> client.getFields(catalogName, databaseName, tableName));
Review Comment:
Thanks for the detailed HMS analysis. Implemented option 1 in a6c718eaf:
both shims now report actionable guidance for getFields MetaException failures,
including the HMS SerDe classpath and
metastore.storage.schema.reader.impl=org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader
followed by an HMS restart. The Hive catalog docs and PR description explain
that this is a server setting, including Hive 2 clients connecting to Hive 3
HMS. The test container config enables the reader. No client-side CSV or
generic type fallback is introduced.
##########
catalogs/hive-metastore-common/src/main/java/org/apache/gravitino/hive/client/HiveShim.java:
##########
@@ -334,4 +342,37 @@ protected EnvironmentContext doNotUpdateStatsContext() {
return new EnvironmentContext(
Collections.singletonMap(StatsSetupConst.DO_NOT_UPDATE_STATS,
StatsSetupConst.TRUE));
}
+
+ /**
+ * Returns whether a table contains column types that Hive expects its SerDe
to resolve.
+ *
+ * @param table The Hive metastore table.
+ * @return {@code true} if at least one storage column has a derived type
marker.
+ */
+ protected static boolean hasDerivedColumnTypes(Table table) {
+ return table.getSd() != null
+ && table.getSd().getCols() != null
+ && table.getSd().getCols().stream()
+ .anyMatch(field -> TYPE_FROM_DESERIALIZER.equals(field.getType()));
+ }
+
+ /**
+ * Replaces storage columns containing Hive's derived type marker with
columns resolved by the
+ * table SerDe.
+ *
+ * @param table The Hive metastore table to update in memory.
+ * @param resolvedColumns The storage columns resolved by the Hive metastore.
+ */
+ protected static void replaceDerivedColumns(Table table, List<FieldSchema>
resolvedColumns) {
+ if (resolvedColumns == null
+ || resolvedColumns.isEmpty()
+ || resolvedColumns.stream()
+ .anyMatch(field ->
TYPE_FROM_DESERIALIZER.equals(field.getType()))) {
+ throw new IllegalArgumentException(
+ String.format(
+ "Hive Metastore did not resolve SerDe-derived column types for
table %s.%s",
Review Comment:
Updated in a6c718eaf. An unresolved schema and a getFields MetaException now
use the same guidance: ensure the SerDe is on the HMS classpath and configure
SerDeStorageSchemaReader on Hive 3 HMS, then restart it. Added tests for both
shims and for null/empty/unresolved results; missing-table exceptions retain
their existing handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
