ramitg254 commented on code in PR #6413:
URL: https://github.com/apache/hive/pull/6413#discussion_r3450030640
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java:
##########
@@ -110,6 +113,10 @@ public class Table implements Serializable {
/**
* These fields are all cached fields. The information comes from tTable.
*/
+ private List<FieldSchema> tablePartCols;
+ private record TableColumn(int index, FieldSchema field) {}
+ private transient Map<String, TableColumn> columnsByName;
+ private transient List<FieldSchema> tableNonPartCols;
Review Comment:
tablePartCols was intentionally made non-transient as if we make it
transient then we were running into Exception in TezAM while loading iceberg
table on the first call to getPartCols() after serialization as it will call
getStorageHandler.getPartitionKeys due to incorrect conf:
eg.
```
org.apache.hadoop.hive.ql.exec.tez.TezRuntimeException: Vertex failed,
vertexName=Map 1, vertexId=vertex_1782075974293_0001_16_00, diagnostics=[Vertex
vertex_1782075974293_0001_16_00 [Map 1] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tbl_orc initializer failed,
vertex=vertex_1782075974293_0001_16_00 [Map 1],
org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive
Metastore
at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:67)
at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:36)
at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:143)
```
```
Caused by: MetaException(message:Got exception:
org.apache.hadoop.hive.metastore.api.MetaException java.net.URISyntaxException:
Relative path in absolute URI:
hdfs://localhost:36709./$%7Btest.warehouse.dir%7D)
at
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.throwMetaException(MetaStoreUtils.java:191)
```
so need to keep it as non-transient so it would be only cached during
compile phase and be reused even after serialization and deserialization
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
