zhangbutao commented on code in PR #4645:
URL: https://github.com/apache/hive/pull/4645#discussion_r1314995394
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java:
##########
@@ -201,6 +200,15 @@ public List<InputSplit> getSplits(JobContext context) {
.ofNullable(HiveIcebergStorageHandler.table(conf,
conf.get(InputFormatConfig.TABLE_IDENTIFIER)))
.orElseGet(() -> Catalogs.loadTable(conf));
+ if (conf.get(InputFormatConfig.TABLE_IDENTIFIER) == null) {
+ conf.set(InputFormatConfig.TABLE_IDENTIFIER, table.name());
+ }
+
+ if (conf.get(InputFormatConfig.SERIALIZED_TABLE_PREFIX +
conf.get(InputFormatConfig.TABLE_IDENTIFIER)) == null) {
+ conf.set(InputFormatConfig.SERIALIZED_TABLE_PREFIX +
conf.get(InputFormatConfig.TABLE_IDENTIFIER),
Review Comment:
IIUC, every split has a `conf `and each `conf `would contain a srialized
iceberg table. I want to know if the srialized iceberg table would make `conf
`bigger? and also would cause AM oom in case of many splits?
Did we have a benchmark test to verify the validation of this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
