deniskuzZ commented on code in PR #4575:
URL: https://github.com/apache/hive/pull/4575#discussion_r1294255273
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java:
##########
@@ -405,45 +380,63 @@ private Set<FileStatus> listForCommits(JobConf jobConf,
String jobLocation) thro
* @param executor The executor used to read the forCommit files
* @param outputTable The table used for loading from the catalog
*/
- private void commitTable(FileIO io, ExecutorService executor, OutputTable
outputTable) {
+ private void commitTable(FileIO io, ExecutorService executor, OutputTable
outputTable,
+ List<JobContext> jobContextList) {
String name = outputTable.tableName;
- JobContext jobContext = outputTable.jobContext;
- JobConf conf = jobContext.getJobConf();
Properties catalogProperties = new Properties();
catalogProperties.put(Catalogs.NAME, name);
catalogProperties.put(Catalogs.LOCATION, outputTable.table.location());
if (outputTable.catalogName != null) {
catalogProperties.put(InputFormatConfig.CATALOG_NAME,
outputTable.catalogName);
}
- Table table = Catalogs.loadTable(conf, catalogProperties);
+ Collection<DataFile> dataFiles = Lists.newArrayList();
+ Collection<DeleteFile> deleteFiles = Lists.newArrayList();
+
+ Table table = null;
+ String branchName = null;
+ boolean isOverWrite = false;
+
+ for (JobContext jobContext : jobContextList) {
+ JobConf conf = jobContext.getJobConf();
+ table = Optional.ofNullable(table).orElse(Catalogs.loadTable(conf,
catalogProperties));
+
+ branchName = conf.get(InputFormatConfig.OUTPUT_TABLE_SNAPSHOT_REF);
+ isOverWrite = conf.getBoolean(InputFormatConfig.IS_OVERWRITE, false);
Review Comment:
list of job context is grouped by table name to handle multi-insert
statements after the split update/merge. so it should be the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
