pvary commented on a change in pull request #2228:
URL: https://github.com/apache/iceberg/pull/2228#discussion_r573145733
##########
File path:
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java
##########
@@ -90,22 +93,34 @@ public void commitTask(TaskAttemptContext originalContext)
throws IOException {
TaskAttemptContext context =
TezUtil.enrichContextWithAttemptWrapper(originalContext);
TaskAttemptID attemptID = context.getTaskAttemptID();
- String fileForCommitLocation =
generateFileForCommitLocation(context.getJobConf(),
- attemptID.getJobID(), attemptID.getTaskID().getId());
- HiveIcebergRecordWriter writer =
HiveIcebergRecordWriter.getWriter(attemptID);
+ JobConf jobConf = context.getJobConf();
+ Map<String, HiveIcebergRecordWriter> writers =
HiveIcebergRecordWriter.getWriters(attemptID);
+ Map<String, String> outputs =
SerializationUtil.deserializeFromBase64(jobConf.get(InputFormatConfig.OUTPUT_TABLES));
- DataFile[] closedFiles;
- if (writer != null) {
- closedFiles = writer.dataFiles();
- } else {
- closedFiles = new DataFile[0];
+ ExecutorService tableExecutor = tableExecutor(jobConf, outputs.size());
+ try {
+ // Generates commit files for the target tables in parallel
+ Tasks.foreach(outputs.entrySet())
+ .stopOnFailure()
Review comment:
I do not see the enough added value for the complexity:
- Here: Temporary outage of the FS could be mitigated - but we should handle
if the file already exists
- `commitJob`: Reading files are already done with retries - but we should
handle if we already committed the changes to the table
Do I miss something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
