pvary commented on a change in pull request #1407:
URL: https://github.com/apache/iceberg/pull/1407#discussion_r489312917
##########
File path:
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java
##########
@@ -271,28 +271,25 @@ public void setupTask(TaskAttemptContext
taskAttemptContext) {
}
@Override
- public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) {
- return true;
+ public boolean needsTaskCommit(TaskAttemptContext context) {
+ // We need to commit if this is the last phase of a MapReduce process
+ return
TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskID().getTaskType()) ||
+ context.getJobConf().getNumReduceTasks() == 0;
Review comment:
@massdosage, @rdblue: Tested out the stuff on an MR Cluster by running
the following queries successfully:
```
insert into test_table values(3,"3"),(4,"4");
insert into test_table select * from test_table;
insert into test_table select * from test_table order by id;
insert into test_table select * from test_table_2;
```
Had Map only jobs and Map-Reduce jobs too. The result was checked by Hive
select queries and everything seems ok.
So if Adrian's team did not find any issues I would think that this PR is
ready to be pushed.
Your thoughts?
Thanks, Peter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
