massdosage commented on a change in pull request #1407:
URL: https://github.com/apache/iceberg/pull/1407#discussion_r488065180
##########
File path:
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java
##########
@@ -271,28 +271,25 @@ public void setupTask(TaskAttemptContext
taskAttemptContext) {
}
@Override
- public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) {
- return true;
+ public boolean needsTaskCommit(TaskAttemptContext context) {
+ // We need to commit if this is the last phase of a MapReduce process
+ return
TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskID().getTaskType()) ||
+ context.getJobConf().getNumReduceTasks() == 0;
Review comment:
@pvary I guess it would look very similar to one of the existing
HiveRunner tests
[here](https://github.com/apache/iceberg/blob/master/mr/src/test/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandlerBaseTest.java)
but one could do a hiveShell.execute() with an insert, update, etc statement
and then do a query to select the results back?
I'm not sure whether these examples
(https://github.com/klarna/HiveRunner/blob/master/src/test/java/com/klarna/hiverunner/InsertIntoTableIntegrationTest.java
and
https://github.com/klarna/HiveRunner/blob/master/src/test/java/com/klarna/hiverunner/data/InsertIntoTableTest.java)
from HiveRunner itself would trigger the OutputFormat for the insert
operations? Presumably so if the table under test created is created using the
StorageHandler?
I'd also highly recommend testing this in distributed mode, we found lots of
issues with the InputFormat that only manifested themselves then. I can help
with this if you want as we have a test environment for this setup already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
