[GitHub] [iceberg] massdosage commented on a change in pull request #1407: Hive: HiveIcebergOutputFormat first implementation for handling Hive inserts into unpartitioned Iceberg tables - WIP

[GitBox]

massdosage commented on a change in pull request #1407:
URL: https://github.com/apache/iceberg/pull/1407#discussion_r487185451



##########
File path: 
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java
##########
@@ -271,28 +271,25 @@ public void setupTask(TaskAttemptContext 
taskAttemptContext) {
     }
 
     @Override
-    public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) {
-      return true;
+    public boolean needsTaskCommit(TaskAttemptContext context) {
+      // We need to commit if this is the last phase of a MapReduce process
+      return 
TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskID().getTaskType()) ||
+          context.getJobConf().getNumReduceTasks() == 0;

Review comment:
       Some HiveRunner tests that do writes that involves a MR job might help 
too?

##########
File path: 
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java
##########
@@ -271,28 +271,25 @@ public void setupTask(TaskAttemptContext 
taskAttemptContext) {
     }
 
     @Override
-    public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) {
-      return true;
+    public boolean needsTaskCommit(TaskAttemptContext context) {
+      // We need to commit if this is the last phase of a MapReduce process
+      return 
TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskID().getTaskType()) ||
+          context.getJobConf().getNumReduceTasks() == 0;

Review comment:
       Some HiveRunner tests that do writes that involves a MR job might help 
too?

##########
File path: 
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java
##########
@@ -271,28 +271,25 @@ public void setupTask(TaskAttemptContext 
taskAttemptContext) {
     }
 
     @Override
-    public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) {
-      return true;
+    public boolean needsTaskCommit(TaskAttemptContext context) {
+      // We need to commit if this is the last phase of a MapReduce process
+      return 
TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskID().getTaskType()) ||
+          context.getJobConf().getNumReduceTasks() == 0;

Review comment:
       Some HiveRunner tests that do writes that involves a MR job might help 
too?

##########
File path: 
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java
##########
@@ -271,28 +271,25 @@ public void setupTask(TaskAttemptContext 
taskAttemptContext) {
     }
 
     @Override
-    public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) {
-      return true;
+    public boolean needsTaskCommit(TaskAttemptContext context) {
+      // We need to commit if this is the last phase of a MapReduce process
+      return 
TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskID().getTaskType()) ||
+          context.getJobConf().getNumReduceTasks() == 0;

Review comment:
       Some HiveRunner tests that do writes that involves a MR job might help 
too?

##########
File path: 
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java
##########
@@ -271,28 +271,25 @@ public void setupTask(TaskAttemptContext 
taskAttemptContext) {
     }
 
     @Override
-    public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) {
-      return true;
+    public boolean needsTaskCommit(TaskAttemptContext context) {
+      // We need to commit if this is the last phase of a MapReduce process
+      return 
TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskID().getTaskType()) ||
+          context.getJobConf().getNumReduceTasks() == 0;

Review comment:
       Some HiveRunner tests that do writes that involves a MR job might help 
too?

##########
File path: 
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java
##########
@@ -271,28 +271,25 @@ public void setupTask(TaskAttemptContext 
taskAttemptContext) {
     }
 
     @Override
-    public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) {
-      return true;
+    public boolean needsTaskCommit(TaskAttemptContext context) {
+      // We need to commit if this is the last phase of a MapReduce process
+      return 
TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskID().getTaskType()) ||
+          context.getJobConf().getNumReduceTasks() == 0;

Review comment:
       Some HiveRunner tests that do writes that involves a MR job might help 
too?

##########
File path: 
mr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputFormat.java
##########
@@ -271,28 +271,25 @@ public void setupTask(TaskAttemptContext 
taskAttemptContext) {
     }
 
     @Override
-    public boolean needsTaskCommit(TaskAttemptContext taskAttemptContext) {
-      return true;
+    public boolean needsTaskCommit(TaskAttemptContext context) {
+      // We need to commit if this is the last phase of a MapReduce process
+      return 
TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskID().getTaskType()) ||
+          context.getJobConf().getNumReduceTasks() == 0;

Review comment:
       Some HiveRunner tests that do writes that involves a MR job might help 
too?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org