marton-bod commented on a change in pull request #2161:
URL: https://github.com/apache/hive/pull/2161#discussion_r609854017
##########
File path:
iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergOutputCommitter.java
##########
@@ -224,6 +224,11 @@ private void cleanup(JobContext jobContext) throws
IOException {
// If there are reducers, then every reducer will generate a result file.
// If this is a map only task, then every mapper will generate a result
file.
int expectedFiles = conf.getNumReduceTasks() > 0 ?
conf.getNumReduceTasks() : conf.getNumMapTasks();
+ // For Tez, we can only reliably get the number of tasks from the Tez AM,
not the job conf
Review comment:
My initial approach was to set the numReduceTasks to the correct value
in the `commitInsertTable`, but I didn't want to mess up MR, although since
it's unsupported in upstream Hive we could do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
