[
https://issues.apache.org/jira/browse/HIVE-1403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ning Zhang updated HIVE-1403:
-----------------------------
Attachment: HIVE-1403.patch
this patch report progress periodically during closing all the files.
> Reporting progress to JT during closing files in FileSinkOperator
> -----------------------------------------------------------------
>
> Key: HIVE-1403
> URL: https://issues.apache.org/jira/browse/HIVE-1403
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Ning Zhang
> Assignee: Ning Zhang
> Attachments: HIVE-1403.patch
>
>
> If there are too many files need to be closed in FileSinkOperator (e.g., if
> DynamicPartition/FileSpray is turned on), there could be many files generated
> by each task and they need to be closed at the FileSinkOperator.closeOp(). If
> the NN is overloaded each file close could take more than 1 sec. This
> sometimes make JT think the task is dead since it takes too long to close all
> the files and without any progress report. We need to report progress after a
> while during file closing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
