[
https://issues.apache.org/jira/browse/PIG-3356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13684627#comment-13684627
]
xplenty commented on PIG-3356:
------------------------------
A solution to this issue would be adding a call to:
PigStatusReporter.getInstance().progress();
for each file opened, for instance in the PigRecordReader in nextKeyValue().
this way Hadoop knows the pig job is still running.
> Timeout when processing large amounts of empty files.
> -----------------------------------------------------
>
> Key: PIG-3356
> URL: https://issues.apache.org/jira/browse/PIG-3356
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.11.1
> Reporter: xplenty
> Priority: Minor
> Labels: hadoop, progress, timeout,
>
> When processing large amounts of empty files (for example traversing empty
> files on S3) takes a long time and Pig tasks don't report progress and end up
> being killed for being unresponsive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
