[
https://issues.apache.org/jira/browse/PIG-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546391
]
Benjamin Reed commented on PIG-14:
----------------------------------
+1 code looks good. My only suggestion is that you comment numNotifies to
indicate that it is only used by the unit tests. (It would be nice if we could
test without that variable.)
> large key cause pig reduce jobs to die
> --------------------------------------
>
> Key: PIG-14
> URL: https://issues.apache.org/jira/browse/PIG-14
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Olga Natkovich
> Assignee: Olga Natkovich
> Attachments: heartbeat.patch
>
>
> The reducer sends a heartbeat to the task tracker every time it starts
> processing new key. The task tracker expects to
> get a message every 10 minutes. If processing of an individual key takes
> longer, which could be the case for your job,
> the task tracker would not get a heartbeat in time and would kill the task.
> The current patch is to add <property>
> <name>mapred.task.timeout</name>
> <value>0</value>
> <description>timeout value</description>
> </property>
> to the cluster's hadoop-site.xml. This results in disabling heartbeat
> functionality which might not be what we want
> long term.
> A more flexible approach is to periodically report from map and reduce job via
> http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/Reporter.html#setStatus(java.lang.String)
> As a workaround for a UDF, call: PigMapReduce.reporter.progress() every
> 1000th time
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
