[
https://issues.apache.org/jira/browse/TEZ-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kuhu Shukla updated TEZ-3317:
-----------------------------
Attachment: TEZ-3317.007.patch
Thanks a lot [~rajesh.balamohan] for the review!
1. Removed stackTrace and added warning log line.
2. Removed shuffledBytes, but still need updated shuffledInputs for updated
progress. Thoughts?
3. Yes, 50% of the progress depends on shuffledInputs, rest on the iterator's
progress. Please share any thoughts you might have on this.
4. Changed interval to 100 ms. Should it be larger?
Additionally, made changes to MapProcessor and ReduceProcessor analogous to
other processors and not update progress for every KV Read.
I modified PigProcessor similar to SimpleProcessor it does update progress as
intended. You are right that this change won't get picked up by Hive upfront.
We will need to change TezHiveProcessor on Hive side.
> Speculative execution starts too early due to 0 progress
> --------------------------------------------------------
>
> Key: TEZ-3317
> URL: https://issues.apache.org/jira/browse/TEZ-3317
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Jonathan Eagles
> Assignee: Kuhu Shukla
> Attachments: TEZ-3317.001.patch, TEZ-3317.002.patch,
> TEZ-3317.003.patch, TEZ-3317.004.patch, TEZ-3317.005.patch,
> TEZ-3317.006.patch, TEZ-3317.007.patch
>
>
> Don't know at this point if this is a tez or a PigProcessor issue. There is
> some setProgress chain that is keeping task progress from being correctly
> reported. Task status is always zero, so as soon as the first task finishes,
> tasks up to the speculation limit are always launched.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
