[
https://issues.apache.org/jira/browse/TEZ-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14986084#comment-14986084
]
Jason Lowe commented on TEZ-2918:
---------------------------------
Thanks for the patch, Bikas!
I agree that using AtomicBoolean.lazySet is probably a better way to set the
flag than just using a volatile. That appears to be so cheap that we shouldn't
be sweating it until we actually have a real-world benchmark showing a problem.
I think we may need a progress call in ConcatenatedMergedKeyValueReader when we
move between sub-readers. I realize that it is simply a collection of other
readers and is relying on the {{next}} method of those readers to post progress
on its behalf. However there's a case where this can fail to do so in a timely
manner. In MapReduce we ran into some corner cases where a conglomerate input
like this ended up with tons of empty sub-readers. Those sub-readers never
posted progress since they never had any inputs. The top-level reader simply
iterated through each one, finding they had no inputs, and kept moving to the
next one but reporting no progress before it was too late. See MAPREDUCE-5670.
The same comment applies to ConcatenatedMergedKeyValuesReader and also
potentially OrderedGroupedMergedKeyValuesReader
Do we need to report progress in OrderedGroupedKeyValuesReader?
> Make progress notifications in IOs
> ----------------------------------
>
> Key: TEZ-2918
> URL: https://issues.apache.org/jira/browse/TEZ-2918
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2918.1.patch, TEZ-2918.2.patch,
> jmh-microbenchmark.tar.gz
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
