[
https://issues.apache.org/jira/browse/TEZ-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14988974#comment-14988974
]
Bikas Saha commented on TEZ-2918:
---------------------------------
The conf read has been considered. That code path will be rarely hit only for
stuck tasks. I am not concerned about it vs creating another memory var for
every TA in every job.
I dont think we need an option to disable progress reporting since we want this
to be enabled always. There is no point adding code to make it optional which
will be removed shortly. The option in the AM is not to disable notifications
but to disable acting on them. This is mainly for backwards compatibility with
existing jobs where processors dont report progress and take a long time - e.g.
badly configured map joins in hive that spill to disk or skewed joins that take
a long time to process. In these cases, enabling progress detection can fail
existing jobs. Once Hive/Pig and other processors implement their own progress
notification then this would be on by default.
[~rajesh.balamohan] posted some numbers on the absolute effect
boolean.lazySet() and they were in the ballpark of 1-2 nanoseconds per
operation. The methods to notify progress are all final and should hopefully be
inlined by JIT. Thus the penalty in theory should be 1-2ns per progress
notification. That seems negligible. In many cases, there are other things like
counters increments happening on those code paths anyways.
I have been looking on ways to measure real life perf of the built-in IOs. Can
you please point me to any existing perf measuring experiments/code that have
been used to measure the perf of these components for previous code changes?
> Make progress notifications in IOs
> ----------------------------------
>
> Key: TEZ-2918
> URL: https://issues.apache.org/jira/browse/TEZ-2918
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-2918.1.patch, TEZ-2918.2.patch, TEZ-2918.3.patch,
> jmh-microbenchmark.tar.gz
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
