[
https://issues.apache.org/jira/browse/TEZ-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739179#comment-14739179
]
Siddharth Seth commented on TEZ-2775:
-------------------------------------
bq. Fetcher/FetcherOrderedGrouped - should we retain the decomp, compressed
size, type information?. That has helped in couple of scenarios when debugging.
Is it sufficient if this information is available on failure ? I believe the
information is already logged for a successful fetch.
bq. MRInput/MROutput/MultiMRInput - space needed in "newmapreduce" or is that
intentional?
Intentional, but can be changed.
bq. in flush(), indexcache empty check is due to TEZ-2440. This is completely
harmeless and can be removed. If we do not have this check, and if the task
gets killed in the middle, it can throw NPE leading to distraction when
debugging.
This should be documented - with the details you provided.
bq. numObsoletionEvenets - spelling?
Will fix.
bq. If there are only 10 or 30 events, it might only print the first time?.
Would having a method like getProgress() or something like that in
ShuffleEventHandler, where this statement can be logged to provide more
accurate number?. This can be periodically called as well as from
UnorderedKVInput.close().
We don't know how many events to expect upfront. The log can be augmented to
say something like - "next log after X events". In terms of a getProgress
method - at what point will that get called, and how often should it log ?
bq. logProgress(); //can be used for mining later. Would printing it only for
50 times might make it hard to interpret?
This is always logged after all fetches complete; is that not sufficient ? In
terms of mining at a later point, we may want to think of an alternate
mechanism like metrics, rather than having this information in the logs.
bq. ShuffleUtils:logIndividualFetchComplete() - Changes can break the perf
analysis tool for different versions?
It should be possible to change the perf analysis tool to handle to handle
different versions of the log line ? This is far shorter than the original log
line, and can actually be made shorter.
On the rest of the comments, more a little later.
> Reduce logging in runtime components
> ------------------------------------
>
> Key: TEZ-2775
> URL: https://issues.apache.org/jira/browse/TEZ-2775
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-2775.1.txt
>
>
> Specifically Shuffle, which logs a lot for each event being processed and
> data being fetched.
> Also PipelinedShuffle is fairly noisy - some of the information from here
> could be consolidated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
