[
https://issues.apache.org/jira/browse/TEZ-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388454#comment-15388454
]
Hitesh Shah commented on TEZ-3359:
----------------------------------
Comments:
- ConcurrentHashMap instead of HashMap<TezDAGID, HistoryLogLevel>?
{code}
HistoryEvent historyEvent = event.getHistoryEvent();
142 if (historyEvent.getEventType() == HistoryEventType.DAG_SUBMITTED) {
143 dagLogLevel = ((DAGSubmittedEvent)historyEvent).getConf().getEnum(
144 TezConfiguration.TEZ_HISTORY_LOGLEVEL, amHistoryLogLevel);
145 dagIdToLogLevel.put(dagId, dagLogLevel);
146 } else if (historyEvent.getEventType() ==
HistoryEventType.DAG_RECOVERED) {
147 if (context.getCurrentDAG() != null) {
148 dagLogLevel = context.getCurrentDAG().getConf().getEnum(
149 TezConfiguration.TEZ_HISTORY_LOGLEVEL, amHistoryLogLevel);
150 dagIdToLogLevel.put(dagId, dagLogLevel);
151 }
{code}
- should we change this to a bit more robust code i.e. allow for case
insensitive matches. log a warning if there is an invalid which does not match
any valid enum or maybe even error out on the client side if either the AM or
DAG config setting is invalid? A client side error might be better instead of
silently falling back to logging everything which might prove dangerous.
Patch looks mostly good.
Minor nit on the tests: I dont believe we are testing a single session with
mulitple dags where the AM log level and the 2 dags are all using different log
levels. Possibly one of the tests could be enhanced to cover this complex case?
> Add granular log levels for HistoryLoggingService.
> --------------------------------------------------
>
> Key: TEZ-3359
> URL: https://issues.apache.org/jira/browse/TEZ-3359
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Harish Jaiprakash
> Assignee: Harish Jaiprakash
> Attachments: TEZ-3359.01.patch
>
>
> We publish too many events to ATS, this increases the file size for ATS.
> Reduce data size logged into ATS by:
> * Having a more granular control over the loglevel, disable task level logs,
> or all logs and so on.
> * Disable logging counters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
