[
https://issues.apache.org/jira/browse/TEZ-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192271#comment-17192271
]
László Bodor edited comment on TEZ-4039 at 10/21/22 8:36 AM:
-------------------------------------------------------------
there is a working version in: [^TEZ-4039.05.patch]
testing:
{code}
git clone https://github.com/abstractdog/tez --branch TEZ-4039 --single-branch
cd tez
mvn clean install -Dtest=TestTezJobs#testHashJoinExampleWithLogPattern
cd
tez-tests/target/org.apache.tez.test.TestTezJobs/org.apache.tez.test.TestTezJobs-logDir-nm-0_0
# check syslog_dag* and syslog_attempt* files
{code}
[~jeagles], [~prasanth_j]: could you please take a look?
[^TEZ-4039.05.patch] is a working version, I've included manual testing steps
the change includes all features discussed above + a configurable layout
pattern (which turned out to be a very convenient way, that can
replace/eliminate painful enviroment-dependent steps, etc. deploying log4j
configuration files), which is turned off by default
was (Author: abstractdog):
there is a working version in: [^TEZ-4039.05.patch]
testing:
{code}
git clone https://github.com/abstractdog/tez --branch TEZ-4039 --single-branch
cd tez
mvn clean install -pl tez-api -pl tez-dag -pl tez-common -pl
tez-runtime-internals -pl tez-tests -pl tez-examples
-Dtest=TestTezJobs#testHashJoinExampleWithLogPattern
cd
tez-tests/target/org.apache.tez.test.TestTezJobs/org.apache.tez.test.TestTezJobs-logDir-nm-0_0
# check syslog_dag* and syslog_attempt* files
{code}
[~jeagles], [~prasanth_j]: could you please take a look?
[^TEZ-4039.05.patch] is a working version, I've included manual testing steps
the change includes all features discussed above + a configurable layout
pattern (which turned out to be a very convenient way, that can
replace/eliminate painful enviroment-dependent steps, etc. deploying log4j
configuration files), which is turned off by default
> Tez should inject dag id, query id into MDC
> -------------------------------------------
>
> Key: TEZ-4039
> URL: https://issues.apache.org/jira/browse/TEZ-4039
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Prasanth Jayachandran
> Assignee: László Bodor
> Priority: Major
> Attachments: TEZ-4039.01.patch, TEZ-4039.02.patch, TEZ-4039.03.patch,
> TEZ-4039.04.patch, TEZ-4039.05.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Tez currently uses CallableWithNdc to store thread specific context. It
> should also inject the context into MDC so that pattern layout can dump the
> contexts from MDC (with NDC it is not possible to read the context in pattern
> lyaout).
> Hive for example, sets queryId in the MDC and pattern layout prints the
> queryId
>
> {code:java}
> %d{ISO8601} %-5p [%t (%X{queryId})] %c{2}: %m%n
> {code}
> Llap sets dagId, fragmentId and queryId into MDC which is used for queryId
> based routing of logging.
> Similarly, Tez AM should set dagId and queryId (if available) into MDC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
