[
https://issues.apache.org/jira/browse/HIVE-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972085#comment-14972085
]
Vikram Dixit K edited comment on HIVE-12249 at 10/23/15 11:18 PM:
------------------------------------------------------------------
Update with a test.
Sample output
{code}
vikram$ grep -r mrrTest * | grep syslog
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:22,104
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151021_35b01963-5cf6-4dcc-b6ef-5279fb08e2a9:32,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=mrrTestmainvikram_20151023151021_35b01963-5cf6-4dcc-b6ef-5279fb08e2a9,
blob=
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:28,274
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151028_024fa54e-2d53-4c20-8d82-f282cd438955:34,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=mrrTestmainvikram_20151023151028_024fa54e-2d53-4c20-8d82-f282cd438955,
blob=
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:30,430
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151030_2c0cd0e9-3174-42f3-8258-205b480bde3f:36,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=mrrTestmainvikram_20151023151030_2c0cd0e9-3174-42f3-8258-205b480bde3f,
blob=
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:32,802
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151032_2c6065c2-d54b-47f1-921f-f44e678bb942:38,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=mrrTestmainvikram_20151023151032_2c6065c2-d54b-47f1-921f-f44e678bb942,
blob=
vikram$ grep -r Test2 * | grep syslog
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:34,545
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151034_0db3180c-4ccc-4076-81ba-217da09a4eb9:40,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=Test2mainvikram_20151023151034_0db3180c-4ccc-4076-81ba-217da09a4eb9,
blob=
{code}
In the client logs:
{code}
2015-10-23T15:10:34,278 INFO [Test2main]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(117)) - <PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver>
2015-10-23T15:10:34,278 DEBUG [Test2main]: hbase.HBaseReadWrite
(HBaseReadWrite.java:flushCatalogCache(1909)) - Dumping metric: table cache
hits 5
2015-10-23T15:10:34,278 DEBUG [Test2main]: hbase.HBaseReadWrite
(HBaseReadWrite.java:flushCatalogCache(1909)) - Dumping metric: table cache
misses 1
2015-10-23T15:10:34,278 DEBUG [Test2main]: hbase.HBaseReadWrite
(HBaseReadWrite.java:flushCatalogCache(1909)) - Dumping metric: table cache
overflows 0
2015-10-23T15:10:34,278 DEBUG [Test2main]: hbase.HBaseReadWrite
(HBaseReadWrite.java:flushCatalogCache(1909)) - Dumping metric: partition cache
hits 0
2015-10-23T15:10:34,278 DEBUG [Test2main]: hbase.HBaseReadWrite
(HBaseReadWrite.java:flushCatalogCache(1909)) - Dumping metric: partition cache
misses 0
2015-10-23T15:10:34,278 DEBUG [Test2main]: hbase.HBaseReadWrite
(HBaseReadWrite.java:flushCatalogCache(1909)) - Dumping metric: partition cache
overflows 0
2015-10-23T15:10:34,279 DEBUG [Test2main]: hbase.HBaseReadWrite
(HBaseReadWrite.java:flushCatalogCache(1909)) - Dumping metric: storage
descriptor cache hits 0
{code}
was (Author: vikram.dixit):
Update with a test.
Sample output
{code}
vikram$ grep -r mrrTest * | grep syslog
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:22,104
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151021_35b01963-5cf6-4dcc-b6ef-5279fb08e2a9:32,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=mrrTestmainvikram_20151023151021_35b01963-5cf6-4dcc-b6ef-5279fb08e2a9,
blob=
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:28,274
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151028_024fa54e-2d53-4c20-8d82-f282cd438955:34,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=mrrTestmainvikram_20151023151028_024fa54e-2d53-4c20-8d82-f282cd438955,
blob=
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:30,430
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151030_2c0cd0e9-3174-42f3-8258-205b480bde3f:36,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=mrrTestmainvikram_20151023151030_2c0cd0e9-3174-42f3-8258-205b480bde3f,
blob=
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:32,802
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151032_2c6065c2-d54b-47f1-921f-f44e678bb942:38,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=mrrTestmainvikram_20151023151032_2c6065c2-d54b-47f1-921f-f44e678bb942,
blob=
vikram$ grep -r Test2 * | grep syslog
target/hive/hive-logDir-nm-0_3/application_1445638125157_0001/container_1445638125157_0001_01_000001/syslog:2015-10-23T15:10:34,545
INFO [IPC Server handler 0 on 56693] app.DAGAppMaster: Running DAG:
vikram_20151023151034_0db3180c-4ccc-4076-81ba-217da09a4eb9:40,
callerContext=context=HIVE, callerType=HIVE_QUERY_ID,
callerId=Test2mainvikram_20151023151034_0db3180c-4ccc-4076-81ba-217da09a4eb9,
blob=
{code}
> Improve logging with tez
> ------------------------
>
> Key: HIVE-12249
> URL: https://issues.apache.org/jira/browse/HIVE-12249
> Project: Hive
> Issue Type: Bug
> Components: Tez
> Affects Versions: 1.2.1
> Reporter: Vikram Dixit K
> Assignee: Vikram Dixit K
> Attachments: HIVE-12249.1.patch, HIVE-12249.2.patch,
> HIVE-12249.3.patch
>
>
> We need to improve logging across the board. TEZ-2851 added a caller context
> so that one can correlate logs with the application. This jira adds a new
> configuration for users that can be used to correlate the logs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
