[
https://issues.apache.org/jira/browse/HIVE-20861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675860#comment-16675860
]
Aihua Xu edited comment on HIVE-20861 at 11/5/18 10:56 PM:
-----------------------------------------------------------
Seems spark-16759 has the limitation that "When running on Spark Yarn cluster
mode, the driver is unable to pass 'spark.log.callerContext' to Yarn client and
AM since Yarn client and AM have already started before the driver performs
.config("spark.log.callerContext", "infoSpecifiedByUpstreamApp")."
[~WeiqingYang]] I'm wondering if we could implement such callerContext in
SparkContext as JobGroup within sparkContext to solve such issue? [~stakiar] do
you have any idea on this?
was (Author: aihuaxu):
Seems spark-16759 has the limitation that "When running on Spark Yarn cluster
mode, the driver is unable to pass 'spark.log.callerContext' to Yarn client and
AM since Yarn client and AM have already started before the driver performs
.config("spark.log.callerContext", "infoSpecifiedByUpstreamApp")."
[~Weiqingy] I'm wondering if we could implement such callerContext in
SparkContext as JobGroup within sparkContext to solve such issue? [~stakiar] do
you have any idea on this?
> Pass queryId as the client CallerContext to Spark
> --------------------------------------------------
>
> Key: HIVE-20861
> URL: https://issues.apache.org/jira/browse/HIVE-20861
> Project: Hive
> Issue Type: Improvement
> Reporter: Aihua Xu
> Assignee: Aihua Xu
> Priority: Major
>
> SPARK-16759 exposes a way for the client to pass the client CallerContext
> such as QueryId. For better debug, hive should pass queryId to spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
