[jira] [Updated] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[ https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-13517: Issue Type: Sub-task (was: Bug) Parent: HIVE-17718 > Hive logs in Spark Executor and Driver should show thread-id. > - > > Key: HIVE-13517 > URL: https://issues.apache.org/jira/browse/HIVE-13517 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 1.2.1, 2.0.0 >Reporter: Szehon Ho >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-13517.1.patch, HIVE-13517.2.patch, > executor-driver-log.PNG > > > In Spark, there might be more than one task running in one executor. > Similarly, there may be more than one thread running in Driver. > This makes debugging through the logs a nightmare. It would be great if there > could be thread-ids in the logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[ https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-13517: Status: Open (was: Patch Available) > Hive logs in Spark Executor and Driver should show thread-id. > - > > Key: HIVE-13517 > URL: https://issues.apache.org/jira/browse/HIVE-13517 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.0.0, 1.2.1 >Reporter: Szehon Ho >Assignee: Sahil Takiar > Attachments: executor-driver-log.PNG, HIVE-13517.1.patch, > HIVE-13517.2.patch > > > In Spark, there might be more than one task running in one executor. > Similarly, there may be more than one thread running in Driver. > This makes debugging through the logs a nightmare. It would be great if there > could be thread-ids in the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[
https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-13517:
Attachment: HIVE-13517.2.patch
Attaching updated patch that adds some testing. Instead of explicitly copying
the {{log4j2.properties}} file from {{data/conf/spark/}} into the Spark
distribution, the new config {{hive.spark.log4j.file}} is now used.
I'm not 100% sure this was the original intention of this JIRA, but it is a
useful feature. Users can now add custom log4j files without modifying the
Spark installation.
In a future patch, we could add a better default log4j properties file, which
has the thread-id.
[~kellyzly] what do you think of this patch?
> Hive logs in Spark Executor and Driver should show thread-id.
> -
>
> Key: HIVE-13517
> URL: https://issues.apache.org/jira/browse/HIVE-13517
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Szehon Ho
>Assignee: Sahil Takiar
> Attachments: executor-driver-log.PNG, HIVE-13517.1.patch,
> HIVE-13517.2.patch
>
>
> In Spark, there might be more than one task running in one executor.
> Similarly, there may be more than one thread running in Driver.
> This makes debugging through the logs a nightmare. It would be great if there
> could be thread-ids in the logs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[ https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-13517: Status: Patch Available (was: Open) > Hive logs in Spark Executor and Driver should show thread-id. > - > > Key: HIVE-13517 > URL: https://issues.apache.org/jira/browse/HIVE-13517 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 2.0.0, 1.2.1 >Reporter: Szehon Ho >Assignee: Sahil Takiar > Attachments: executor-driver-log.PNG, HIVE-13517.1.patch > > > In Spark, there might be more than one task running in one executor. > Similarly, there may be more than one thread running in Driver. > This makes debugging through the logs a nightmare. It would be great if there > could be thread-ids in the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[ https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-13517: Attachment: HIVE-13517.1.patch Attaching a pre-lim patch. This basically makes it possible for users to specify the location of a custom log4j.properties to be used in the HoS driver and executor. The advantage is that different sessions can now use different log4j configs, rather than a single log4j config for the entire spark installation. > Hive logs in Spark Executor and Driver should show thread-id. > - > > Key: HIVE-13517 > URL: https://issues.apache.org/jira/browse/HIVE-13517 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.1, 2.0.0 >Reporter: Szehon Ho >Assignee: Sahil Takiar > Attachments: executor-driver-log.PNG, HIVE-13517.1.patch > > > In Spark, there might be more than one task running in one executor. > Similarly, there may be more than one thread running in Driver. > This makes debugging through the logs a nightmare. It would be great if there > could be thread-ids in the logs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[ https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyunzhang_intel updated HIVE-13517: Attachment: executor-driver-log.PNG > Hive logs in Spark Executor and Driver should show thread-id. > - > > Key: HIVE-13517 > URL: https://issues.apache.org/jira/browse/HIVE-13517 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.1, 2.0.0 >Reporter: Szehon Ho >Assignee: liyunzhang_intel > Attachments: executor-driver-log.PNG > > > In Spark, there might be more than one task running in one executor. > Similarly, there may be more than one thread running in Driver. > This makes debugging through the logs a nightmare. It would be great if there > could be thread-ids in the logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[ https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-13517: - It might be possible to pass in log4j.properties via some flag of spark-submit. We need to investigate. > Hive logs in Spark Executor and Driver should show thread-id. > - > > Key: HIVE-13517 > URL: https://issues.apache.org/jira/browse/HIVE-13517 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.2.1, 2.0.0 >Reporter: Szehon Ho >Assignee: Szehon Ho > > In Spark, there might be more than one task running in one executor. > Similarly, there may be more than one thread running in Driver. > This makes debugging through the logs a nightmare. It would be great if there > could be thread-ids in the logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
