[jira] [Commented] (SPARK-5439) Expose yarn app id for yarn mode

[ https://issues.apache.org/jira/browse/SPARK-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294872#comment-14294872 ] Chengxiang Li commented on SPARK-5439: -- I think the gap here is that, when launch a

[jira] [Created] (SPARK-5377) Dynamically add jar into Spark Driver's classpath.

Chengxiang Li created SPARK-5377: Summary: Dynamically add jar into Spark Driver's classpath. Key: SPARK-5377 URL: https://issues.apache.org/jira/browse/SPARK-5377 Project: Spark Issue Type:

[jira] [Commented] (SPARK-5377) Dynamically add jar into Spark Driver's classpath.

[ https://issues.apache.org/jira/browse/SPARK-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14288783#comment-14288783 ] Chengxiang Li commented on SPARK-5377: -- cc [~xuefuz], [~rxin], [~Grace Huang].

[jira] [Created] (SPARK-4955) Executor does not get killed after configured interval.

Chengxiang Li created SPARK-4955: Summary: Executor does not get killed after configured interval. Key: SPARK-4955 URL: https://issues.apache.org/jira/browse/SPARK-4955 Project: Spark Issue

[jira] [Commented] (SPARK-4955) Executor does not get killed after configured interval.

[ https://issues.apache.org/jira/browse/SPARK-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258138#comment-14258138 ] Chengxiang Li commented on SPARK-4955: -- I verified this feature with Hive on Spark,

[jira] [Commented] (SPARK-4955) Executor does not get killed after configured interval.

[ https://issues.apache.org/jira/browse/SPARK-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14258139#comment-14258139 ] Chengxiang Li commented on SPARK-4955: -- cc:[~andrewor14] Executor does not get

[jira] [Created] (SPARK-4585) Spark dynamic scaling executors use upper limit value as default.

Chengxiang Li created SPARK-4585: Summary: Spark dynamic scaling executors use upper limit value as default. Key: SPARK-4585 URL: https://issues.apache.org/jira/browse/SPARK-4585 Project: Spark

[jira] [Commented] (SPARK-2321) Design a proper progress reporting event listener API

[ https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143191#comment-14143191 ] Chengxiang Li commented on SPARK-2321: -- I agree that a stable, immutable, and

[jira] [Commented] (SPARK-3543) Write TaskContext in Java and expose it through a static accessor

[ https://issues.apache.org/jira/browse/SPARK-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135016#comment-14135016 ] Chengxiang Li commented on SPARK-3543: -- I think this would solve SPARK-2895 as well,

[jira] [Commented] (SPARK-2321) Design a proper progress reporting event listener API

[ https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121085#comment-14121085 ] Chengxiang Li commented on SPARK-2321: -- I collect some hive side requirement here,

[jira] [Commented] (SPARK-2321) Design a proper progress reporting event listener API

[ https://issues.apache.org/jira/browse/SPARK-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121173#comment-14121173 ] Chengxiang Li commented on SPARK-2321: -- I'm not sure whether i understand you right,

[jira] [Updated] (SPARK-2633) enhance spark listener API to gather more spark job information

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2633: - Attachment: Spark listener enhancement for Hive on Spark job monitor and statistic.docx enhance

[jira] [Updated] (SPARK-2633) enhance spark listener API to gather more spark job information

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2633: - Attachment: (was: Spark listener enhancement for Hive on Spark job monitor and

[jira] [Updated] (SPARK-2895) Support mapPartitionsWithContext in Spark Java API

[ https://issues.apache.org/jira/browse/SPARK-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2895: - Description: This is a requirement from Hive on Spark, mapPartitionsWithContext only exists in

[jira] [Updated] (SPARK-2633) enhance spark listener API to gather more spark job information

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2633: - Attachment: (was: Spark listener enhancement for Hive on Spark job monitor and

[jira] [Updated] (SPARK-2633) enhance spark listener API to gather more spark job information

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2633: - Attachment: Spark listener enhancement for Hive on Spark job monitor and statistic.docx

[jira] [Commented] (SPARK-2895) Support mapPartitionsWithContext in Spark Java API

[ https://issues.apache.org/jira/browse/SPARK-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116221#comment-14116221 ] Chengxiang Li commented on SPARK-2895: -- pull

[jira] [Commented] (SPARK-2633) enhance spark listener API to gather more spark job information

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113611#comment-14113611 ] Chengxiang Li commented on SPARK-2633: -- It's quite subjective I think, like Hive on

[jira] [Created] (SPARK-3199) native Java spark listener API support

Chengxiang Li created SPARK-3199: Summary: native Java spark listener API support Key: SPARK-3199 URL: https://issues.apache.org/jira/browse/SPARK-3199 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-2633) enhance spark listener API to gather more spark job information

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2633: - Summary: enhance spark listener API to gather more spark job information (was: support register

[jira] [Updated] (SPARK-2633) enhance spark listener API to gather more spark job information

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2633: - Description: Based on Hive on Spark job status monitoring and statistic collection requirement,

[jira] [Commented] (SPARK-2633) enhance spark listener API to gather more spark job information

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108887#comment-14108887 ] Chengxiang Li commented on SPARK-2633: -- I would start to work on this issue, for

[jira] [Commented] (SPARK-2895) Support mapPartitionsWithContext in Spark Java API

[ https://issues.apache.org/jira/browse/SPARK-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088895#comment-14088895 ] Chengxiang Li commented on SPARK-2895: -- {quote} Can we add the label hive to all the

[jira] [Updated] (SPARK-2633) support register spark listener to listener bus with Java API

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2633: - Labels: hive (was: ) support register spark listener to listener bus with Java API

[jira] [Updated] (SPARK-2636) no where to get job identifier while submit spark job through spark API

[ https://issues.apache.org/jira/browse/SPARK-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2636: - Component/s: Java API no where to get job identifier while submit spark job through spark API

[jira] [Updated] (SPARK-2636) no where to get job identifier while submit spark job through spark API

[ https://issues.apache.org/jira/browse/SPARK-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2636: - Labels: hive (was: ) no where to get job identifier while submit spark job through spark API

[jira] [Created] (SPARK-2895) Support mapPartitionsWithContext in Spark Java API

Chengxiang Li created SPARK-2895: Summary: Support mapPartitionsWithContext in Spark Java API Key: SPARK-2895 URL: https://issues.apache.org/jira/browse/SPARK-2895 Project: Spark Issue Type:

[jira] [Commented] (SPARK-2895) Support mapPartitionsWithContext in Spark Java API

[ https://issues.apache.org/jira/browse/SPARK-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088736#comment-14088736 ] Chengxiang Li commented on SPARK-2895: -- cc [~rxin] [~brocknoland] [~szehon] Support

[jira] [Commented] (SPARK-2636) no where to get job identifier while submit spark job through spark API

[ https://issues.apache.org/jira/browse/SPARK-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087102#comment-14087102 ] Chengxiang Li commented on SPARK-2636: -- {quote} There are two ways I think. One is

[jira] [Updated] (SPARK-2633) support register spark listener to listener bus with Java API

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2633: - Attachment: Spark listener enhancement for Hive on Spark job monitor and statistic.docx I add a

[jira] [Commented] (SPARK-2633) support register spark listener to listener bus with Java API

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080425#comment-14080425 ] Chengxiang Li commented on SPARK-2633: -- Thanks, [~vanzin] {quote} Registering

[jira] [Commented] (SPARK-2633) support register spark listener to listener bus with Java API

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071402#comment-14071402 ] Chengxiang Li commented on SPARK-2633: -- For Hive job status monitor, spark listener

[jira] [Commented] (SPARK-2633) support register spark listener to listener bus with Java API

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071478#comment-14071478 ] Chengxiang Li commented on SPARK-2633: -- add 2 more: # StageInfo class is not well

[jira] [Created] (SPARK-2633) support register spark listener to listener bus with Java API

Chengxiang Li created SPARK-2633: Summary: support register spark listener to listener bus with Java API Key: SPARK-2633 URL: https://issues.apache.org/jira/browse/SPARK-2633 Project: Spark

[jira] [Commented] (SPARK-2633) support register spark listener to listener bus with Java API

[ https://issues.apache.org/jira/browse/SPARK-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071310#comment-14071310 ] Chengxiang Li commented on SPARK-2633: -- cc [~rxin] [~xuefuz] support register spark

[jira] [Created] (SPARK-2636) no where to get job identifier while submit spark job through spark API

Chengxiang Li created SPARK-2636: Summary: no where to get job identifier while submit spark job through spark API Key: SPARK-2636 URL: https://issues.apache.org/jira/browse/SPARK-2636 Project: Spark

[jira] [Updated] (SPARK-2636) no where to get job identifier while submit spark job through spark API

[ https://issues.apache.org/jira/browse/SPARK-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2636: - Description: In Hive on Spark, we want to track spark job status through Spark API, the basic

[jira] [Commented] (SPARK-2636) no where to get job identifier while submit spark job through spark API

[ https://issues.apache.org/jira/browse/SPARK-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14071311#comment-14071311 ] Chengxiang Li commented on SPARK-2636: -- cc [~rxin] [~xuefuz] no where to get job

[jira] [Updated] (SPARK-2636) no where to get job identifier while submit spark job through spark API

[ https://issues.apache.org/jira/browse/SPARK-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengxiang Li updated SPARK-2636: - Description: In Hive on Spark, we want to track spark job status through Spark API, the basic