[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220395#comment-16220395 ] Saisai Shao commented on SPARK-22357: - bq. In our specific case this was a problem, because we

[jira] [Commented] (SPARK-22357) SparkContext.binaryFiles ignore minPartitions parameter

[ https://issues.apache.org/jira/browse/SPARK-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16220160#comment-16220160 ] Saisai Shao commented on SPARK-22357: - [~WeichenXu123] would you please format the code in JIRA

[jira] [Assigned] (SPARK-21840) Allow multiple SparkSubmit invocations in same JVM without polluting system properties

[ https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21840: --- Assignee: Marcelo Vanzin > Allow multiple SparkSubmit invocations in same JVM without

[jira] [Resolved] (SPARK-21840) Allow multiple SparkSubmit invocations in same JVM without polluting system properties

[ https://issues.apache.org/jira/browse/SPARK-21840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21840. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19519

[jira] [Updated] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22319: Fix Version/s: 2.2.1 > SparkSubmit calls getFileStatus before calling loginUserFromKeytab >

[jira] [Closed] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-22272. --- > killing task may cause the executor progress hang because of the JVM bug >

[jira] [Resolved] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22272. - Resolution: Won't Fix > killing task may cause the executor progress hang because of the JVM bug

[jira] [Commented] (SPARK-22272) killing task may cause the executor progress hang because of the JVM bug

[ https://issues.apache.org/jira/browse/SPARK-22272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214569#comment-16214569 ] Saisai Shao commented on SPARK-22272: - Setting "spark.file.transferTo" to "false" will potentially

[jira] [Assigned] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22319: --- Assignee: Steven Rand > SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[jira] [Resolved] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22319. - Resolution: Fixed Fix Version/s: 2.3.0 > SparkSubmit calls getFileStatus before calling

[jira] [Updated] (SPARK-22319) SparkSubmit calls getFileStatus before calling loginUserFromKeytab

[ https://issues.apache.org/jira/browse/SPARK-22319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22319: Affects Version/s: 2.2.0 > SparkSubmit calls getFileStatus before calling loginUserFromKeytab >

[jira] [Commented] (SPARK-22312) Spark job stuck with no executor due to bug in Executor Allocation Manager

[ https://issues.apache.org/jira/browse/SPARK-22312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16212071#comment-16212071 ] Saisai Shao commented on SPARK-22312: - I think it is duplicated to SPARK-11334 according to your

[jira] [Resolved] (SPARK-22290) Starting second context in same JVM fails to get new Hive delegation token

[ https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22290. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19509

[jira] [Assigned] (SPARK-22290) Starting second context in same JVM fails to get new Hive delegation token

[ https://issues.apache.org/jira/browse/SPARK-22290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22290: --- Assignee: Marcelo Vanzin > Starting second context in same JVM fails to get new Hive

[jira] [Commented] (SPARK-22242) streaming job failed to restart from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16205018#comment-16205018 ] Saisai Shao commented on SPARK-22242: - It is not resolved, but it is the same type of problem as

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202900#comment-16202900 ] Saisai Shao commented on SPARK-9: - {quote} I don't think that limited familiarity with a new

[jira] [Commented] (SPARK-22229) SPIP: RDMA Accelerated Shuffle Engine

[ https://issues.apache.org/jira/browse/SPARK-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201950#comment-16201950 ] Saisai Shao commented on SPARK-9: - My concern is about how to maintain this code in the

[jira] [Commented] (SPARK-22062) BlockManager does not account for memory consumed by remote fetches

[ https://issues.apache.org/jira/browse/SPARK-22062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201305#comment-16201305 ] Saisai Shao commented on SPARK-22062: - Yes, there potentially has OOM problem, but I think this kind

[jira] [Commented] (SPARK-22243) streaming job failed to restart from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200372#comment-16200372 ] Saisai Shao commented on SPARK-22243: - Yes, it is a related issue regarding to Spark Streaming

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198674#comment-16198674 ] Saisai Shao commented on SPARK-21737: - I was trying to understand how Spark communicate with Mesos,

[jira] [Commented] (SPARK-22199) Spark Job on YARN fails with executors "Slave registration failed"

[ https://issues.apache.org/jira/browse/SPARK-22199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198258#comment-16198258 ] Saisai Shao commented on SPARK-22199: - Can you please list the steps to reproduce this issue? Also

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198204#comment-16198204 ] Saisai Shao commented on SPARK-21737: - Hi [~tgraves], I'm trying to understand the design of this. As

[jira] [Assigned] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22074: --- Assignee: Li Yuanjian > Task killed by other attempt task should not be resubmitted >

[jira] [Resolved] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22074. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19287

[jira] [Assigned] (SPARK-22135) metrics in spark-dispatcher not being registered properly

[ https://issues.apache.org/jira/browse/SPARK-22135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22135: --- Assignee: paul mackles > metrics in spark-dispatcher not being registered properly >

[jira] [Resolved] (SPARK-22135) metrics in spark-dispatcher not being registered properly

[ https://issues.apache.org/jira/browse/SPARK-22135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22135. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183581#comment-16183581 ] Saisai Shao commented on SPARK-22151: - Checking the Yarn client code, looks like there's specific

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16183563#comment-16183563 ] Saisai Shao commented on SPARK-21737: - [~yoonlee95], are you still working on this thing? I think

[jira] [Updated] (SPARK-22123) Add latest failure reason for task set blacklist

[ https://issues.apache.org/jira/browse/SPARK-22123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22123: Priority: Minor (was: Major) > Add latest failure reason for task set blacklist >

[jira] [Assigned] (SPARK-22123) Add latest failure reason for task set blacklist

[ https://issues.apache.org/jira/browse/SPARK-22123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22123: --- Assignee: zhoukang > Add latest failure reason for task set blacklist >

[jira] [Resolved] (SPARK-22123) Add latest failure reason for task set blacklist

[ https://issues.apache.org/jira/browse/SPARK-22123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22123. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19338

[jira] [Resolved] (SPARK-20785) Spark should provide jump links and add (count) in the SQL web ui.

[ https://issues.apache.org/jira/browse/SPARK-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-20785. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19346

[jira] [Assigned] (SPARK-20785) Spark should provide jump links and add (count) in the SQL web ui.

[ https://issues.apache.org/jira/browse/SPARK-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-20785: --- Assignee: guoxiaolongzte > Spark should provide jump links and add (count) in the SQL web

[jira] [Commented] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16182062#comment-16182062 ] Saisai Shao commented on SPARK-22074: - So if I understand correctly, this happens when speculation is

[jira] [Commented] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181924#comment-16181924 ] Saisai Shao commented on SPARK-22074: - Hey [~XuanYuan], I'm a little confused why there will be a

[jira] [Commented] (SPARK-9103) Tracking spark's memory usage

[ https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181905#comment-16181905 ] Saisai Shao commented on SPARK-9103: Hi [~irashid], thanks a lot for your response. I agree that your

[jira] [Commented] (SPARK-22074) Task killed by other attempt task should not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180506#comment-16180506 ] Saisai Shao commented on SPARK-22074: - Hi [~XuanYuan], can you please help me to understand your

[jira] [Commented] (SPARK-9103) Tracking spark's memory usage

[ https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176073#comment-16176073 ] Saisai Shao commented on SPARK-9103: [~irashid] I would like to hear your suggestion on displaying

[jira] [Assigned] (SPARK-21934) Expose Netty memory usage via Metrics System

[ https://issues.apache.org/jira/browse/SPARK-21934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21934: --- Assignee: Saisai Shao > Expose Netty memory usage via Metrics System >

[jira] [Resolved] (SPARK-21934) Expose Netty memory usage via Metrics System

[ https://issues.apache.org/jira/browse/SPARK-21934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21934. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19160

[jira] [Assigned] (SPARK-22030) GraphiteSink fails to re-connect to Graphite instances behind an ELB or any other auto-scaled LB

[ https://issues.apache.org/jira/browse/SPARK-22030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-22030: --- Assignee: Alex Mikhailau > GraphiteSink fails to re-connect to Graphite instances behind an

[jira] [Updated] (SPARK-22030) GraphiteSink fails to re-connect to Graphite instances behind an ELB or any other auto-scaled LB

[ https://issues.apache.org/jira/browse/SPARK-22030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-22030: Priority: Minor (was: Critical) > GraphiteSink fails to re-connect to Graphite instances behind

[jira] [Resolved] (SPARK-22030) GraphiteSink fails to re-connect to Graphite instances behind an ELB or any other auto-scaled LB

[ https://issues.apache.org/jira/browse/SPARK-22030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-22030. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19210

[jira] [Updated] (SPARK-21902) BlockManager.doPut will hide actually exception when exception thrown in finally block

[ https://issues.apache.org/jira/browse/SPARK-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21902: Priority: Trivial (was: Major) > BlockManager.doPut will hide actually exception when exception

[jira] [Updated] (SPARK-21902) BlockManager.doPut will hide actually exception when exception thrown in finally block

[ https://issues.apache.org/jira/browse/SPARK-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21902: Issue Type: Improvement (was: Wish) > BlockManager.doPut will hide actually exception when

[jira] [Assigned] (SPARK-21902) BlockManager.doPut will hide actually exception when exception thrown in finally block

[ https://issues.apache.org/jira/browse/SPARK-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21902: --- Assignee: zhoukang > BlockManager.doPut will hide actually exception when exception thrown

[jira] [Resolved] (SPARK-21902) BlockManager.doPut will hide actually exception when exception thrown in finally block

[ https://issues.apache.org/jira/browse/SPARK-21902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21902. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19171

[jira] [Resolved] (SPARK-21922) When executor failed and task metrics have not send to driver,the status will always be 'RUNNING' and the duration will be 'CurrentTime - launchTime'

[ https://issues.apache.org/jira/browse/SPARK-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21922. - Resolution: Fixed Assignee: zhoukang Target Version/s: 2.3.0 > When

[jira] [Updated] (SPARK-21922) When executor failed and task metrics have not send to driver,the status will always be 'RUNNING' and the duration will be 'CurrentTime - launchTime'

[ https://issues.apache.org/jira/browse/SPARK-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21922: Fix Version/s: 2.3.0 > When executor failed and task metrics have not send to driver,the status

[jira] [Updated] (SPARK-21922) When executor failed and task metrics have not send to driver,the status will always be 'RUNNING' and the duration will be 'CurrentTime - launchTime'

[ https://issues.apache.org/jira/browse/SPARK-21922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21922: Target Version/s: (was: 2.3.0) > When executor failed and task metrics have not send to

[jira] [Commented] (SPARK-21513) SQL to_json should support all column types

[ https://issues.apache.org/jira/browse/SPARK-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164000#comment-16164000 ] Saisai Shao commented on SPARK-21513: - [~goldmedal] is it your correct JIRA name? [~hyukjin.kwon] I

[jira] [Assigned] (SPARK-21513) SQL to_json should support all column types

[ https://issues.apache.org/jira/browse/SPARK-21513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21513: --- Assignee: Jia-Xuan Liu > SQL to_json should support all column types >

[jira] [Commented] (SPARK-21943) When I use rest Api (/ applications / [app-id] / jobs / [job-id]) to view some of the jobs that are running jobs, the returned json information is missing the “descrip

[ https://issues.apache.org/jira/browse/SPARK-21943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16161008#comment-16161008 ] Saisai Shao commented on SPARK-21943: - If you're trying to report bugs, I think you should provide a

[jira] [Commented] (SPARK-21943) When I use rest Api (/ applications / [app-id] / jobs / [job-id]) to view some of the jobs that are running jobs, the returned json information is missing the “descrip

[ https://issues.apache.org/jira/browse/SPARK-21943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160814#comment-16160814 ] Saisai Shao commented on SPARK-21943: - >From the code, it says that job description is gotten from

[jira] [Closed] (SPARK-21906) No need to runAsSparkUser to switch UserGroupInformation in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-21906. --- Resolution: Not A Problem > No need to runAsSparkUser to switch UserGroupInformation in YARN mode >

[jira] [Assigned] (SPARK-20098) DataType's typeName method returns with 'StructF' in case of StructField

[ https://issues.apache.org/jira/browse/SPARK-20098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-20098: --- Assignee: Peter Szalai > DataType's typeName method returns with 'StructF' in case of

[jira] [Commented] (SPARK-21942) DiskBlockManager crashing when a root local folder has been externally deleted by OS

[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158320#comment-16158320 ] Saisai Shao commented on SPARK-21942: - Personally I would like to fail fast if such things happened,

[jira] [Commented] (SPARK-21942) DiskBlockManager crashing when a root local folder has been externally deleted by OS

[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158271#comment-16158271 ] Saisai Shao commented on SPARK-21942: - {quote}

[jira] [Assigned] (SPARK-21939) Use TimeLimits instead of Timeouts

[ https://issues.apache.org/jira/browse/SPARK-21939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21939: --- Assignee: Dongjoon Hyun > Use TimeLimits instead of Timeouts >

[jira] [Resolved] (SPARK-21939) Use TimeLimits instead of Timeouts

[ https://issues.apache.org/jira/browse/SPARK-21939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21939. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19150

[jira] [Commented] (SPARK-21942) ix DiskBlockManager crashing when a root local folder has been externally deleted by OS

[ https://issues.apache.org/jira/browse/SPARK-21942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156914#comment-16156914 ] Saisai Shao commented on SPARK-21942: - I think in a production environment, user should always

[jira] [Updated] (SPARK-9104) expose network layer memory usage

[ https://issues.apache.org/jira/browse/SPARK-9104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-9104: --- Summary: expose network layer memory usage (was: expose network layer memory usage in shuffle part)

[jira] [Updated] (SPARK-21934) Expose Netty memory usage via Metrics System

[ https://issues.apache.org/jira/browse/SPARK-21934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21934: Issue Type: Sub-task (was: New Feature) Parent: SPARK-9103 > Expose Netty memory usage

[jira] [Created] (SPARK-21934) Expose Netty memory usage via Metrics System

Saisai Shao created SPARK-21934: --- Summary: Expose Netty memory usage via Metrics System Key: SPARK-21934 URL: https://issues.apache.org/jira/browse/SPARK-21934 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-21917) Remote http(s) resources is not supported in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154744#comment-16154744 ] Saisai Shao commented on SPARK-21917: - Thanks [~vanzin], I think your suggestion is great, this will

[jira] [Assigned] (SPARK-18061) Spark Thriftserver needs to create SPNego principal

[ https://issues.apache.org/jira/browse/SPARK-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-18061: --- Assignee: Saisai Shao > Spark Thriftserver needs to create SPNego principal >

[jira] [Resolved] (SPARK-18061) Spark Thriftserver needs to create SPNego principal

[ https://issues.apache.org/jira/browse/SPARK-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-18061. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18628

[jira] [Comment Edited] (SPARK-21917) Remote http(s) resources is not supported in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153141#comment-16153141 ] Saisai Shao edited comment on SPARK-21917 at 9/5/17 8:36 AM: - I'm inclining

[jira] [Commented] (SPARK-21917) Remote http(s) resources is not supported in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153141#comment-16153141 ] Saisai Shao commented on SPARK-21917: - I'm inclining to choose option 1, the only overhead is

[jira] [Created] (SPARK-21917) Remote http(s) resources is not supported in YARN mode

Saisai Shao created SPARK-21917: --- Summary: Remote http(s) resources is not supported in YARN mode Key: SPARK-21917 URL: https://issues.apache.org/jira/browse/SPARK-21917 Project: Spark Issue

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150258#comment-16150258 ] Saisai Shao commented on SPARK-21888: - Jars added by "--jars" will be added to client classpath in

[jira] [Commented] (SPARK-21882) OutputMetrics doesn't count written bytes correctly in the saveAsHadoopDataset function

[ https://issues.apache.org/jira/browse/SPARK-21882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150260#comment-16150260 ] Saisai Shao commented on SPARK-21882: - Please submit the patch to Github Apache Spark repo. >

[jira] [Assigned] (SPARK-21658) Adds the default None for value in na.replace in PySpark to match

[ https://issues.apache.org/jira/browse/SPARK-21658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-21658: --- Assignee: Chin Han Yu > Adds the default None for value in na.replace in PySpark to match >

[jira] [Commented] (SPARK-21658) Adds the default None for value in na.replace in PySpark to match

[ https://issues.apache.org/jira/browse/SPARK-21658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150059#comment-16150059 ] Saisai Shao commented on SPARK-21658: - Done :). > Adds the default None for value in na.replace in

[jira] [Commented] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148713#comment-16148713 ] Saisai Shao commented on SPARK-11574: - Thanks a lot [~srowen]! > Spark should support StatsD sink

[jira] [Commented] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148701#comment-16148701 ] Saisai Shao commented on SPARK-11574: - Ping [~srowen]. Hi Sean I cannot assign this JIRA to Xiaofeng,

[jira] [Commented] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148333#comment-16148333 ] Saisai Shao commented on SPARK-11574: - Maybe it is my permission issue, will ask PMC to handle it. >

[jira] [Assigned] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reassigned SPARK-17321: --- Assignee: Saisai Shao > YARN shuffle service should use good disk from

[jira] [Resolved] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-17321. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19032

[jira] [Commented] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148292#comment-16148292 ] Saisai Shao commented on SPARK-11574: - Hi Xiaofeng, is your JIRA username still available, I cannot

[jira] [Resolved] (SPARK-11574) Spark should support StatsD sink out of box

[ https://issues.apache.org/jira/browse/SPARK-11574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-11574. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 9518

[jira] [Commented] (SPARK-21829) Enable config to permanently blacklist a list of nodes

[ https://issues.apache.org/jira/browse/SPARK-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144889#comment-16144889 ] Saisai Shao commented on SPARK-21829: - I understand you concern, since it is much easier for you to

[jira] [Commented] (SPARK-21829) Enable config to permanently blacklist a list of nodes

[ https://issues.apache.org/jira/browse/SPARK-21829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141081#comment-16141081 ] Saisai Shao commented on SPARK-21829: - Cross post the comment here. Since you're running Spark on

[jira] [Closed] (SPARK-21819) UserGroupInformation initialization in SparkHadoopUtilwill overwrite user config

[ https://issues.apache.org/jira/browse/SPARK-21819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-21819. --- Resolution: Not A Problem > UserGroupInformation initialization in SparkHadoopUtilwill overwrite

[jira] [Commented] (SPARK-21819) UserGroupInformation initialization in SparkHadoopUtilwill overwrite user config

[ https://issues.apache.org/jira/browse/SPARK-21819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139590#comment-16139590 ] Saisai Shao commented on SPARK-21819: - Then I think there should no issue in Spark, right?

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139502#comment-16139502 ] Saisai Shao commented on SPARK-17321: - 1. if NM recovery is enabled, then yarn will provide a

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138320#comment-16138320 ] Saisai Shao commented on SPARK-17321: - We're facing the same issue. I think YARN shuffle service

[jira] [Commented] (SPARK-21819) UserGroupInformation initialization in SparkHadoopUtilwill overwrite user config

[ https://issues.apache.org/jira/browse/SPARK-21819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138308#comment-16138308 ] Saisai Shao commented on SPARK-21819: - I'm not sure if Spark expose the user API to set

[jira] [Commented] (SPARK-21819) UserGroupInformation initialization in SparkHadoopUtilwill overwrite user config

[ https://issues.apache.org/jira/browse/SPARK-21819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138300#comment-16138300 ] Saisai Shao commented on SPARK-21819: - I think here because `Configuration` object created in the

[jira] [Commented] (SPARK-21660) Yarn ShuffleService failed to start when the chosen directory become read-only

[ https://issues.apache.org/jira/browse/SPARK-21660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137842#comment-16137842 ] Saisai Shao commented on SPARK-21660: - Yes [~hyukjin.kwon] this is a dup JIRA. > Yarn ShuffleService

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136454#comment-16136454 ] Saisai Shao commented on SPARK-21733: - [~1028344...@qq.com], I'm really not sure what are you trying

[jira] [Resolved] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved SPARK-21733. - Resolution: Not A Problem > ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM >

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136334#comment-16136334 ] Saisai Shao commented on SPARK-21733: - I'm going to close this issue again because the behavior is

[jira] [Comment Edited] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136326#comment-16136326 ] Saisai Shao edited comment on SPARK-21733 at 8/22/17 5:41 AM: -- This is

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136326#comment-16136326 ] Saisai Shao commented on SPARK-21733: - This is because executor is killed by NM with SIGTERM, this is

[jira] [Commented] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136196#comment-16136196 ] Saisai Shao commented on SPARK-21798: - I think this one could be used, looks like there's no other

[jira] [Commented] (SPARK-21752) Config spark.jars.packages is ignored in SparkSession config

[ https://issues.apache.org/jira/browse/SPARK-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130078#comment-16130078 ] Saisai Shao commented on SPARK-21752: - For configurations like "spark.jars.packages" should be

[jira] [Commented] (SPARK-21714) SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again

[ https://issues.apache.org/jira/browse/SPARK-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127199#comment-16127199 ] Saisai Shao commented on SPARK-21714: - Let me take a crack on this if no one is working on it. >

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126974#comment-16126974 ] Saisai Shao commented on SPARK-21733: - Should the resolution of this JIRA be "invalid" or else?

[jira] [Commented] (SPARK-21733) ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

[ https://issues.apache.org/jira/browse/SPARK-21733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126894#comment-16126894 ] Saisai Shao commented on SPARK-21733: - Is there any issue here? > ERROR

<    1   2   3   4   5   6   7   8   9   10   >