[jira] [Commented] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340433#comment-14340433 ] Thomas Graves commented on SPARK-6050: -- Note, the problem is in determining whether

[jira] [Commented] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340431#comment-14340431 ] Thomas Graves commented on SPARK-6050: -- Ah good point. Note that 1.2 didn't always

[jira] [Commented] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340195#comment-14340195 ] Thomas Graves commented on SPARK-6050: -- Thanks for investigating this more. This is

[jira] [Commented] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340773#comment-14340773 ] Thomas Graves commented on SPARK-6050: -- Sounds like you figured it out already, I cut

[jira] [Commented] (SPARK-6050) Spark on YARN does not work --executor-cores is specified

[ https://issues.apache.org/jira/browse/SPARK-6050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340968#comment-14340968 ] Thomas Graves commented on SPARK-6050: -- Note that there is one other problem Mridul

[jira] [Commented] (SPARK-3508) annotate the Spark configs to indicate which ones are meant for the end user

[ https://issues.apache.org/jira/browse/SPARK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338457#comment-14338457 ] Thomas Graves commented on SPARK-3508: -- I wasn't suggesting renaming existing

[jira] [Commented] (SPARK-6017) Provide transparent secure communication channel on Yarn

[ https://issues.apache.org/jira/browse/SPARK-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338423#comment-14338423 ] Thomas Graves commented on SPARK-6017: -- when you say: Currently the method of

[jira] [Closed] (SPARK-3508) annotate the Spark configs to indicate which ones are meant for the end user

[ https://issues.apache.org/jira/browse/SPARK-3508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-3508. Resolution: Won't Fix annotate the Spark configs to indicate which ones are meant for the end user

[jira] [Commented] (SPARK-6017) Provide transparent secure communication channel on Yarn

[ https://issues.apache.org/jira/browse/SPARK-6017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338787#comment-14338787 ] Thomas Graves commented on SPARK-6017: -- spark.authenticate.secret is only used in

[jira] [Updated] (SPARK-5493) Support proxy users under kerberos

[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-5493: - Description: When using kerberos, services may want to use spark-submit to submit jobs as a

[jira] [Updated] (SPARK-6469) The YARN driver in yarn-client mode will not use the local directories configured for YARN

[ https://issues.apache.org/jira/browse/SPARK-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-6469: - Component/s: (was: Spark Core) YARN The YARN driver in yarn-client mode

[jira] [Commented] (SPARK-6469) The YARN driver in yarn-client mode will not use the local directories configured for YARN

[ https://issues.apache.org/jira/browse/SPARK-6469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376119#comment-14376119 ] Thomas Graves commented on SPARK-6469: -- looks good, thanks. The YARN driver in

[jira] [Resolved] (SPARK-6662) Allow variable substitution in spark.yarn.historyServer.address

[ https://issues.apache.org/jira/browse/SPARK-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-6662. -- Resolution: Fixed Fix Version/s: 1.4.0 Assignee: Cheolsoo Park Allow variable

[jira] [Assigned] (SPARK-1701) Inconsistent naming: slice or partition

[ https://issues.apache.org/jira/browse/SPARK-1701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-1701: Assignee: Thomas Graves Inconsistent naming: slice or partition

[jira] [Commented] (SPARK-7110) when use saveAsNewAPIHadoopFile, sometimes it throws Delegation Token can be issued only with kerberos or web authentication

[ https://issues.apache.org/jira/browse/SPARK-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14511007#comment-14511007 ] Thomas Graves commented on SPARK-7110: -- So with the NewApi's, the call to: val

[jira] [Updated] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-5342: - Assignee: Hari Shreedharan Allow long running Spark apps to run on secure YARN/HDFS

[jira] [Resolved] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-5342. -- Resolution: Fixed Fix Version/s: 1.4.0 Allow long running Spark apps to run on secure

[jira] [Created] (SPARK-7302) SPARK building documentation still mentions building for yarn 0.23

Thomas Graves created SPARK-7302: Summary: SPARK building documentation still mentions building for yarn 0.23 Key: SPARK-7302 URL: https://issues.apache.org/jira/browse/SPARK-7302 Project: Spark

[jira] [Resolved] (SPARK-5342) Allow long running Spark apps to run on secure YARN/HDFS

[ https://issues.apache.org/jira/browse/SPARK-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-5342. -- Resolution: Fixed Allow long running Spark apps to run on secure YARN/HDFS

[jira] [Created] (SPARK-7306) SPARK-7224 breaks build with jdk6

Thomas Graves created SPARK-7306: Summary: SPARK-7224 breaks build with jdk6 Key: SPARK-7306 URL: https://issues.apache.org/jira/browse/SPARK-7306 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-7306) SPARK-7224 broke build with jdk6

[ https://issues.apache.org/jira/browse/SPARK-7306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-7306: - Summary: SPARK-7224 broke build with jdk6 (was: SPARK-7224 breaks build with jdk6) SPARK-7224

[jira] [Created] (SPARK-7312) SPARK-6913 broke jdk6 build

Thomas Graves created SPARK-7312: Summary: SPARK-6913 broke jdk6 build Key: SPARK-7312 URL: https://issues.apache.org/jira/browse/SPARK-7312 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-6918) Secure HBase with Kerberos does not work over YARN

[ https://issues.apache.org/jira/browse/SPARK-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-6918. -- Resolution: Fixed Fix Version/s: 1.4.0 Assignee: Dean Chen Secure HBase with

[jira] [Updated] (SPARK-6869) Add pyspark archives path to PYTHONPATH

[ https://issues.apache.org/jira/browse/SPARK-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-6869: - Priority: Blocker (was: Minor) Add pyspark archives path to PYTHONPATH

[jira] [Commented] (SPARK-7110) when use saveAsNewAPIHadoopFile, sometimes it throws Delegation Token can be issued only with kerberos or web authentication

[ https://issues.apache.org/jira/browse/SPARK-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534494#comment-14534494 ] Thomas Graves commented on SPARK-7110: -- [~gu chi] is there some of the stack trace

[jira] [Created] (SPARK-7485) Remove python artifacts from the assembly jar

Thomas Graves created SPARK-7485: Summary: Remove python artifacts from the assembly jar Key: SPARK-7485 URL: https://issues.apache.org/jira/browse/SPARK-7485 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-1920) Spark JAR compiled with Java 7 leads to PySpark not working in YARN

[ https://issues.apache.org/jira/browse/SPARK-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-1920. -- Resolution: Duplicate Spark JAR compiled with Java 7 leads to PySpark not working in YARN

[jira] [Updated] (SPARK-6869) Add pyspark archives path to PYTHONPATH

[ https://issues.apache.org/jira/browse/SPARK-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-6869: - Assignee: Lianhui Wang Add pyspark archives path to PYTHONPATH

[jira] [Resolved] (SPARK-6869) Add pyspark archives path to PYTHONPATH

[ https://issues.apache.org/jira/browse/SPARK-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-6869. -- Resolution: Fixed Fix Version/s: 1.4.0 Add pyspark archives path to PYTHONPATH

[jira] [Updated] (SPARK-6869) Add pyspark archives path to PYTHONPATH

[ https://issues.apache.org/jira/browse/SPARK-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-6869: - Target Version/s: 1.4.0 Add pyspark archives path to PYTHONPATH

[jira] [Updated] (SPARK-6869) Add pyspark archives path to PYTHONPATH

[ https://issues.apache.org/jira/browse/SPARK-6869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-6869: - Issue Type: Bug (was: Improvement) Add pyspark archives path to PYTHONPATH

[jira] [Updated] (SPARK-7110) when use saveAsNewAPIHadoopFile, sometimes it throws Delegation Token can be issued only with kerberos or web authentication

[ https://issues.apache.org/jira/browse/SPARK-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-7110: - Assignee: Sean Owen when use saveAsNewAPIHadoopFile, sometimes it throws Delegation Token can be

[jira] [Updated] (SPARK-7110) when use saveAsNewAPIHadoopFile, sometimes it throws Delegation Token can be issued only with kerberos or web authentication

[ https://issues.apache.org/jira/browse/SPARK-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-7110: - Assignee: Sean Owen when use saveAsNewAPIHadoopFile, sometimes it throws Delegation Token can be

[jira] [Updated] (SPARK-7110) when use saveAsNewAPIHadoopFile, sometimes it throws Delegation Token can be issued only with kerberos or web authentication

[ https://issues.apache.org/jira/browse/SPARK-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-7110: - Assignee: (was: Sean Owen) when use saveAsNewAPIHadoopFile, sometimes it throws Delegation

[jira] [Commented] (SPARK-7110) when use saveAsNewAPIHadoopFile, sometimes it throws Delegation Token can be issued only with kerberos or web authentication

[ https://issues.apache.org/jira/browse/SPARK-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537964#comment-14537964 ] Thomas Graves commented on SPARK-7110: -- [~gu chi] Any chance you can provide the

[jira] [Commented] (SPARK-7110) when use saveAsNewAPIHadoopFile, sometimes it throws Delegation Token can be issued only with kerberos or web authentication

[ https://issues.apache.org/jira/browse/SPARK-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14548248#comment-14548248 ] Thomas Graves commented on SPARK-7110: -- Are you using spark1.1.0 as reported in the

[jira] [Created] (SPARK-6904) SparkSql - HiveContext - optimize reading partition data from metastore

Thomas Graves created SPARK-6904: Summary: SparkSql - HiveContext - optimize reading partition data from metastore Key: SPARK-6904 URL: https://issues.apache.org/jira/browse/SPARK-6904 Project: Spark

[jira] [Resolved] (SPARK-6207) YARN secure cluster mode doesn't obtain a hive-metastore token

[ https://issues.apache.org/jira/browse/SPARK-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-6207. -- Resolution: Fixed Fix Version/s: 1.4.0 YARN secure cluster mode doesn't obtain a

[jira] [Resolved] (SPARK-2669) Hadoop configuration is not localised when submitting job in yarn-cluster mode

[ https://issues.apache.org/jira/browse/SPARK-2669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-2669. -- Resolution: Fixed Fix Version/s: 1.4.0 Assignee: Marcelo Vanzin

[jira] [Resolved] (SPARK-3591) Provide fire and forget option for YARN cluster mode

[ https://issues.apache.org/jira/browse/SPARK-3591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-3591. -- Resolution: Fixed Fix Version/s: 1.4.0 Assignee: Tao Wang Target

[jira] [Created] (SPARK-6751) Spark History Server support multiple application attempts

Thomas Graves created SPARK-6751: Summary: Spark History Server support multiple application attempts Key: SPARK-6751 URL: https://issues.apache.org/jira/browse/SPARK-6751 Project: Spark

[jira] [Commented] (SPARK-6506) python support yarn cluster mode requires SPARK_HOME to be set

[ https://issues.apache.org/jira/browse/SPARK-6506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392978#comment-14392978 ] Thomas Graves commented on SPARK-6506: -- No it was built with maven and the pyspark

[jira] [Resolved] (SPARK-6751) Spark History Server support multiple application attempts

[ https://issues.apache.org/jira/browse/SPARK-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-6751. -- Resolution: Duplicate Dup of SPARK-4705 Spark History Server support multiple application

[jira] [Resolved] (SPARK-7533) Decrease spacing between AM-RM heartbeats.

[ https://issues.apache.org/jira/browse/SPARK-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-7533. -- Resolution: Fixed Fix Version/s: 1.5.0 Assignee: Zoltán Zvara Decrease spacing

[jira] [Created] (SPARK-8569) Documentation for SPARK_HISTORY_OPTS unclear

Thomas Graves created SPARK-8569: Summary: Documentation for SPARK_HISTORY_OPTS unclear Key: SPARK-8569 URL: https://issues.apache.org/jira/browse/SPARK-8569 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-8574) org/apache/spark/unsafe doesn't honor the java source/target versions

Thomas Graves created SPARK-8574: Summary: org/apache/spark/unsafe doesn't honor the java source/target versions Key: SPARK-8574 URL: https://issues.apache.org/jira/browse/SPARK-8574 Project: Spark

[jira] [Resolved] (SPARK-8574) org/apache/spark/unsafe doesn't honor the java source/target versions

[ https://issues.apache.org/jira/browse/SPARK-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-8574. -- Resolution: Fixed Fix Version/s: 1.4.1 org/apache/spark/unsafe doesn't honor the java

[jira] [Assigned] (SPARK-8574) org/apache/spark/unsafe doesn't honor the java source/target versions

[ https://issues.apache.org/jira/browse/SPARK-8574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-8574: Assignee: Thomas Graves org/apache/spark/unsafe doesn't honor the java source/target

[jira] [Closed] (SPARK-8594) History Server doesn't show complete application when one attempt inprogress

[ https://issues.apache.org/jira/browse/SPARK-8594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-8594. Resolution: Duplicate accidentally filed same jira twice History Server doesn't show complete

[jira] [Created] (SPARK-8617) Handle history files better

Thomas Graves created SPARK-8617: Summary: Handle history files better Key: SPARK-8617 URL: https://issues.apache.org/jira/browse/SPARK-8617 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-8593) History Server doesn't show complete application when one attempt inprogress

Thomas Graves created SPARK-8593: Summary: History Server doesn't show complete application when one attempt inprogress Key: SPARK-8593 URL: https://issues.apache.org/jira/browse/SPARK-8593 Project:

[jira] [Created] (SPARK-8594) History Server doesn't show complete application when one attempt inprogress

Thomas Graves created SPARK-8594: Summary: History Server doesn't show complete application when one attempt inprogress Key: SPARK-8594 URL: https://issues.apache.org/jira/browse/SPARK-8594 Project:

[jira] [Commented] (SPARK-1476) 2GB limit in spark for blocks

[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603220#comment-14603220 ] Thomas Graves commented on SPARK-1476: -- we have a lot of jira about the 2G limit.

[jira] [Closed] (SPARK-1476) 2GB limit in spark for blocks

[ https://issues.apache.org/jira/browse/SPARK-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-1476. Resolution: Duplicate 2GB limit in spark for blocks -

[jira] [Commented] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603223#comment-14603223 ] Thomas Graves commented on SPARK-1391: -- duping this to SPARK-6235. If something is

[jira] [Closed] (SPARK-1391) BlockManager cannot transfer blocks larger than 2G in size

[ https://issues.apache.org/jira/browse/SPARK-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-1391. Resolution: Duplicate BlockManager cannot transfer blocks larger than 2G in size

[jira] [Updated] (SPARK-7524) add configs for keytab and principal, move originals to internal

[ https://issues.apache.org/jira/browse/SPARK-7524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-7524: - Assignee: Tao Wang add configs for keytab and principal, move originals to internal

[jira] [Closed] (SPARK-7524) add configs for keytab and principal, move originals to internal

[ https://issues.apache.org/jira/browse/SPARK-7524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-7524. Resolution: Fixed Fix Version/s: 1.5.0 add configs for keytab and principal, move originals

[jira] [Updated] (SPARK-7846) Use different way to pass spark.yarn.keytab and spark.yarn.principal in different modes

[ https://issues.apache.org/jira/browse/SPARK-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-7846: - Assignee: Tao Wang Use different way to pass spark.yarn.keytab and spark.yarn.principal in

[jira] [Commented] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608280#comment-14608280 ] Thomas Graves commented on SPARK-6735: -- Pull request was up but didn't have time to

[jira] [Commented] (SPARK-6951) History server slow startup if the event log directory is large

[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14608287#comment-14608287 ] Thomas Graves commented on SPARK-6951: -- This actually happens more then just at start

[jira] [Commented] (SPARK-7705) Cleanup of .sparkStaging directory fails if application is killed

[ https://issues.apache.org/jira/browse/SPARK-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559114#comment-14559114 ] Thomas Graves commented on SPARK-7705: -- YARN will retry an application for you if it

[jira] [Resolved] (SPARK-9439) ExternalShuffleService should be robust to NodeManager restarts in yarn

[ https://issues.apache.org/jira/browse/SPARK-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-9439. -- Resolution: Fixed Assignee: Imran Rashid Fix Version/s: 1.6.0

[jira] [Commented] (SPARK-3684) Can't configure local dirs in Yarn mode

[ https://issues.apache.org/jira/browse/SPARK-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644359#comment-14644359 ] Thomas Graves commented on SPARK-3684: -- What is thie use case for this? YARN

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644364#comment-14644364 ] Thomas Graves commented on SPARK-4352: -- Sorry I haven't followed this. To be clear

[jira] [Commented] (SPARK-4352) Incorporate locality preferences in dynamic allocation requests

[ https://issues.apache.org/jira/browse/SPARK-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14645988#comment-14645988 ] Thomas Graves commented on SPARK-4352: -- thanks, its definitely better to have it

[jira] [Created] (SPARK-9641) spark.shuffle.service.port is not documented

Thomas Graves created SPARK-9641: Summary: spark.shuffle.service.port is not documented Key: SPARK-9641 URL: https://issues.apache.org/jira/browse/SPARK-9641 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-9641) spark.shuffle.service.port is not documented

[ https://issues.apache.org/jira/browse/SPARK-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658712#comment-14658712 ] Thomas Graves commented on SPARK-9641: -- go ahead if you have time, I won't be able to

[jira] [Commented] (SPARK-9823) Spark application hangs when using dynamic allocations and executor fails connect to shuffle service

[ https://issues.apache.org/jira/browse/SPARK-9823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14682025#comment-14682025 ] Thomas Graves commented on SPARK-9823: -- Sorry it turned out I had some really high

[jira] [Closed] (SPARK-9823) Spark application hangs when using dynamic allocations and executor fails connect to shuffle service

[ https://issues.apache.org/jira/browse/SPARK-9823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-9823. Resolution: Invalid Spark application hangs when using dynamic allocations and executor fails

[jira] [Created] (SPARK-9823) Spark application hangs when using dynamic allocations and executor fails connect to shuffle service

Thomas Graves created SPARK-9823: Summary: Spark application hangs when using dynamic allocations and executor fails connect to shuffle service Key: SPARK-9823 URL:

[jira] [Created] (SPARK-9701) allow not automatically using HiveContext with spark-shell when hive support built in

Thomas Graves created SPARK-9701: Summary: allow not automatically using HiveContext with spark-shell when hive support built in Key: SPARK-9701 URL: https://issues.apache.org/jira/browse/SPARK-9701

[jira] [Commented] (SPARK-9019) spark-submit fails on yarn with kerberos enabled

[ https://issues.apache.org/jira/browse/SPARK-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680109#comment-14680109 ] Thomas Graves commented on SPARK-9019: -- We forgot to close out the jira. This was

[jira] [Closed] (SPARK-9019) spark-submit fails on yarn with kerberos enabled

[ https://issues.apache.org/jira/browse/SPARK-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-9019. Resolution: Won't Fix spark-submit fails on yarn with kerberos enabled

[jira] [Reopened] (SPARK-8405) Show executor logs on Web UI when Yarn log aggregation is enabled

[ https://issues.apache.org/jira/browse/SPARK-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-8405: -- Assignee: Carson Wang Show executor logs on Web UI when Yarn log aggregation is enabled

[jira] [Resolved] (SPARK-8405) Show executor logs on Web UI when Yarn log aggregation is enabled

[ https://issues.apache.org/jira/browse/SPARK-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-8405. -- Resolution: Fixed Fix Version/s: 1.4.2 Show executor logs on Web UI when Yarn log

[jira] [Resolved] (SPARK-8851) in Yarn client mode, Client.scala does not login even when credentials are specified

[ https://issues.apache.org/jira/browse/SPARK-8851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-8851. -- Resolution: Fixed Assignee: Hari Shreedharan Fix Version/s: 1.5.0 in Yarn

[jira] [Commented] (SPARK-9019) spark-submit fails on yarn with kerberos enabled

[ https://issues.apache.org/jira/browse/SPARK-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14633623#comment-14633623 ] Thomas Graves commented on SPARK-9019: -- The RMdelegationtoken is only needed if the

[jira] [Commented] (SPARK-6951) History server slow startup if the event log directory is large

[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711292#comment-14711292 ] Thomas Graves commented on SPARK-6951: -- Sorry I was wrong.. I just went and tested it

[jira] [Commented] (SPARK-4223) Support * (meaning all users) as part of the acls

[ https://issues.apache.org/jira/browse/SPARK-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14711384#comment-14711384 ] Thomas Graves commented on SPARK-4223: -- [~srowen] [~rxin] do one of you have

[jira] [Updated] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups

[ https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11316: -- Description: So I haven't fully debugged this yet but reporting what I'm seeing and think

[jira] [Commented] (SPARK-6951) History server slow startup if the event log directory is large

[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976364#comment-14976364 ] Thomas Graves commented on SPARK-6951: -- Unfortunately for us I don't think the current timeline

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979204#comment-14979204 ] Thomas Graves commented on SPARK-2089: -- maybe I'm missing something, why are you saying dynamic

[jira] [Updated] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups

[ https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11316: -- Priority: Critical (was: Major) > isEmpty before coalesce seems to cause huge performance

[jira] [Created] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups

Thomas Graves created SPARK-11316: - Summary: isEmpty before coalesce seems to cause huge performance issue in setupGroups Key: SPARK-11316 URL: https://issues.apache.org/jira/browse/SPARK-11316

[jira] [Commented] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups

[ https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974236#comment-14974236 ] Thomas Graves commented on SPARK-11316: --- Note I"m wondering if since the isEmpty call is doing a

[jira] [Commented] (SPARK-11154) make specificaition spark.yarn.executor.memoryOverhead consistent with typical JVM options

[ https://issues.apache.org/jira/browse/SPARK-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000426#comment-15000426 ] Thomas Graves commented on SPARK-11154: --- It seems unnecessary to me to add new configs just to

[jira] [Commented] (SPARK-11154) make specificaition spark.yarn.executor.memoryOverhead consistent with typical JVM options

[ https://issues.apache.org/jira/browse/SPARK-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000438#comment-15000438 ] Thomas Graves commented on SPARK-11154: --- sorry actually thinking about this more I'm torn. New

[jira] [Created] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

Thomas Graves created SPARK-11701: - Summary: YARN - dynamic allocation and speculation active task accounting wrong Key: SPARK-11701 URL: https://issues.apache.org/jira/browse/SPARK-11701 Project:

[jira] [Commented] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

[ https://issues.apache.org/jira/browse/SPARK-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002505#comment-15002505 ] Thomas Graves commented on SPARK-11701: --- I'm not exactly sure if this is same issue but trying this

[jira] [Created] (SPARK-11555) spark on yarn spark-class --num-workers allows 0 and shouldn't

Thomas Graves created SPARK-11555: - Summary: spark on yarn spark-class --num-workers allows 0 and shouldn't Key: SPARK-11555 URL: https://issues.apache.org/jira/browse/SPARK-11555 Project: Spark

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers allows 0 and shouldn't

[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993741#comment-14993741 ] Thomas Graves commented on SPARK-11555: --- Note I think this was broken with change for SPARK-10790

[jira] [Commented] (SPARK-10911) Executors should System.exit on clean shutdown

[ https://issues.apache.org/jira/browse/SPARK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993743#comment-14993743 ] Thomas Graves commented on SPARK-10911: --- You could say its a user problem, but its also a cluster

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers allows 0 and shouldn't

[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993750#comment-14993750 ] Thomas Graves commented on SPARK-11555: --- I should probably rephrase because after looking a bit,

[jira] [Updated] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11555: -- Summary: spark on yarn spark-class --num-workers doesn't work (was: spark on yarn spark-class

[jira] [Updated] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11555: -- Description: Using the old spark-class and --num-workers interface, --num-workers parameter

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993765#comment-14993765 ] Thomas Graves commented on SPARK-11555: --- No I'm talking about --num-workers, see the usage in the

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993775#comment-14993775 ] Thomas Graves commented on SPARK-11555: --- Note that it is broken with --num-executors also when

[jira] [Assigned] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-11555: - Assignee: Thomas Graves > spark on yarn spark-class --num-workers doesn't work >

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14993815#comment-14993815 ] Thomas Graves commented on SPARK-11555: --- To fix this we can simply pass in an optional numExecutors

<    1   2   3   4   5   6   7   8   9   10   >