[jira] [Commented] (SPARK-5311) EventLoggingListener throws exception if log directory does not exist

[ https://issues.apache.org/jira/browse/SPARK-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379375#comment-15379375 ] Thomas Graves commented on SPARK-5311: -- We should not be creating the event log dir b

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379361#comment-15379361 ] Thomas Graves commented on SPARK-8425: -- Slightly different scenario since its not wit

[jira] [Resolved] (SPARK-16505) YARN shuffle service should throw errors when it fails to start

[ https://issues.apache.org/jira/browse/SPARK-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16505. --- Resolution: Fixed Fix Version/s: 2.1.0 > YARN shuffle service should throw errors when

[jira] [Updated] (SPARK-16505) YARN shuffle service should throw errors when it fails to start

[ https://issues.apache.org/jira/browse/SPARK-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16505: -- Assignee: Marcelo Vanzin > YARN shuffle service should throw errors when it fails to start > --

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376913#comment-15376913 ] Thomas Graves commented on SPARK-14963: --- committed fix in https://github.com/apache

[jira] [Updated] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16435: -- Assignee: Saisai Shao > Behavior changes if initialExecutor is less than minExecutor for dynami

[jira] [Resolved] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16435. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > Behavior changes if

[jira] [Commented] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370836#comment-15370836 ] Thomas Graves commented on SPARK-16435: --- Yeah I agree,I'm not as worried about this

[jira] [Commented] (SPARK-16451) Spark-shell / pyspark should finish gracefully when "SaslException: GSS initiate failed" is hit

[ https://issues.apache.org/jira/browse/SPARK-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370832#comment-15370832 ] Thomas Graves commented on SPARK-16451: --- I think there are a bunch of cases where i

[jira] [Commented] (SPARK-16455) Add a new hook in CoarseGrainedSchedulerBackend in order to stop scheduling new tasks when cluster is restarting

[ https://issues.apache.org/jira/browse/SPARK-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370828#comment-15370828 ] Thomas Graves commented on SPARK-16455: --- | we are implementing a new mechanism whic

[jira] [Commented] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370762#comment-15370762 ] Thomas Graves commented on SPARK-15703: --- sorry for my delay in responding, we repro

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367683#comment-15367683 ] Thomas Graves commented on SPARK-16422: --- it looks like there is generally about 6 m

[jira] [Updated] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16422: -- Assignee: Sean Owen > maven 3.3.3 missing from mirror, breaks older builds > --

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366829#comment-15366829 ] Thomas Graves commented on SPARK-16422: --- do we know why maven 3.3.3 got removed? I

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366582#comment-15366582 ] Thomas Graves commented on SPARK-16422: --- I'll pop this discussions up on dev list a

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366573#comment-15366573 ] Thomas Graves commented on SPARK-16422: --- maybe I missed some discussions but I thou

[jira] [Commented] (SPARK-16399) Set PYSPARK_PYTHON to point to "python" instead of "python2.7"

[ https://issues.apache.org/jira/browse/SPARK-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366567#comment-15366567 ] Thomas Graves commented on SPARK-16399: --- ok, I guess any other checks/error message

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366549#comment-15366549 ] Thomas Graves commented on SPARK-16422: --- Oh I didn't see it go by because there is

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366544#comment-15366544 ] Thomas Graves commented on SPARK-16422: --- thanks, didn't see that go by. > maven 3

[jira] [Created] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

Thomas Graves created SPARK-16422: - Summary: maven 3.3.3 missing from mirror, breaks older builds Key: SPARK-16422 URL: https://issues.apache.org/jira/browse/SPARK-16422 Project: Spark Issue

[jira] [Commented] (SPARK-16298) spark.yarn.principal not working

[ https://issues.apache.org/jira/browse/SPARK-16298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366310#comment-15366310 ] Thomas Graves commented on SPARK-16298: --- kinit as the user in the keytab before sub

[jira] [Commented] (SPARK-16399) Set PYSPARK_PYTHON to point to "python" instead of "python2.7"

[ https://issues.apache.org/jira/browse/SPARK-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366271#comment-15366271 ] Thomas Graves commented on SPARK-16399: --- so what is the behavior if I run this on o

[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366251#comment-15366251 ] Thomas Graves commented on SPARK-16265: --- Have you tried shipping your jre as gzip f

[jira] [Comment Edited] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366251#comment-15366251 ] Thomas Graves edited comment on SPARK-16265 at 7/7/16 3:10 PM:

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364834#comment-15364834 ] Thomas Graves commented on SPARK-8425: -- Added some questions to the design doc > Add

[jira] [Commented] (SPARK-16382) YARN - Dynamic allocation with spark.executor.instances should increase max executors.

[ https://issues.apache.org/jira/browse/SPARK-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364363#comment-15364363 ] Thomas Graves commented on SPARK-16382: --- I think we should fail and complain and I

[jira] [Resolved] (SPARK-15990) Support rolling log aggregation for Spark running on YARN

[ https://issues.apache.org/jira/browse/SPARK-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15990. --- Resolution: Fixed Fix Version/s: 2.1.0 > Support rolling log aggregation for Spark run

[jira] [Updated] (SPARK-15990) Support rolling log aggregation for Spark running on YARN

[ https://issues.apache.org/jira/browse/SPARK-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15990: -- Assignee: Saisai Shao > Support rolling log aggregation for Spark running on YARN > ---

[jira] [Commented] (SPARK-15955) Failed Spark application returns with exitcode equals to zero

[ https://issues.apache.org/jira/browse/SPARK-15955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15348220#comment-15348220 ] Thomas Graves commented on SPARK-15955: --- there are some corner cases in spark 1.x t

[jira] [Updated] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15725: -- Fix Version/s: (was: 2.0.0) 2.0.1 > Dynamic allocation hangs YARN app wh

[jira] [Resolved] (SPARK-13723) YARN - Change behavior of --num-executors when spark.dynamicAllocation.enabled true

[ https://issues.apache.org/jira/browse/SPARK-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13723. --- Resolution: Fixed Fix Version/s: 2.0.0 > YARN - Change behavior of --num-executors whe

[jira] [Updated] (SPARK-13723) YARN - Change behavior of --num-executors when spark.dynamicAllocation.enabled true

[ https://issues.apache.org/jira/browse/SPARK-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-13723: -- Assignee: Ryan Blue > YARN - Change behavior of --num-executors when > spark.dynamicAllocation

[jira] [Resolved] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15725. --- Resolution: Fixed Fix Version/s: 2.0.0 > Dynamic allocation hangs YARN app when execut

[jira] [Updated] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15725: -- Assignee: Ryan Blue > Dynamic allocation hangs YARN app when executors time out > -

[jira] [Updated] (SPARK-16138) YarnAllocator tries to cancel executor requests when we have none

[ https://issues.apache.org/jira/browse/SPARK-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16138: -- Assignee: Peter Ableda (was: Apache Spark) > YarnAllocator tries to cancel executor requests w

[jira] [Resolved] (SPARK-16138) YarnAllocator tries to cancel executor requests when we have none

[ https://issues.apache.org/jira/browse/SPARK-16138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16138. --- Resolution: Fixed Fix Version/s: 2.1.0 > YarnAllocator tries to cancel executor reques

[jira] [Updated] (SPARK-16080) Config archive not properly added to YARN classpath

[ https://issues.apache.org/jira/browse/SPARK-16080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16080: -- Assignee: Marcelo Vanzin > Config archive not properly added to YARN classpath > --

[jira] [Resolved] (SPARK-16080) Config archive not properly added to YARN classpath

[ https://issues.apache.org/jira/browse/SPARK-16080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16080. --- Resolution: Fixed Fix Version/s: 2.0.0 > Config archive not properly added to YARN cla

[jira] [Commented] (SPARK-16095) Yarn cluster mode should return consistent result for command line and SparkLauncher

[ https://issues.apache.org/jira/browse/SPARK-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341903#comment-15341903 ] Thomas Graves commented on SPARK-16095: --- FINISHED does not mean success, finished m

[jira] [Commented] (SPARK-15941) Netty RPC implementation ignores the executor bind address

[ https://issues.apache.org/jira/browse/SPARK-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339501#comment-15339501 ] Thomas Graves commented on SPARK-15941: --- which version of 1.6 were you running ther

[jira] [Commented] (SPARK-15941) Netty RPC implementation ignores the executor bind address

[ https://issues.apache.org/jira/browse/SPARK-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339450#comment-15339450 ] Thomas Graves commented on SPARK-15941: --- can you give some more details about your

[jira] [Updated] (SPARK-15941) Netty RPC implementation ignores the executor bind address

[ https://issues.apache.org/jira/browse/SPARK-15941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15941: -- Affects Version/s: 1.6.1 > Netty RPC implementation ignores the executor bind address > ---

[jira] [Resolved] (SPARK-16018) Shade netty for shuffle to work on YARN

[ https://issues.apache.org/jira/browse/SPARK-16018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16018. --- Resolution: Fixed > Shade netty for shuffle to work on YARN > ---

[jira] [Updated] (SPARK-16018) Shade netty for shuffle to work on YARN

[ https://issues.apache.org/jira/browse/SPARK-16018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16018: -- Assignee: Dhruve Ashar > Shade netty for shuffle to work on YARN >

[jira] [Commented] (SPARK-16018) Shade netty for shuffle to work on YARN

[ https://issues.apache.org/jira/browse/SPARK-16018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336326#comment-15336326 ] Thomas Graves commented on SPARK-16018: --- Note we are seeing this on hadoop 2.7. I

[jira] [Commented] (SPARK-9103) Tracking spark's memory usage

[ https://issues.apache.org/jira/browse/SPARK-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15333988#comment-15333988 ] Thomas Graves commented on SPARK-9103: -- [~srowen] I assume this hit the to old mark b

[jira] [Commented] (SPARK-15955) Failed Spark application returns with exitcode equals to zero

[ https://issues.apache.org/jira/browse/SPARK-15955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332276#comment-15332276 ] Thomas Graves commented on SPARK-15955: --- what master and deploy mode are you using?

[jira] [Updated] (SPARK-15046) When running hive-thriftserver with yarn on a secure cluster the workers fail with java.lang.NumberFormatException

[ https://issues.apache.org/jira/browse/SPARK-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15046: -- Assignee: Marcelo Vanzin > When running hive-thriftserver with yarn on a secure cluster the wor

[jira] [Resolved] (SPARK-15046) When running hive-thriftserver with yarn on a secure cluster the workers fail with java.lang.NumberFormatException

[ https://issues.apache.org/jira/browse/SPARK-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15046. --- Resolution: Fixed Fix Version/s: 2.0.0 > When running hive-thriftserver with yarn on a

[jira] [Commented] (SPARK-15923) Spark Application rest api returns "no such app: "

[ https://issues.apache.org/jira/browse/SPARK-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15328181#comment-15328181 ] Thomas Graves commented on SPARK-15923: --- can you give some more details? Did you h

[jira] [Commented] (SPARK-15851) Spark 2.0 does not compile in Windows 7

[ https://issues.apache.org/jira/browse/SPARK-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324452#comment-15324452 ] Thomas Graves commented on SPARK-15851: --- Can we also please document that windows i

[jira] [Commented] (HADOOP-13184) Add "Apache" to Hadoop project logo

[ https://issues.apache.org/jira/browse/HADOOP-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318477#comment-15318477 ] Thomas Graves commented on HADOOP-13184: my vote would be option 4. > Add "Apac

[jira] [Resolved] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-14331. --- Resolution: Not A Problem can't reproduce this anymore. > Exceptions saving to parquetFile a

[jira] [Commented] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310800#comment-15310800 ] Thomas Graves commented on SPARK-15703: --- Note that the history UI also has the same

[jira] [Created] (SPARK-15708) Tasks table in Detailed Stage page shows ip instead of hostname under Executor ID/Host

Thomas Graves created SPARK-15708: - Summary: Tasks table in Detailed Stage page shows ip instead of hostname under Executor ID/Host Key: SPARK-15708 URL: https://issues.apache.org/jira/browse/SPARK-15708

[jira] [Updated] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Summary: Spark UI doesn't show all tasks as completed when it should (was: Spark UI doesn't sh

[jira] [Commented] (SPARK-15700) Spark 2.0 dataframes using more driver memory (reading/writing parquet)

[ https://issues.apache.org/jira/browse/SPARK-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310657#comment-15310657 ] Thomas Graves commented on SPARK-15700: --- It looks like executors are also requiring

[jira] [Updated] (SPARK-15700) Spark 2.0 dataframes using more memory (reading/writing parquet)

[ https://issues.apache.org/jira/browse/SPARK-15700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15700: -- Summary: Spark 2.0 dataframes using more memory (reading/writing parquet) (was: Spark 2.0 data

[jira] [Updated] (SPARK-15703) Spark UI doesn't show all tasks as completed when they are

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Attachment: Screen Shot 2016-06-01 at 11.23.48 AM.png Screen Shot 2016-06-01 at

[jira] [Created] (SPARK-15703) Spark UI doesn't show all tasks as completed when they are

Thomas Graves created SPARK-15703: - Summary: Spark UI doesn't show all tasks as completed when they are Key: SPARK-15703 URL: https://issues.apache.org/jira/browse/SPARK-15703 Project: Spark

[jira] [Commented] (SPARK-15671) performance regression CoalesceRDD large # partitions

[ https://issues.apache.org/jira/browse/SPARK-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15310487#comment-15310487 ] Thomas Graves commented on SPARK-15671: --- Note the performance impact is in the 10's

[jira] [Created] (SPARK-15700) Spark 2.0 dataframes using more driver memory (reading/writing parquet)

Thomas Graves created SPARK-15700: - Summary: Spark 2.0 dataframes using more driver memory (reading/writing parquet) Key: SPARK-15700 URL: https://issues.apache.org/jira/browse/SPARK-15700 Project: Sp

[jira] [Resolved] (SPARK-15683) spark sql local FS spark.sql.warehouse.dir throws on YARN

[ https://issues.apache.org/jira/browse/SPARK-15683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15683. --- Resolution: Duplicate > spark sql local FS spark.sql.warehouse.dir throws on YARN > -

[jira] [Reopened] (SPARK-15671) performance regression CoalesceRDD large # partitions

[ https://issues.apache.org/jira/browse/SPARK-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-15671: --- > performance regression CoalesceRDD large # partitions > ---

[jira] [Resolved] (SPARK-15671) performance regression CoalesceRDD large # partitions

[ https://issues.apache.org/jira/browse/SPARK-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15671. --- Resolution: Duplicate dup of SPARK-15659 > performance regression CoalesceRDD large # partit

[jira] [Commented] (SPARK-15683) spark sql local FS spark.sql.warehouse.dir throws on YARN

[ https://issues.apache.org/jira/browse/SPARK-15683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308705#comment-15308705 ] Thomas Graves commented on SPARK-15683: --- Note InMemoryCatalog.scala is getting the

[jira] [Updated] (SPARK-15683) spark sql local FS spark.sql.warehouse.dir throws on YARN

[ https://issues.apache.org/jira/browse/SPARK-15683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15683: -- Description: I'm trying to use dataframes with spark 2.0. It was built with hive but when I t

[jira] [Created] (SPARK-15683) spark sql local FS spark.sql.warehouse.dir throws on YARN

Thomas Graves created SPARK-15683: - Summary: spark sql local FS spark.sql.warehouse.dir throws on YARN Key: SPARK-15683 URL: https://issues.apache.org/jira/browse/SPARK-15683 Project: Spark I

[jira] [Updated] (SPARK-15671) performance regression CoalesceRDD large # partitions

[ https://issues.apache.org/jira/browse/SPARK-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15671: -- Target Version/s: 2.0.0 > performance regression CoalesceRDD large # partitions > -

[jira] [Commented] (SPARK-15671) performance regression CoalesceRDD large # partitions

[ https://issues.apache.org/jira/browse/SPARK-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307934#comment-15307934 ] Thomas Graves commented on SPARK-15671: --- I should have a patch up for this shortly.

[jira] [Created] (SPARK-15671) performance regression CoalesceRDD large # partitions

Thomas Graves created SPARK-15671: - Summary: performance regression CoalesceRDD large # partitions Key: SPARK-15671 URL: https://issues.apache.org/jira/browse/SPARK-15671 Project: Spark Issue

[jira] [Commented] (SPARK-13148) document zero-keytab Oozie application launch; add diagnostics

[ https://issues.apache.org/jira/browse/SPARK-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302687#comment-15302687 ] Thomas Graves commented on SPARK-13148: --- note we changed this to just document what

[jira] [Resolved] (SPARK-13148) document zero-keytab Oozie application launch; add diagnostics

[ https://issues.apache.org/jira/browse/SPARK-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13148. --- Resolution: Fixed Fix Version/s: 2.0.0 > document zero-keytab Oozie application launch

[jira] [Updated] (SPARK-13148) document zero-keytab Oozie application launch; add diagnostics

[ https://issues.apache.org/jira/browse/SPARK-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-13148: -- Assignee: Steve Loughran > document zero-keytab Oozie application launch; add diagnostics > ---

[jira] [Updated] (SPARK-13148) document zero-keytab Oozie application launch; add diagnostics

[ https://issues.apache.org/jira/browse/SPARK-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-13148: -- Summary: document zero-keytab Oozie application launch; add diagnostics (was: support zero-key

[jira] [Commented] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296705#comment-15296705 ] Thomas Graves commented on SPARK-14331: --- that might be https://github.com/apache/sp

[jira] [Updated] (SPARK-14279) Improve the spark build to pick the version information from the pom file and add git commit information

[ https://issues.apache.org/jira/browse/SPARK-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14279: -- Assignee: (was: Sanket Reddy) > Improve the spark build to pick the version information fro

[jira] [Commented] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296440#comment-15296440 ] Thomas Graves commented on SPARK-14331: --- Note I was running spark on yarn. > Excep

[jira] [Commented] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296416#comment-15296416 ] Thomas Graves commented on SPARK-14331: --- I was trying to reproduce this to get you

[jira] [Created] (SPARK-15410) spark-submit --help throws exception

Thomas Graves created SPARK-15410: - Summary: spark-submit --help throws exception Key: SPARK-15410 URL: https://issues.apache.org/jira/browse/SPARK-15410 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-15405) YARN uploading the same __spark_conf__.zip twice

[ https://issues.apache.org/jira/browse/SPARK-15405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291180#comment-15291180 ] Thomas Graves commented on SPARK-15405: --- [~vanzin] I missed this when I did the re

[jira] [Created] (SPARK-15405) YARN uploading the same __spark_conf__.zip twice

Thomas Graves created SPARK-15405: - Summary: YARN uploading the same __spark_conf__.zip twice Key: SPARK-15405 URL: https://issues.apache.org/jira/browse/SPARK-15405 Project: Spark Issue Type

[jira] [Commented] (SPARK-4924) Factor out code to launch Spark applications into a separate library

[ https://issues.apache.org/jira/browse/SPARK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284483#comment-15284483 ] Thomas Graves commented on SPARK-4924: -- [~javadba] If you have ideas on improving th

[jira] [Resolved] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-14963. --- Resolution: Fixed Fix Version/s: 2.1.0 > YarnShuffleService should use YARN getRecover

[jira] [Updated] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14963: -- Assignee: Saisai Shao > YarnShuffleService should use YARN getRecoveryPath() for leveldb locati

[jira] [Created] (SPARK-15178) Remove LazyFileRegion

Thomas Graves created SPARK-15178: - Summary: Remove LazyFileRegion Key: SPARK-15178 URL: https://issues.apache.org/jira/browse/SPARK-15178 Project: Spark Issue Type: Improvement Com

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272466#comment-15272466 ] Thomas Graves commented on SPARK-14963: --- [~jerryshao] I think the other pr is being

[jira] [Updated] (SPARK-15121) Improve logging of external shuffle handler

[ https://issues.apache.org/jira/browse/SPARK-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15121: -- Assignee: (was: Thomas Graves) > Improve logging of external shuffle handler >

[jira] [Created] (SPARK-15121) Improve logging of external shuffle handler

Thomas Graves created SPARK-15121: - Summary: Improve logging of external shuffle handler Key: SPARK-15121 URL: https://issues.apache.org/jira/browse/SPARK-15121 Project: Spark Issue Type: Imp

[jira] [Resolved] (SPARK-4224) Support group acls

[ https://issues.apache.org/jira/browse/SPARK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-4224. -- Resolution: Fixed Fix Version/s: 2.0.0 > Support group acls > -- > >

[jira] [Updated] (SPARK-4224) Support group acls

[ https://issues.apache.org/jira/browse/SPARK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-4224: - Assignee: Dhruve Ashar > Support group acls > -- > > Key: SPARK-42

[jira] [Updated] (SPARK-4224) Support group acls

[ https://issues.apache.org/jira/browse/SPARK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-4224: - Target Version/s: 2.0.0 > Support group acls > -- > > Key: SPARK-4

[jira] [Commented] (SPARK-11316) coalesce doesn't handle UnionRDD with partial locality properly

[ https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264205#comment-15264205 ] Thomas Graves commented on SPARK-11316: --- Simple steps to reproduce an RDD with part

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264010#comment-15264010 ] Thomas Graves commented on SPARK-14963: --- I'm definitely fine with it but someone el

[jira] [Commented] (YARN-5010) maxActiveApplications and maxActiveApplicationsPerUser are missing from REST API

[ https://issues.apache.org/jira/browse/YARN-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262914#comment-15262914 ] Thomas Graves commented on YARN-5010: - we shouldn't just remove them as its an API comp

[jira] [Commented] (SPARK-1989) Exit executors faster if they get into a cycle of heavy GC

[ https://issues.apache.org/jira/browse/SPARK-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262761#comment-15262761 ] Thomas Graves commented on SPARK-1989: -- Personally I don't agree with this and think

[jira] [Updated] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-6735: - Assignee: Saisai Shao > Provide options to make maximum executor failure count ( which kills the

[jira] [Resolved] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-6735. -- Resolution: Fixed Fix Version/s: 2.0.0 > Provide options to make maximum executor failure

[jira] [Updated] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14963: -- Issue Type: Improvement (was: Bug) > YarnShuffleService should use YARN getRecoveryPath() for

[jira] [Created] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

Thomas Graves created SPARK-14963: - Summary: YarnShuffleService should use YARN getRecoveryPath() for leveldb location Key: SPARK-14963 URL: https://issues.apache.org/jira/browse/SPARK-14963 Project:

<    7   8   9   10   11   12   13   14   15   16   >