[jira] [Updated] (SPARK-15951) Change Executors Page to use datatables to support sorting columns and searching

2016-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15951: -- Assignee: Kishor Patil > Change Executors Page to use datatables to support sorting columns

[jira] [Commented] (SPARK-2666) Always try to cancel running tasks when a stage is marked as zombie

2016-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386235#comment-15386235 ] Thomas Graves commented on SPARK-2666: -- I think eventually adding prestart (MapReduce slowstart type

[jira] [Commented] (SPARK-5311) EventLoggingListener throws exception if log directory does not exist

2016-07-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379375#comment-15379375 ] Thomas Graves commented on SPARK-5311: -- We should not be creating the event log dir because you don't

[jira] [Commented] (SPARK-5311) EventLoggingListener throws exception if log directory does not exist

2016-07-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379376#comment-15379376 ] Thomas Graves commented on SPARK-5311: -- Note as a follow up, obviously you can configure it to be any

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

2016-07-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379361#comment-15379361 ] Thomas Graves commented on SPARK-8425: -- Slightly different scenario since its not with the

[jira] [Comment Edited] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366251#comment-15366251 ] Thomas Graves edited comment on SPARK-16265 at 7/7/16 3:10 PM: --- Have you

[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366251#comment-15366251 ] Thomas Graves commented on SPARK-16265: --- Have you tried shipping your jre as gzip file in the

[jira] [Commented] (SPARK-16298) spark.yarn.principal not working

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366310#comment-15366310 ] Thomas Graves commented on SPARK-16298: --- kinit as the user in the keytab before submitting the job.

[jira] [Commented] (SPARK-16399) Set PYSPARK_PYTHON to point to "python" instead of "python2.7"

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366271#comment-15366271 ] Thomas Graves commented on SPARK-16399: --- so what is the behavior if I run this on older rhel/centos

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366549#comment-15366549 ] Thomas Graves commented on SPARK-16422: --- Oh I didn't see it go by because there is no jira

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366573#comment-15366573 ] Thomas Graves commented on SPARK-16422: --- maybe I missed some discussions but I thought we were

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366544#comment-15366544 ] Thomas Graves commented on SPARK-16422: --- thanks, didn't see that go by. > maven 3.3.3 missing

[jira] [Commented] (SPARK-16399) Set PYSPARK_PYTHON to point to "python" instead of "python2.7"

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366567#comment-15366567 ] Thomas Graves commented on SPARK-16399: --- ok, I guess any other checks/error messages can be added

[jira] [Created] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

2016-07-07 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-16422: - Summary: maven 3.3.3 missing from mirror, breaks older builds Key: SPARK-16422 URL: https://issues.apache.org/jira/browse/SPARK-16422 Project: Spark Issue

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366582#comment-15366582 ] Thomas Graves commented on SPARK-16422: --- I'll pop this discussions up on dev list again. > maven

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

2016-07-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367683#comment-15367683 ] Thomas Graves commented on SPARK-16422: --- it looks like there is generally about 6 months between

[jira] [Updated] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

2016-07-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16422: -- Assignee: Sean Owen > maven 3.3.3 missing from mirror, breaks older builds >

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

2016-07-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364834#comment-15364834 ] Thomas Graves commented on SPARK-8425: -- Added some questions to the design doc > Add blacklist

[jira] [Commented] (SPARK-16382) YARN - Dynamic allocation with spark.executor.instances should increase max executors.

2016-07-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15364363#comment-15364363 ] Thomas Graves commented on SPARK-16382: --- I think we should fail and complain and I actually thought

[jira] [Commented] (SPARK-16455) Add a new hook in CoarseGrainedSchedulerBackend in order to stop scheduling new tasks when cluster is restarting

2016-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370828#comment-15370828 ] Thomas Graves commented on SPARK-16455: --- | we are implementing a new mechanism which will let

[jira] [Commented] (SPARK-16451) Spark-shell / pyspark should finish gracefully when "SaslException: GSS initiate failed" is hit

2016-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370832#comment-15370832 ] Thomas Graves commented on SPARK-16451: --- I think there are a bunch of cases where if the

[jira] [Commented] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

2016-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370836#comment-15370836 ] Thomas Graves commented on SPARK-16435: --- Yeah I agree,I'm not as worried about this one as I am the

[jira] [Commented] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

2016-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370762#comment-15370762 ] Thomas Graves commented on SPARK-15703: --- sorry for my delay in responding, we reproduced this again

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

2016-07-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366829#comment-15366829 ] Thomas Graves commented on SPARK-16422: --- do we know why maven 3.3.3 got removed? I assume this is

[jira] [Updated] (SPARK-15990) Support rolling log aggregation for Spark running on YARN

2016-06-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15990: -- Assignee: Saisai Shao > Support rolling log aggregation for Spark running on YARN >

[jira] [Resolved] (SPARK-15990) Support rolling log aggregation for Spark running on YARN

2016-06-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15990. --- Resolution: Fixed Fix Version/s: 2.1.0 > Support rolling log aggregation for Spark

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

2017-02-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849965#comment-15849965 ] Thomas Graves commented on SPARK-18085: --- Are the ui updates just updates to use new api or are you

[jira] (SPARK-16742) Kerberos support for Spark on Mesos

2017-01-30 Thread Thomas Graves (JIRA)
Title: Message Title Thomas Graves commented on SPARK-16742

[jira] [Resolved] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow

2017-01-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18750. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19179. --- Resolution: Fixed Fix Version/s: 2.2.0 > spark.yarn.access.namenodes description is

[jira] [Updated] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19179: -- Assignee: Saisai Shao > spark.yarn.access.namenodes description is wrong >

[jira] [Commented] (SPARK-17148) NodeManager exit because of exception “Executor is not registered”

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428289#comment-15428289 ] Thomas Graves commented on SPARK-17148: --- If this is causing the nodemanager to die this is bad and

[jira] [Resolved] (SPARK-11227) Spark1.5+ HDFS HA mode throw java.net.UnknownHostException: nameservice1

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-11227. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > Spark1.5+ HDFS HA

[jira] [Resolved] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16673. --- Resolution: Fixed Fix Version/s: 2.1.0 > New Executor Page displays columns that used

[jira] [Updated] (SPARK-11227) Spark1.5+ HDFS HA mode throw java.net.UnknownHostException: nameservice1

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11227: -- Assignee: Kousuke Saruta > Spark1.5+ HDFS HA mode throw java.net.UnknownHostException:

[jira] [Updated] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16673: -- Assignee: Alex Bozarth > New Executor Page displays columns that used to be conditionally

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428285#comment-15428285 ] Thomas Graves commented on SPARK-16914: --- Actually with SPARK-14963, the cluster admin is supposed

[jira] [Commented] (SPARK-16158) Support pluggable dynamic allocation heuristics

2016-08-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421803#comment-15421803 ] Thomas Graves commented on SPARK-16158: --- seems like an ok idea to me, but you have to make sure to

[jira] [Commented] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

2016-08-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430855#comment-15430855 ] Thomas Graves commented on SPARK-17181: --- check your log files to see if you see something like:

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

2016-08-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430725#comment-15430725 ] Thomas Graves commented on SPARK-16914: --- that is considered a fatal issue for the nodemanager and

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-02-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888430#comment-15888430 ] Thomas Graves commented on SPARK-18769: --- {quote} A little clarification in case the summary is not

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-03-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890629#comment-15890629 ] Thomas Graves commented on SPARK-18769: --- [~yuming] I already made a comment on that, I don't think

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-03-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892318#comment-15892318 ] Thomas Graves commented on SPARK-18769: --- I definitely understand there is an actual problem here,

[jira] [Created] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

2016-09-07 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-17433: - Summary: YarnShuffleService doesn't handle moving credentials levelDb Key: SPARK-17433 URL: https://issues.apache.org/jira/browse/SPARK-17433 Project: Spark

[jira] [Updated] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

2016-09-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16711: -- Fix Version/s: 2.0.1 > YarnShuffleService doesn't re-init properly on YARN rolling upgrade >

[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history

2016-08-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449250#comment-15449250 ] Thomas Graves commented on SPARK-17243: --- I agree, there are a ton of ways to improve the history

[jira] [Commented] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

2016-09-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470693#comment-15470693 ] Thomas Graves commented on SPARK-17433: --- will have path up shortly. > YarnShuffleService doesn't

[jira] [Comment Edited] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

2016-09-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470693#comment-15470693 ] Thomas Graves edited comment on SPARK-17433 at 9/7/16 1:50 PM: --- will have

[jira] [Resolved] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history

2016-08-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17243. --- Resolution: Fixed Assignee: Alex Bozarth Fix Version/s: 2.1.0 > Spark 2.0

[jira] [Updated] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history

2016-08-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17243: -- Fix Version/s: 2.0.1 > Spark 2.0 history server summary page gets stuck at "loading history

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474557#comment-15474557 ] Thomas Graves commented on SPARK-17321: --- so there are 2 possible things here: 1) You are using

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477061#comment-15477061 ] Thomas Graves commented on SPARK-17321: --- Note that if we want to fix this without yarn NM recovery

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477028#comment-15477028 ] Thomas Graves commented on SPARK-17321: --- it is possible but the point of the recovery dir is it

[jira] [Resolved] (SPARK-17511) Dynamic allocation race condition: Containers getting marked failed while releasing

2016-09-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17511. --- Resolution: Fixed Assignee: Kishor Patil Fix Version/s: 2.1.0

[jira] [Resolved] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

2016-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17433. --- Resolution: Fixed Assignee: Thomas Graves Fix Version/s: 2.1.0 >

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484748#comment-15484748 ] Thomas Graves commented on SPARK-17321: --- Not sure I follow this comment. So you are using NM

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487203#comment-15487203 ] Thomas Graves commented on SPARK-17321: --- yes that makes sense and as I stated I think the fix for

[jira] [Commented] (SPARK-17582) Dead executors shouldn't show in the SparkUI

2016-09-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503461#comment-15503461 ] Thomas Graves commented on SPARK-17582: --- Yes they are meant to be there, status is DEAD. This

[jira] [Updated] (SPARK-15083) History Server would OOM due to unlimited TaskUIData in some stages

2016-08-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15083: -- Fix Version/s: 2.0.1 > History Server would OOM due to unlimited TaskUIData in some stages >

[jira] [Resolved] (SPARK-15083) History Server would OOM due to unlimited TaskUIData in some stages

2016-08-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15083. --- Resolution: Fixed Assignee: Alex Bozarth Fix Version/s: 2.1.0 > History

[jira] [Resolved] (SPARK-16757) Set up caller context to HDFS and Yarn

2016-09-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16757. --- Resolution: Fixed Assignee: Weiqing Yang Fix Version/s: 2.1.0 > Set up

[jira] [Commented] (SPARK-16757) Set up caller context to HDFS and Yarn

2016-09-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531056#comment-15531056 ] Thomas Graves commented on SPARK-16757: --- we are seeing issues with ReplSuite after adding the

[jira] [Resolved] (SPARK-17710) ReplSuite fails with ClassCircularityError in master Maven builds

2016-09-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17710. --- Resolution: Fixed Assignee: Weiqing Yang Fix Version/s: 2.1.0 > ReplSuite

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608719#comment-15608719 ] Thomas Graves commented on SPARK-18085: --- So there are a few things unclear to me about the details

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608724#comment-15608724 ] Thomas Graves commented on SPARK-18085: --- Hadn't refreshed and seen your comment about application

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608760#comment-15608760 ] Thomas Graves commented on SPARK-18085: --- I should also say I don't think we should throw out the

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609099#comment-15609099 ] Thomas Graves commented on SPARK-18085: --- Can you expand on is the source of most of the issues?

[jira] [Commented] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647941#comment-15647941 ] Thomas Graves commented on SPARK-18357: --- [~kishorvpatil] can you fix please > YARN

[jira] [Created] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-18357: - Summary: YARN --files/--archives broke Key: SPARK-18357 URL: https://issues.apache.org/jira/browse/SPARK-18357 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-18357: -- Priority: Blocker (was: Major) > YARN --files/--archives broke >

[jira] [Updated] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-18357: -- Assignee: Kishor Patil > YARN --files/--archives broke > - > >

[jira] [Resolved] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18357. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.0 > YARN

[jira] [Updated] (SPARK-17417) Fix sorting of part files while reconstructing RDD/partition from checkpointed files.

2016-10-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17417: -- Affects Version/s: 1.6.0 2.0.0 > Fix sorting of part files while

[jira] [Resolved] (SPARK-17417) Fix sorting of part files while reconstructing RDD/partition from checkpointed files.

2016-10-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17417. --- Resolution: Fixed Assignee: Dhruve Ashar Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-15708) Tasks table in Detailed Stage page shows ip instead of hostname under Executor ID/Host

2016-10-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585451#comment-15585451 ] Thomas Graves commented on SPARK-15708: --- [~ajbozarth] It was closed because we weren't able to

[jira] [Commented] (SPARK-17979) Remove deprecated support for config SPARK_YARN_USER_ENV

2016-10-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585664#comment-15585664 ] Thomas Graves commented on SPARK-17979: --- We already had a jira to remove all the deprecated

[jira] [Resolved] (SPARK-17843) History Server gives no feedback about the application list being incomplete

2016-11-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17843. --- Resolution: Fixed Assignee: Vinayak Joshi Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

2016-11-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667226#comment-15667226 ] Thomas Graves commented on SPARK-18085: --- I think at a high level its fine. You mention using a /v2

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612086#comment-15612086 ] Thomas Graves commented on SPARK-18085: --- Perhaps we can clarify the title on this jira to be

[jira] [Resolved] (SPARK-18099) Spark distributed cache should throw exception if same file is specified to dropped in --files --archives

2016-11-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18099. --- Resolution: Fixed Assignee: Kishor Patil Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-18733) Spark history server file cleaner excludes in-progress files

2016-12-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726671#comment-15726671 ] Thomas Graves commented on SPARK-18733: --- yes looks like a dup but I'm not sure on current solution.

[jira] [Created] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-11 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-19179: - Summary: spark.yarn.access.namenodes description is wrong Key: SPARK-19179 URL: https://issues.apache.org/jira/browse/SPARK-19179 Project: Spark Issue

[jira] [Resolved] (SPARK-19021) Generailize HDFSCredentialProvider to support non HDFS security FS

2017-01-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19021. --- Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.2.0 > Generailize

[jira] [Updated] (SPARK-17843) History Server gives no feedback about the application list being incomplete

2016-11-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17843: -- Fix Version/s: 2.0.3 > History Server gives no feedback about the application list being

[jira] [Created] (SPARK-19112) add codec for ZStandard

2017-01-06 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-19112: - Summary: add codec for ZStandard Key: SPARK-19112 URL: https://issues.apache.org/jira/browse/SPARK-19112 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795863#comment-15795863 ] Thomas Graves commented on SPARK-19033: --- this should actually be an improvement to allow admin acls

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-01-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811935#comment-15811935 ] Thomas Graves commented on SPARK-19112: --- For any sort of Hadoop output formats we get it for free

[jira] [Updated] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19033: -- Assignee: Saisai Shao > HistoryServer still uses old ACLs even if ACLs are updated >

[jira] [Resolved] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19033. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > HistoryServer still

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-01-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815078#comment-15815078 ] Thomas Graves commented on SPARK-19143: --- [~mridulm80] You say "we added", you are saying you have

[jira] [Commented] (SPARK-18733) Spark history server file cleaner excludes in-progress files

2016-12-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726701#comment-15726701 ] Thomas Graves commented on SPARK-18733: --- oh nevermind its looking at lastupdated time. > Spark

[jira] [Commented] (SPARK-19904) SPIP Add Spark Project Improvement Proposal doc to website

2017-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943799#comment-15943799 ] Thomas Graves commented on SPARK-19904: --- Is this done or what is this waiting on? > SPIP Add

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944046#comment-15944046 ] Thomas Graves commented on SPARK-19143: --- Yes I can be Shephard. > API in Spark for distributing

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943284#comment-15943284 ] Thomas Graves commented on SPARK-19143: --- I assume this needs to go through the new spip process:

[jira] [Created] (SPARK-20178) Improve Scheduler fetch failures

2017-03-31 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-20178: - Summary: Improve Scheduler fetch failures Key: SPARK-20178 URL: https://issues.apache.org/jira/browse/SPARK-20178 Project: Spark Issue Type: Epic

[jira] [Commented] (SPARK-20340) Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM

2017-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971071#comment-15971071 ] Thomas Graves commented on SPARK-20340: --- Right, I figured it was probably for performance, the

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

2017-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975029#comment-15975029 ] Thomas Graves commented on SPARK-20391: --- I agree that if its been released we can't change it, the

[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API

2017-04-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975115#comment-15975115 ] Thomas Graves commented on SPARK-20391: --- > My proposal was to add 2 extra fields which duplicate

[jira] [Commented] (SPARK-14245) webUI should display the user

2017-04-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971365#comment-15971365 ] Thomas Graves commented on SPARK-14245: --- see the commend in the PR, I think there was a race with

[jira] [Commented] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

2017-04-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979349#comment-15979349 ] Thomas Graves commented on SPARK-19812: --- Sorry wasn't clear in the original description, it errors

<    3   4   5   6   7   8   9   10   11   12   >