[jira] [Commented] (SPARK-21082) Consider Executor's memory usage when scheduling task

[ https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056118#comment-16056118 ] Thomas Graves commented on SPARK-21082: --- Yes executor should not oom if you are try

[jira] [Assigned] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20898: - Assignee: Saisai Shao > spark.blacklist.killBlacklistedExecutors doesn't work in YARN >

[jira] [Resolved] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN

[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20898. --- Resolution: Fixed Fix Version/s: 2.3.0 > spark.blacklist.killBlacklistedExecutors does

[jira] [Assigned] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-13669: - Assignee: Saisai Shao > Job will always fail in the external shuffle service unavailable

[jira] [Resolved] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation

[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13669. --- Resolution: Fixed Fix Version/s: 2.3.0 > Job will always fail in the external shuffle

[jira] [Commented] (SPARK-21122) Address starvation issues when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075147#comment-16075147 ] Thomas Graves commented on SPARK-21122: --- I agree with Sean on this. if you are aim

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080340#comment-16080340 ] Thomas Graves commented on SPARK-19659: --- can you be more specific here, do you mean

[jira] [Created] (SPARK-21383) YARN: can allocate to many containers

Thomas Graves created SPARK-21383: - Summary: YARN: can allocate to many containers Key: SPARK-21383 URL: https://issues.apache.org/jira/browse/SPARK-21383 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21383) YARN can allocate to many executors

[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083321#comment-16083321 ] Thomas Graves commented on SPARK-21383: --- Note we saw this with dynamic allocation o

[jira] [Updated] (SPARK-21383) YARN can allocate to many executors

[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21383: -- Summary: YARN can allocate to many executors (was: YARN: can allocate to many containers) > Y

[jira] [Comment Edited] (SPARK-21383) YARN can allocate to many executors

[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083321#comment-16083321 ] Thomas Graves edited comment on SPARK-21383 at 7/12/17 2:17 AM: ---

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084075#comment-16084075 ] Thomas Graves commented on SPARK-21376: --- Can you please clarify the title and descr

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084458#comment-16084458 ] Thomas Graves commented on SPARK-21376: --- so you are referring to the org.apache.spa

[jira] [Resolved] (SPARK-21321) Spark very verbose on shutdown confusing users

[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21321. --- Resolution: Fixed Assignee: Jong Yoon Lee Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091531#comment-16091531 ] Thomas Graves commented on SPARK-15703: --- It should not be breaking dynamic allocati

[jira] [Assigned] (SPARK-21243) Limit the number of maps in a single shuffle fetch

[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-21243: - Assignee: Dhruve Ashar > Limit the number of maps in a single shuffle fetch > --

[jira] [Resolved] (SPARK-21243) Limit the number of maps in a single shuffle fetch

[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21243. --- Resolution: Fixed Fix Version/s: 2.3.0 > Limit the number of maps in a single shuffle

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095546#comment-16095546 ] Thomas Graves commented on SPARK-21460: --- I didn't think that was the case, but took

[jira] [Created] (SPARK-21501) Spark shuffle index cache size should be memory based

Thomas Graves created SPARK-21501: - Summary: Spark shuffle index cache size should be memory based Key: SPARK-21501 URL: https://issues.apache.org/jira/browse/SPARK-21501 Project: Spark Issue

[jira] [Updated] (SPARK-21243) Limit the number of maps in a single shuffle fetch

[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21243: -- Fix Version/s: 2.2.1 > Limit the number of maps in a single shuffle fetch > ---

[jira] [Updated] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21501: -- Affects Version/s: (was: 2.0.0) 2.1.0 > Spark shuffle index cache si

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098907#comment-16098907 ] Thomas Graves commented on SPARK-21501: --- The issue was actually introduced with SPA

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1616#comment-1616 ] Thomas Graves commented on SPARK-21501: --- We want to change it from a # of entries t

[jira] [Created] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

Thomas Graves created SPARK-21530: - Summary: Update description of spark.shuffle.maxChunksBeingTransferred Key: SPARK-21530 URL: https://issues.apache.org/jira/browse/SPARK-21530 Project: Spark

[jira] [Resolved] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21541. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Spark Logs sh

[jira] [Commented] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105133#comment-16105133 ] Thomas Graves commented on SPARK-21541: --- it was merged https://github.com/apache/s

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105164#comment-16105164 ] Thomas Graves commented on SPARK-17321: --- Can you clarify? as stated above you sho

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108901#comment-16108901 ] Thomas Graves commented on SPARK-21585: --- [~srowen]do you know if the github pull re

[jira] [Resolved] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21585. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Application M

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108913#comment-16108913 ] Thomas Graves commented on SPARK-21585: --- https://github.com/apache/spark/pull/18788

[jira] [Resolved] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20713. --- Resolution: Fixed Assignee: Nuochen Lyu Fix Version/s: 2.3.0 > Speculative ta

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: Nuochen Lyu > Speculative task that got CommitDenied exception shows up as fai

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: (was: Nuochen Lyu) > Speculative task that got CommitDenied exception show

[jira] [Updated] (SPARK-15990) Support rolling log aggregation for Spark running on YARN

[ https://issues.apache.org/jira/browse/SPARK-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15990: -- Assignee: Saisai Shao > Support rolling log aggregation for Spark running on YARN > ---

[jira] [Resolved] (SPARK-15990) Support rolling log aggregation for Spark running on YARN

[ https://issues.apache.org/jira/browse/SPARK-15990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15990. --- Resolution: Fixed Fix Version/s: 2.1.0 > Support rolling log aggregation for Spark run

[jira] [Commented] (SPARK-16382) YARN - Dynamic allocation with spark.executor.instances should increase max executors.

[ https://issues.apache.org/jira/browse/SPARK-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364363#comment-15364363 ] Thomas Graves commented on SPARK-16382: --- I think we should fail and complain and I

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364834#comment-15364834 ] Thomas Graves commented on SPARK-8425: -- Added some questions to the design doc > Add

[jira] [Comment Edited] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366251#comment-15366251 ] Thomas Graves edited comment on SPARK-16265 at 7/7/16 3:10 PM:

[jira] [Commented] (SPARK-16265) Add option to SparkSubmit to ship driver JRE to YARN

[ https://issues.apache.org/jira/browse/SPARK-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366251#comment-15366251 ] Thomas Graves commented on SPARK-16265: --- Have you tried shipping your jre as gzip f

[jira] [Commented] (SPARK-16399) Set PYSPARK_PYTHON to point to "python" instead of "python2.7"

[ https://issues.apache.org/jira/browse/SPARK-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366271#comment-15366271 ] Thomas Graves commented on SPARK-16399: --- so what is the behavior if I run this on o

[jira] [Commented] (SPARK-16298) spark.yarn.principal not working

[ https://issues.apache.org/jira/browse/SPARK-16298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366310#comment-15366310 ] Thomas Graves commented on SPARK-16298: --- kinit as the user in the keytab before sub

[jira] [Created] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

Thomas Graves created SPARK-16422: - Summary: maven 3.3.3 missing from mirror, breaks older builds Key: SPARK-16422 URL: https://issues.apache.org/jira/browse/SPARK-16422 Project: Spark Issue

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366544#comment-15366544 ] Thomas Graves commented on SPARK-16422: --- thanks, didn't see that go by. > maven 3

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366549#comment-15366549 ] Thomas Graves commented on SPARK-16422: --- Oh I didn't see it go by because there is

[jira] [Commented] (SPARK-16399) Set PYSPARK_PYTHON to point to "python" instead of "python2.7"

[ https://issues.apache.org/jira/browse/SPARK-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366567#comment-15366567 ] Thomas Graves commented on SPARK-16399: --- ok, I guess any other checks/error message

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366573#comment-15366573 ] Thomas Graves commented on SPARK-16422: --- maybe I missed some discussions but I thou

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366582#comment-15366582 ] Thomas Graves commented on SPARK-16422: --- I'll pop this discussions up on dev list a

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366829#comment-15366829 ] Thomas Graves commented on SPARK-16422: --- do we know why maven 3.3.3 got removed? I

[jira] [Updated] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16422: -- Assignee: Sean Owen > maven 3.3.3 missing from mirror, breaks older builds > --

[jira] [Commented] (SPARK-16422) maven 3.3.3 missing from mirror, breaks older builds

[ https://issues.apache.org/jira/browse/SPARK-16422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367683#comment-15367683 ] Thomas Graves commented on SPARK-16422: --- it looks like there is generally about 6 m

[jira] [Commented] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370762#comment-15370762 ] Thomas Graves commented on SPARK-15703: --- sorry for my delay in responding, we repro

[jira] [Commented] (SPARK-16455) Add a new hook in CoarseGrainedSchedulerBackend in order to stop scheduling new tasks when cluster is restarting

[ https://issues.apache.org/jira/browse/SPARK-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370828#comment-15370828 ] Thomas Graves commented on SPARK-16455: --- | we are implementing a new mechanism whic

[jira] [Commented] (SPARK-16451) Spark-shell / pyspark should finish gracefully when "SaslException: GSS initiate failed" is hit

[ https://issues.apache.org/jira/browse/SPARK-16451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370832#comment-15370832 ] Thomas Graves commented on SPARK-16451: --- I think there are a bunch of cases where i

[jira] [Commented] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15370836#comment-15370836 ] Thomas Graves commented on SPARK-16435: --- Yeah I agree,I'm not as worried about this

[jira] [Resolved] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16435. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > Behavior changes if

[jira] [Updated] (SPARK-16435) Behavior changes if initialExecutor is less than minExecutor for dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16435: -- Assignee: Saisai Shao > Behavior changes if initialExecutor is less than minExecutor for dynami

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376913#comment-15376913 ] Thomas Graves commented on SPARK-14963: --- committed fix in https://github.com/apache

[jira] [Updated] (SPARK-16505) YARN shuffle service should throw errors when it fails to start

[ https://issues.apache.org/jira/browse/SPARK-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16505: -- Assignee: Marcelo Vanzin > YARN shuffle service should throw errors when it fails to start > --

[jira] [Resolved] (SPARK-16505) YARN shuffle service should throw errors when it fails to start

[ https://issues.apache.org/jira/browse/SPARK-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16505. --- Resolution: Fixed Fix Version/s: 2.1.0 > YARN shuffle service should throw errors when

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379361#comment-15379361 ] Thomas Graves commented on SPARK-8425: -- Slightly different scenario since its not wit

[jira] [Commented] (SPARK-5311) EventLoggingListener throws exception if log directory does not exist

[ https://issues.apache.org/jira/browse/SPARK-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379375#comment-15379375 ] Thomas Graves commented on SPARK-5311: -- We should not be creating the event log dir b

[jira] [Commented] (SPARK-5311) EventLoggingListener throws exception if log directory does not exist

[ https://issues.apache.org/jira/browse/SPARK-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379376#comment-15379376 ] Thomas Graves commented on SPARK-5311: -- Note as a follow up, obviously you can config

[jira] [Commented] (SPARK-12661) Drop Python 2.6 support in PySpark

[ https://issues.apache.org/jira/browse/SPARK-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384392#comment-15384392 ] Thomas Graves commented on SPARK-12661: --- I think [~nchammas] is talking about dropp

[jira] [Commented] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384571#comment-15384571 ] Thomas Graves commented on SPARK-15703: --- I think we should do both. event queue si

[jira] [Created] (SPARK-16630) Blacklist a node if executors won't launch on it.

Thomas Graves created SPARK-16630: - Summary: Blacklist a node if executors won't launch on it. Key: SPARK-16630 URL: https://issues.apache.org/jira/browse/SPARK-16630 Project: Spark Issue Typ

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384744#comment-15384744 ] Thomas Graves commented on SPARK-16630: --- Note this is more at the resource manager

[jira] [Updated] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16630: -- Description: On YARN, its possible that a node is messed or misconfigured such that a containe

[jira] [Commented] (SPARK-16650) Improve documentation of spark.task.maxFailures

[ https://issues.apache.org/jira/browse/SPARK-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385868#comment-15385868 ] Thomas Graves commented on SPARK-16650: --- I'll try to put up a patch for this later

[jira] [Created] (SPARK-16650) Improve documentation of spark.task.maxFailures

Thomas Graves created SPARK-16650: - Summary: Improve documentation of spark.task.maxFailures Key: SPARK-16650 URL: https://issues.apache.org/jira/browse/SPARK-16650 Project: Spark Issue Ty

[jira] [Commented] (SPARK-2666) Always try to cancel running tasks when a stage is marked as zombie

[ https://issues.apache.org/jira/browse/SPARK-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386235#comment-15386235 ] Thomas Graves commented on SPARK-2666: -- I think eventually adding prestart (MapReduce

[jira] [Resolved] (SPARK-15951) Change Executors Page to use datatables to support sorting columns and searching

[ https://issues.apache.org/jira/browse/SPARK-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15951. --- Resolution: Fixed Fix Version/s: 2.1.0 > Change Executors Page to use datatables to su

[jira] [Updated] (SPARK-15951) Change Executors Page to use datatables to support sorting columns and searching

[ https://issues.apache.org/jira/browse/SPARK-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15951: -- Assignee: Kishor Patil > Change Executors Page to use datatables to support sorting columns and

[jira] [Commented] (SPARK-2666) Always try to cancel running tasks when a stage is marked as zombie

[ https://issues.apache.org/jira/browse/SPARK-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386561#comment-15386561 ] Thomas Graves commented on SPARK-2666: -- thanks for the explanation. I guess we would

[jira] [Created] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

Thomas Graves created SPARK-16711: - Summary: YarnShuffleService doesn't re-init properly on YARN rolling upgrade Key: SPARK-16711 URL: https://issues.apache.org/jira/browse/SPARK-16711 Project: Spark

[jira] [Commented] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

[ https://issues.apache.org/jira/browse/SPARK-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392135#comment-15392135 ] Thomas Graves commented on SPARK-16711: --- Note that what happens here is the yarnshu

[jira] [Commented] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

[ https://issues.apache.org/jira/browse/SPARK-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392145#comment-15392145 ] Thomas Graves commented on SPARK-16711: --- Note this happens when security is on. Th

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Summary: Make ListenerBus event queue size configurable (was: Spark UI doesn't show all tasks

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Issue Type: Improvement (was: Bug) > Make ListenerBus event queue size configurable >

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394271#comment-15394271 ] Thomas Graves commented on SPARK-15703: --- Using this jira to make lsitenerbus config

[jira] [Resolved] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15703. --- Resolution: Fixed Fix Version/s: 2.1.0 > Make ListenerBus event queue size configurabl

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Priority: Minor (was: Critical) > Make ListenerBus event queue size configurable > ---

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Assignee: Dhruve Ashar > Make ListenerBus event queue size configurable > -

[jira] [Commented] (SPARK-16744) spark.yarn.appMasterEnv handling assumes values should be appended

[ https://issues.apache.org/jira/browse/SPARK-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15395611#comment-15395611 ] Thomas Graves commented on SPARK-16744: --- Along with this we may consider thinking a

[jira] [Commented] (SPARK-16664) Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.

[ https://issues.apache.org/jira/browse/SPARK-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399168#comment-15399168 ] Thomas Graves commented on SPARK-16664: --- I am out of the office until 8/8. Please c

[jira] [Commented] (SPARK-27005) Design sketch: Accelerator-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784555#comment-16784555 ] Thomas Graves commented on SPARK-27005: --- so we have both a google design doc and t

[jira] [Comment Edited] (SPARK-27005) Design sketch: Accelerator-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784555#comment-16784555 ] Thomas Graves edited comment on SPARK-27005 at 3/5/19 3:40 PM: ---

[jira] [Updated] (SPARK-27024) Executor interface for cluster managers to support GPU resources

[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27024: -- Description: The executor interface shall deal with the resources allocated to the executor by

[jira] [Updated] (SPARK-27024) Executor interface for cluster managers to support GPU resources

[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27024: -- Description: The executor interface shall deal with the resources allocated to the executor b

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809893#comment-16809893 ] Thomas Graves commented on SPARK-27364: --- working on this, will post a basic design

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812488#comment-16812488 ] Thomas Graves commented on SPARK-27364: --- There are 3 main user facing impacts for

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812488#comment-16812488 ] Thomas Graves edited comment on SPARK-27364 at 4/8/19 3:01 PM: ---

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812488#comment-16812488 ] Thomas Graves edited comment on SPARK-27364 at 4/8/19 3:10 PM: ---

[jira] [Assigned] (SPARK-27361) YARN support for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27361: - Assignee: Thomas Graves > YARN support for GPU-aware scheduling > -

[jira] [Commented] (SPARK-27396) SPIP: Public APIs for extended Columnar Processing Support

[ https://issues.apache.org/jira/browse/SPARK-27396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815403#comment-16815403 ] Thomas Graves commented on SPARK-27396: --- I can shephard it. > SPIP: Public APIs f

[jira] [Commented] (SPARK-27176) Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4

[ https://issues.apache.org/jira/browse/SPARK-27176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815408#comment-16815408 ] Thomas Graves commented on SPARK-27176: --- It looks like the hadoop-3.2 profile no l

[jira] [Commented] (SPARK-27176) Upgrade hadoop-3's built-in Hive maven dependencies to 2.3.4

[ https://issues.apache.org/jira/browse/SPARK-27176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815414#comment-16815414 ] Thomas Graves commented on SPARK-27176: --- looks like I see one: https://github.com/

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812488#comment-16812488 ] Thomas Graves edited comment on SPARK-27364 at 4/11/19 1:48 PM: --

[jira] [Commented] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815465#comment-16815465 ] Thomas Graves commented on SPARK-27364: --- So there is actually another one we need

[jira] [Comment Edited] (SPARK-27364) User-facing APIs for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815465#comment-16815465 ] Thomas Graves edited comment on SPARK-27364 at 4/11/19 2:37 PM: --

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818053#comment-16818053 ] Thomas Graves commented on SPARK-25250: --- [~cloud_fan] can you please add details a

  1   2   3   4   5   6   7   8   9   10   >