[jira] [Commented] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576700#comment-16576700 ] Thomas Graves commented on SPARK-25081: --- thanks, wanted to clarify since the descr

[jira] [Commented] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576234#comment-16576234 ] Thomas Graves commented on SPARK-25081: --- Does this ever result in the task reading

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16575267#comment-16575267 ] Thomas Graves commented on SPARK-25024: --- ok, I'm not familiar with mesos hardly at

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574956#comment-16574956 ] Thomas Graves commented on SPARK-23207: --- ok, I guess I disagree with that. Any cor

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16574886#comment-16574886 ] Thomas Graves commented on SPARK-23207: --- [~jiangxb1987] ^ > Shuffle+Repartition o

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573221#comment-16573221 ] Thomas Graves commented on SPARK-24924: --- | There was a discussion about why we sho

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572232#comment-16572232 ] Thomas Graves commented on SPARK-23207: --- does this affect spark 2.2 and < ? from t

[jira] [Commented] (SPARK-24598) SPARK SQL:Datatype overflow conditions gives incorrect result

[ https://issues.apache.org/jira/browse/SPARK-24598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572011#comment-16572011 ] Thomas Graves commented on SPARK-24598: --- In the very least we should file a separa

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16571908#comment-16571908 ] Thomas Graves commented on SPARK-24924: --- so originally when I started on this I di

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16571852#comment-16571852 ] Thomas Graves commented on SPARK-24924: --- thanks, I missed it in the output for spa

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570840#comment-16570840 ] Thomas Graves commented on SPARK-24924: --- so officially the spark api compatibility

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570736#comment-16570736 ] Thomas Graves commented on SPARK-24924: --- so if the user includes the databricks ja

[jira] [Resolved] (SPARK-24992) spark should randomize yarn local dir selection

[ https://issues.apache.org/jira/browse/SPARK-24992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24992. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 > spark shou

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570638#comment-16570638 ] Thomas Graves commented on SPARK-24924: --- So something I just thought of that I wan

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570586#comment-16570586 ] Thomas Graves commented on SPARK-24924: --- For compatibility we can't remove it unle

[jira] [Resolved] (SPARK-24981) ShutdownHook timeout causes job to fail when succeeded when SparkContext stop() not called by user program

[ https://issues.apache.org/jira/browse/SPARK-24981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24981. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 > ShutdownHo

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570220#comment-16570220 ] Thomas Graves commented on SPARK-24924: --- {quote}I have followed the changes in Avr

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570194#comment-16570194 ] Thomas Graves commented on SPARK-25024: --- We need to make it clear what mesos suppo

[jira] [Commented] (SPARK-25023) Clarify Spark security documentation

[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568805#comment-16568805 ] Thomas Graves commented on SPARK-25023: --- note some of this was already updated wit

[jira] [Issue Comment Deleted] (SPARK-25024) Update mesos documentation to be clear about security supported

[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25024: -- Comment: was deleted (was: I'm going to work on this.) > Update mesos documentation to be cle

[jira] [Commented] (SPARK-25023) Clarify Spark security documentation

[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568767#comment-16568767 ] Thomas Graves commented on SPARK-25023: --- I'm going to work on this > Clarify Spar

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568766#comment-16568766 ] Thomas Graves commented on SPARK-25024: --- I'm going to work on this. > Update meso

[jira] [Created] (SPARK-25024) Update mesos documentation to be clear about security supported

Thomas Graves created SPARK-25024: - Summary: Update mesos documentation to be clear about security supported Key: SPARK-25024 URL: https://issues.apache.org/jira/browse/SPARK-25024 Project: Spark

[jira] [Created] (SPARK-25023) Clarify Spark security documentation

Thomas Graves created SPARK-25023: - Summary: Clarify Spark security documentation Key: SPARK-25023 URL: https://issues.apache.org/jira/browse/SPARK-25023 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-25016) remove Support for hadoop 2.6

Thomas Graves created SPARK-25016: - Summary: remove Support for hadoop 2.6 Key: SPARK-25016 URL: https://issues.apache.org/jira/browse/SPARK-25016 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-25016) remove Support for hadoop 2.6

[ https://issues.apache.org/jira/browse/SPARK-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25016: -- Target Version/s: 3.0.0 > remove Support for hadoop 2.6 > - > >

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568393#comment-16568393 ] Thomas Graves commented on SPARK-24924: --- | It wouldn't be very different for 2.4.0

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568204#comment-16568204 ] Thomas Graves commented on SPARK-24924: --- [~felixcheung] did your discussion on the

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568199#comment-16568199 ] Thomas Graves commented on SPARK-24924: --- Hmm, so we are adding this for ease of up

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Target Version/s: 2.4.0 > Spark scheduler can hang when fetch failures, executor lost, task ru

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16567045#comment-16567045 ] Thomas Graves commented on SPARK-24924: --- why are we doing this? If a user ships th

[jira] [Commented] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565513#comment-16565513 ] Thomas Graves commented on SPARK-24909: --- looking more I think the fix may actually

[jira] [Commented] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565437#comment-16565437 ] Thomas Graves commented on SPARK-24909: --- this is unfortunately not a straight forw

[jira] [Commented] (SPARK-24986) OOM in BufferHolder during writes to a stream

[ https://issues.apache.org/jira/browse/SPARK-24986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565296#comment-16565296 ] Thomas Graves commented on SPARK-24986: --- fyi [~irashid] I know you were looking at

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563711#comment-16563711 ] Thomas Graves commented on SPARK-24615: --- so I guess my question is this the right

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563689#comment-16563689 ] Thomas Graves commented on SPARK-24579: --- going from Spark feeds data into DL/AI fr

[jira] [Commented] (SPARK-24934) Complex type and binary type in in-memory partition pruning does not work due to missing upper/lower bounds cases

[ https://issues.apache.org/jira/browse/SPARK-24934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16561944#comment-16561944 ] Thomas Graves commented on SPARK-24934: --- what is the real affected versions here? 

[jira] [Updated] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23243: -- Priority: Blocker (was: Major) > Shuffle+Repartition on an RDD could lead to incorrect answer

[jira] [Resolved] (SPARK-13343) speculative tasks that didn't commit shouldn't be marked as success

[ https://issues.apache.org/jira/browse/SPARK-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13343. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 > speculativ

[jira] [Commented] (SPARK-24918) Executor Plugin API

[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16558331#comment-16558331 ] Thomas Graves commented on SPARK-24918: --- I think this is a good idea. I thought I

[jira] [Commented] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554803#comment-16554803 ] Thomas Graves commented on SPARK-24909: --- I haven't come up with a fix yet but have

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Description: The DAGScheduler can hang if the executor was lost (due to fetch failure) and al

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Summary: Spark scheduler can hang when fetch failures, executor lost, task running on lost exe

[jira] [Commented] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554701#comment-16554701 ] Thomas Graves commented on SPARK-24909: --- Note this may have been introduced as par

[jira] [Updated] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Description: The DAGScheduler can hang if the executor was lost (due to fetch failure) and al

[jira] [Created] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

Thomas Graves created SPARK-24909: - Summary: Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts Key: SPARK-24909 URL: https://issues.apache.org/jira/browse/SPARK-24909

[jira] [Updated] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Priority: Critical (was: Major) > Spark scheduler can hang with fetch failures and executor l

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554327#comment-16554327 ] Thomas Graves commented on SPARK-24615: --- Right so I think part of this is trying t

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554306#comment-16554306 ] Thomas Graves commented on SPARK-23128: --- we also did some initial evaluation with

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551299#comment-16551299 ] Thomas Graves commented on SPARK-23128: --- [~carsonwang]  I'm curious if you are sti

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551240#comment-16551240 ] Thomas Graves commented on SPARK-24615: --- the other thing which I think I mentioned

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551233#comment-16551233 ] Thomas Graves commented on SPARK-24615: --- Ok so thinking about this a bit more I sl

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550803#comment-16550803 ] Thomas Graves commented on SPARK-24615: --- yes if any requirement can't be satisfied

[jira] [Assigned] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24755: - Assignee: Hieu Tri Huynh > Executor loss can cause task to not be resubmitted > ---

[jira] [Resolved] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24755. --- Resolution: Fixed Fix Version/s: 2.3.3 2.4.0 > Executor loss can c

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549356#comment-16549356 ] Thomas Graves commented on SPARK-22151: --- ok thanks, must have missed that. > PYTH

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549348#comment-16549348 ] Thomas Graves commented on SPARK-22151: --- [~srowen] why did you blank out fixed ver

[jira] [Resolved] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-22151. --- Resolution: Fixed Fix Version/s: 2.4.0 > PYTHONPATH not picked up from the spark.yarn

[jira] [Comment Edited] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549345#comment-16549345 ] Thomas Graves edited comment on SPARK-24615 at 7/19/18 2:28 PM: --

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549345#comment-16549345 ] Thomas Graves commented on SPARK-24615: --- but my point is exactly that, it shouldn'

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16549266#comment-16549266 ] Thomas Graves commented on SPARK-24615: --- I think the usage for cpu/memory is the s

[jira] [Assigned] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-22151: - Assignee: Parth Gandhi > PYTHONPATH not picked up from the spark.yarn.appMasterEnv prop

[jira] [Updated] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22151: -- Fix Version/s: 2.4.0 > PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly > --

[jira] [Updated] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24677: -- Fix Version/s: 2.2.3 > TaskSetManager not updating successfulTaskDurations for old stage attem

[jira] [Resolved] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24677. --- Resolution: Fixed Fix Version/s: 2.4.0 2.3.3 > TaskSetManager not

[jira] [Assigned] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24677: - Assignee: dzcxzl > TaskSetManager not updating successfulTaskDurations for old stage at

[jira] [Updated] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24677: -- Summary: TaskSetManager not updating successfulTaskDurations for old stage attempts (was: Avo

[jira] [Comment Edited] (SPARK-24677) Avoid NoSuchElementException from MedianHeap

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548210#comment-16548210 ] Thomas Graves edited comment on SPARK-24677 at 7/18/18 6:22 PM: --

[jira] [Commented] (SPARK-24677) Avoid NoSuchElementException from MedianHeap

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16548210#comment-16548210 ] Thomas Graves commented on SPARK-24677: --- In this case one of the older stage attem

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547903#comment-16547903 ] Thomas Graves commented on SPARK-24615: --- did the design doc permissions change? I

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546617#comment-16546617 ] Thomas Graves commented on SPARK-24615: --- ok, I agree, I think this SPIP should at

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546580#comment-16546580 ] Thomas Graves commented on SPARK-24615: --- The user is responsible for asking yarn f

[jira] [Commented] (SPARK-24615) Accelerator aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16545253#comment-16545253 ] Thomas Graves commented on SPARK-24615: --- [~jerryshao] ^ > Accelerator aware task

[jira] [Resolved] (SPARK-24610) wholeTextFiles broken for small files

[ https://issues.apache.org/jira/browse/SPARK-24610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24610. --- Resolution: Fixed Fix Version/s: 2.4.0 > wholeTextFiles broken for small files >

[jira] [Assigned] (SPARK-24610) wholeTextFiles broken for small files

[ https://issues.apache.org/jira/browse/SPARK-24610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24610: - Assignee: Dhruve Ashar > wholeTextFiles broken for small files > --

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16537035#comment-16537035 ] Thomas Graves commented on SPARK-16534: --- I agree it seems a bit of a bad user stor

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16537034#comment-16537034 ] Thomas Graves commented on SPARK-16534: --- If we aren't going to do this we should c

[jira] [Updated] (SPARK-13343) speculative tasks that didn't commit shouldn't be marked as success

[ https://issues.apache.org/jira/browse/SPARK-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-13343: -- Description: Currently Speculative tasks that didn't commit can show up as success  (dependin

[jira] [Commented] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529869#comment-16529869 ] Thomas Graves commented on SPARK-17181: --- that would be a question for [~marymwu]

[jira] [Commented] (SPARK-24615) Accelerator aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16528293#comment-16528293 ] Thomas Graves commented on SPARK-24615: --- maybe I"m missing it but how is this work

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527865#comment-16527865 ] Thomas Graves commented on SPARK-23309: --- We tried this on a newest 2.3.1 and haven

[jira] [Resolved] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-23309. --- Resolution: Works for Me > Spark 2.3 cached query performance 20-30% worse then spark 2.2 >

[jira] [Resolved] (SPARK-24372) Create script for preparing RCs

[ https://issues.apache.org/jira/browse/SPARK-24372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24372. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0 > Create scr

[jira] [Resolved] (SPARK-24519) MapStatus has 2000 hardcoded

[ https://issues.apache.org/jira/browse/SPARK-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24519. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 > MapStatus

[jira] [Updated] (SPARK-22897) Expose stageAttemptId in TaskContext

[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22897: -- Fix Version/s: 2.2.2 > Expose stageAttemptId in TaskContext > ---

[jira] [Resolved] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24589. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:02 PM: --

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:01 PM: --

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519445#comment-16519445 ] Thomas Graves commented on SPARK-24552: --- more details on hadoop committer side: S

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519430#comment-16519430 ] Thomas Graves commented on SPARK-24552: --- this is actually a problem with hadoop co

[jira] [Commented] (SPARK-24611) Clean up OutputCommitCoordinator

[ https://issues.apache.org/jira/browse/SPARK-24611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519420#comment-16519420 ] Thomas Graves commented on SPARK-24611: --- [~joshrosen]  just noticed you were the l

[jira] [Commented] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

[ https://issues.apache.org/jira/browse/SPARK-24622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519416#comment-16519416 ] Thomas Graves commented on SPARK-24622: --- Need to investigate further/test to make

[jira] [Created] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

Thomas Graves created SPARK-24622: - Summary: Task attempts in other stage attempts not killed when one task attempt succeeds Key: SPARK-24622 URL: https://issues.apache.org/jira/browse/SPARK-24622 Pro

[jira] [Updated] (SPARK-24519) MapStatus has 2000 hardcoded

[ https://issues.apache.org/jira/browse/SPARK-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24519: -- Description: MapStatus uses hardcoded value of 2000 partitions to determine if it should use

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Affects Version/s: 2.2.0 2.2.1 2.3.0

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Blocker (was: Critical) > Task attempt numbers are reused when stages are retried >

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512691#comment-16512691 ] Thomas Graves commented on SPARK-22148: --- ok, just update if you start working on i

[jira] [Commented] (SPARK-24539) HistoryServer does not display metrics from tasks that complete after stage failure

[ https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512643#comment-16512643 ] Thomas Graves commented on SPARK-24539: --- Its possible, I thought when I checked th

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16512519#comment-16512519 ] Thomas Graves commented on SPARK-24552: --- sorry just realized the v2 api is still m

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Critical (was: Blocker) > Task attempt numbers are reused when stages are retried >

<    2   3   4   5   6   7   8   9   10   11   >