[jira] [Assigned] (SPARK-17663) SchedulableBuilder should handle invalid data access via scheduler.allocation.file

[ https://issues.apache.org/jira/browse/SPARK-17663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-17663: Assignee: Eren Avsarogullari (was: Imran Rashid) > SchedulableBuilder should handle

[jira] [Assigned] (SPARK-17663) SchedulableBuilder should handle invalid data access via scheduler.allocation.file

[ https://issues.apache.org/jira/browse/SPARK-17663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-17663: Assignee: Imran Rashid > SchedulableBuilder should handle invalid data access via >

[jira] [Resolved] (SPARK-17663) SchedulableBuilder should handle invalid data access via scheduler.allocation.file

[ https://issues.apache.org/jira/browse/SPARK-17663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-17663. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 15237

[jira] [Commented] (SPARK-12297) Add work-around for Parquet/Hive int96 timestamp bug.

[ https://issues.apache.org/jira/browse/SPARK-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849974#comment-15849974 ] Imran Rashid commented on SPARK-12297: -- This is related to HIVE-12767. It doesn't make sense to

[jira] [Updated] (SPARK-19069) Expose task 'status' and 'duration' in spark history server REST API.

[ https://issues.apache.org/jira/browse/SPARK-19069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-19069: - Assignee: Parag Chaudhari > Expose task 'status' and 'duration' in spark history server REST

[jira] [Resolved] (SPARK-19069) Expose task 'status' and 'duration' in spark history server REST API.

[ https://issues.apache.org/jira/browse/SPARK-19069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-19069. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16473

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830530#comment-15830530 ] Imran Rashid commented on SPARK-18886: -- I had another idea for how to fix this. In addition to

[jira] [Commented] (SPARK-19276) FetchFailures can be hidden by user (or sql) exception handling

[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830487#comment-15830487 ] Imran Rashid commented on SPARK-19276: -- [~markhamstra] bq. I guess my only real question is if we

[jira] [Resolved] (SPARK-16654) UI Should show blacklisted executors & nodes

[ https://issues.apache.org/jira/browse/SPARK-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-16654. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16346

[jira] [Commented] (SPARK-19276) FetchFailures can be hidden be user (or sql) exception handling

[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828518#comment-15828518 ] Imran Rashid commented on SPARK-19276: -- I haven't been successful in creating a test case to

[jira] [Updated] (SPARK-19276) FetchFailures can be hidden by user (or sql) exception handling

[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-19276: - Summary: FetchFailures can be hidden by user (or sql) exception handling (was: FetchFailures

[jira] [Updated] (SPARK-19276) FetchFailures can be hidden be user (or sql) exception handling

[ https://issues.apache.org/jira/browse/SPARK-19276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-19276: - Description: The scheduler handles node failures by looking for a special

[jira] [Created] (SPARK-19276) FetchFailures can be hidden be user (or sql) exception handling

Imran Rashid created SPARK-19276: Summary: FetchFailures can be hidden be user (or sql) exception handling Key: SPARK-19276 URL: https://issues.apache.org/jira/browse/SPARK-19276 Project: Spark

[jira] [Closed] (SPARK-19262) DAGScheduler should handle stage's pendingPartitions properly in handleTaskCompletion.

[ https://issues.apache.org/jira/browse/SPARK-19262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid closed SPARK-19262. Resolution: Duplicate Looks like there was an accidental duplicate w/ SPARK-19623. Pr has already

[jira] [Issue Comment Deleted] (SPARK-18784) Managed memory leak - spark-2.0.2

[ https://issues.apache.org/jira/browse/SPARK-18784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-18784: - Comment: was deleted (was: That warning is a false alarm when you use {{rdd.take()}}.) >

[jira] [Commented] (SPARK-18784) Managed memory leak - spark-2.0.2

[ https://issues.apache.org/jira/browse/SPARK-18784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816180#comment-15816180 ] Imran Rashid commented on SPARK-18784: -- That warning is a false alarm when you use {{rdd.take()}}.

[jira] [Commented] (SPARK-18181) Huge managed memory leak (2.7G) when running reduceByKey

[ https://issues.apache.org/jira/browse/SPARK-18181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816160#comment-15816160 ] Imran Rashid commented on SPARK-18181: -- [~barrybecker4] I'm pretty sure this is just a misleading

[jira] [Closed] (SPARK-18181) Huge managed memory leak (2.7G) when running reduceByKey

[ https://issues.apache.org/jira/browse/SPARK-18181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid closed SPARK-18181. Resolution: Information Provided > Huge managed memory leak (2.7G) when running reduceByKey >

[jira] [Commented] (SPARK-19108) Broadcast all shared parts of tasks (to reduce task serialization time)

[ https://issues.apache.org/jira/browse/SPARK-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812130#comment-15812130 ] Imran Rashid commented on SPARK-19108: -- yeah, great point. I was thinking two broadcasts just for

[jira] [Commented] (SPARK-19108) Broadcast all shared parts of tasks (to reduce task serialization time)

[ https://issues.apache.org/jira/browse/SPARK-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812127#comment-15812127 ] Imran Rashid commented on SPARK-19108: -- On SPARK-18890, [~gq] pointed out something really important

[jira] [Commented] (SPARK-18890) Do all task serialization in CoarseGrainedExecutorBackend thread (rather than TaskSchedulerImpl)

[ https://issues.apache.org/jira/browse/SPARK-18890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811951#comment-15811951 ] Imran Rashid commented on SPARK-18890: -- oh, great point, I was missing something really important --

[jira] [Commented] (SPARK-18890) Do all task serialization in CoarseGrainedExecutorBackend thread (rather than TaskSchedulerImpl)

[ https://issues.apache.org/jira/browse/SPARK-18890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805087#comment-15805087 ] Imran Rashid commented on SPARK-18890: -- [~gq] I think you misunderstood my suggestion about using a

[jira] [Updated] (SPARK-17931) taskScheduler has some unneeded serialization

[ https://issues.apache.org/jira/browse/SPARK-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-17931: - Assignee: Kay Ousterhout > taskScheduler has some unneeded serialization >

[jira] [Resolved] (SPARK-17931) taskScheduler has some unneeded serialization

[ https://issues.apache.org/jira/browse/SPARK-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-17931. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16053

[jira] [Commented] (SPARK-18967) Locality preferences should be used when scheduling even when delay scheduling is turned off

[ https://issues.apache.org/jira/browse/SPARK-18967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767824#comment-15767824 ] Imran Rashid commented on SPARK-18967: -- cc [~kayousterhout] [~markhamstra] [~mridulm80] > Locality

[jira] [Updated] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-18886: - Description: Delay scheduling can introduce an unbounded delay and underutilization of cluster

[jira] [Created] (SPARK-18967) Locality preferences should be used when scheduling even when delay scheduling is turned off

Imran Rashid created SPARK-18967: Summary: Locality preferences should be used when scheduling even when delay scheduling is turned off Key: SPARK-18967 URL: https://issues.apache.org/jira/browse/SPARK-18967

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765322#comment-15765322 ] Imran Rashid commented on SPARK-18886: -- You understand correctly -- that is precisely what I'm

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762296#comment-15762296 ] Imran Rashid commented on SPARK-18886: -- Thanks [~mridul], that helps -- in particular I was only

[jira] [Updated] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-18886: - Priority: Major (was: Blocker) > Delay scheduling should not delay some executors indefinitely

[jira] [Updated] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-18886: - Priority: Blocker (was: Major) > Delay scheduling should not delay some executors indefinitely

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752722#comment-15752722 ] Imran Rashid commented on SPARK-18886: -- [~mridul] sorry if I am being slow here, but do you mind

[jira] [Resolved] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-8425. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 14079

[jira] [Updated] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-8425: Assignee: Mao, Wei (was: Imran Rashid) > Add blacklist mechanism for task scheduling >

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752462#comment-15752462 ] Imran Rashid commented on SPARK-18886: -- [~mridulm80] good point, perhaps the right answer here is

[jira] [Updated] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-18886: - Description: Delay scheduling can introduce an unbounded delay and underutilization of cluster

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752451#comment-15752451 ] Imran Rashid commented on SPARK-18886: -- [~markhamstra] I'm not really sure yet, I can't really

[jira] [Updated] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-18886: - Description: Delay scheduling can introduce an unbounded delay and underutilization of cluster

[jira] [Comment Edited] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751956#comment-15751956 ] Imran Rashid edited comment on SPARK-18886 at 12/15/16 5:32 PM: Here's a

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751956#comment-15751956 ] Imran Rashid commented on SPARK-18886: -- Here's a failing test case: (you can check it out directly

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15751949#comment-15751949 ] Imran Rashid commented on SPARK-18886: -- cc [~kayousterhout] [~markhamstra] [~zsxwing] > Delay

[jira] [Updated] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-18886: - Description: Delay scheduling can introduce an unbounded delay and underutilization of cluster

[jira] [Created] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

Imran Rashid created SPARK-18886: Summary: Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout Key: SPARK-18886 URL:

[jira] [Resolved] (SPARK-18846) Fix flakiness in SchedulerIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-18846. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 16270

[jira] [Created] (SPARK-18846) Fix flakiness in SchedulerIntegrationSuite

Imran Rashid created SPARK-18846: Summary: Fix flakiness in SchedulerIntegrationSuite Key: SPARK-18846 URL: https://issues.apache.org/jira/browse/SPARK-18846 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-16554) Spark should kill executors when they are blacklisted

[ https://issues.apache.org/jira/browse/SPARK-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-16554: - Assignee: Jose Soltren > Spark should kill executors when they are blacklisted >

[jira] [Updated] (SPARK-16654) UI Should show blacklisted executors & nodes

[ https://issues.apache.org/jira/browse/SPARK-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-16654: - Assignee: Jose Soltren > UI Should show blacklisted executors & nodes >

[jira] [Commented] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715719#comment-15715719 ] Imran Rashid commented on SPARK-15815: -- [~SuYan] I've been mulling this over for a while, and I

[jira] [Resolved] (SPARK-18117) Add Test for Interaction of TaskSchedulerImpl with TaskSetBlacklist

[ https://issues.apache.org/jira/browse/SPARK-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-18117. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 15644

[jira] [Created] (SPARK-18117) Add Test for Interaction of TaskSchedulerImpl with TaskSetBlacklist

Imran Rashid created SPARK-18117: Summary: Add Test for Interaction of TaskSchedulerImpl with TaskSetBlacklist Key: SPARK-18117 URL: https://issues.apache.org/jira/browse/SPARK-18117 Project: Spark

[jira] [Commented] (SPARK-17911) Scheduler does not need messageScheduler for ResubmitFailedStages

[ https://issues.apache.org/jira/browse/SPARK-17911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572344#comment-15572344 ] Imran Rashid commented on SPARK-17911: -- bq. In other words, handling a ResubmitFailedStages event

[jira] [Comment Edited] (SPARK-17911) Scheduler does not need messageScheduler for ResubmitFailedStages

[ https://issues.apache.org/jira/browse/SPARK-17911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572289#comment-15572289 ] Imran Rashid edited comment on SPARK-17911 at 10/13/16 3:44 PM: Copying

[jira] [Commented] (SPARK-17911) Scheduler does not need messageScheduler for ResubmitFailedStages

[ https://issues.apache.org/jira/browse/SPARK-17911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572289#comment-15572289 ] Imran Rashid commented on SPARK-17911: -- Copying the earlier discussion on the PR here from squito:

[jira] [Updated] (SPARK-17911) Scheduler does not need messageScheduler for ResubmitFailedStages

[ https://issues.apache.org/jira/browse/SPARK-17911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-17911: - Summary: Scheduler does not need messageScheduler for ResubmitFailedStages (was: Scheduler does

[jira] [Created] (SPARK-17911) Scheduler does not messageScheduler for ResubmitFailedStages

Imran Rashid created SPARK-17911: Summary: Scheduler does not messageScheduler for ResubmitFailedStages Key: SPARK-17911 URL: https://issues.apache.org/jira/browse/SPARK-17911 Project: Spark

[jira] [Resolved] (SPARK-17675) Add Blacklisting of Executors & Nodes within one TaskSet

[ https://issues.apache.org/jira/browse/SPARK-17675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-17675. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 15249

[jira] [Updated] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-8425: Attachment: DesignDocforBlacklistMechanism.pdf Seems like there is agreement on the design, so I'm

[jira] [Reopened] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reopened SPARK-15815: -- [~sowen] I think "won't fix" is the wrong resolution here. This is a real bug that should be

[jira] [Created] (SPARK-17676) FsHistoryProvider should ignore hidden files

Imran Rashid created SPARK-17676: Summary: FsHistoryProvider should ignore hidden files Key: SPARK-17676 URL: https://issues.apache.org/jira/browse/SPARK-17676 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524811#comment-15524811 ] Imran Rashid edited comment on SPARK-8425 at 9/27/16 2:20 AM: -- Breaking off a

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524811#comment-15524811 ] Imran Rashid commented on SPARK-8425: - Breaking off a smaller chunk of this that can be added

[jira] [Created] (SPARK-17675) Add Blacklisting of Executors & Nodes within one TaskSet

Imran Rashid created SPARK-17675: Summary: Add Blacklisting of Executors & Nodes within one TaskSet Key: SPARK-17675 URL: https://issues.apache.org/jira/browse/SPARK-17675 Project: Spark

[jira] [Commented] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15517344#comment-15517344 ] Imran Rashid commented on SPARK-17648: -- in this particular case, the index is used because there are

[jira] [Updated] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

[ https://issues.apache.org/jira/browse/SPARK-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-17648: - Description: {{TaskSchedulerImpl.resourceOffer}} takes in a {{Seq[WorkerOffer]}}. however,

[jira] [Created] (SPARK-17648) TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq

Imran Rashid created SPARK-17648: Summary: TaskSchedulerImpl.resourceOffers should take an IndexedSeq, not a Seq Key: SPARK-17648 URL: https://issues.apache.org/jira/browse/SPARK-17648 Project: Spark

[jira] [Created] (SPARK-17623) Failed tasks end reason is always a TaskFailedReason, types should reflect this

Imran Rashid created SPARK-17623: Summary: Failed tasks end reason is always a TaskFailedReason, types should reflect this Key: SPARK-17623 URL: https://issues.apache.org/jira/browse/SPARK-17623

[jira] [Commented] (SPARK-14560) Cooperative Memory Management for Spillables

[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432883#comment-15432883 ] Imran Rashid commented on SPARK-14560: -- One minor clarification -- SPARK-4452 is not included in

[jira] [Commented] (SPARK-2666) Always try to cancel running tasks when a stage is marked as zombie

[ https://issues.apache.org/jira/browse/SPARK-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386524#comment-15386524 ] Imran Rashid commented on SPARK-2666: - [~tgraves] [~lianhuiwang]. When there is a fetch failure,

[jira] [Commented] (SPARK-16654) UI Should show blacklisted executors & nodes

[ https://issues.apache.org/jira/browse/SPARK-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386242#comment-15386242 ] Imran Rashid commented on SPARK-16654: -- cc [~tgraves] > UI Should show blacklisted executors &

[jira] [Created] (SPARK-16654) UI Should show blacklisted executors & nodes

Imran Rashid created SPARK-16654: Summary: UI Should show blacklisted executors & nodes Key: SPARK-16654 URL: https://issues.apache.org/jira/browse/SPARK-16654 Project: Spark Issue Type:

[jira] [Updated] (SPARK-16554) Spark should kill executors when they are blacklisted

[ https://issues.apache.org/jira/browse/SPARK-16554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-16554: - Description: SPARK-8425 will allow blacklisting faulty executors and nodes. However, these

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384719#comment-15384719 ] Imran Rashid commented on SPARK-8425: - [~tgraves] I don't know of any jira for that. Sounds like you

[jira] [Created] (SPARK-16554) Spark should kill executors when they are blacklisted

Imran Rashid created SPARK-16554: Summary: Spark should kill executors when they are blacklisted Key: SPARK-16554 URL: https://issues.apache.org/jira/browse/SPARK-16554 Project: Spark Issue

[jira] [Commented] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15377283#comment-15377283 ] Imran Rashid commented on SPARK-15815: -- [~SuYan] I've been thinking about this more, and I agree

[jira] [Updated] (SPARK-14138) Generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames

[ https://issues.apache.org/jira/browse/SPARK-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-14138: - Fix Version/s: 1.6.2 > Generated SpecificColumnarIterator code can exceed JVM size limit for

[jira] [Resolved] (SPARK-15865) Blacklist should not result in job hanging with less than 4 executors

[ https://issues.apache.org/jira/browse/SPARK-15865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-15865. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 13603

[jira] [Assigned] (SPARK-16106) TaskSchedulerImpl does not correctly handle new executors on existing hosts

[ https://issues.apache.org/jira/browse/SPARK-16106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-16106: Assignee: Imran Rashid > TaskSchedulerImpl does not correctly handle new executors on

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353277#comment-15353277 ] Imran Rashid commented on SPARK-8425: - Adding the design doc from SPARK-8426 here:

[jira] [Resolved] (SPARK-16106) TaskSchedulerImpl does not correctly handle new executors on existing hosts

[ https://issues.apache.org/jira/browse/SPARK-16106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-16106. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 13826

[jira] [Resolved] (SPARK-16136) Flaky Test: TaskManagerSuite "Kill other task attempts when one attempt belonging to the same task succeeds"

[ https://issues.apache.org/jira/browse/SPARK-16136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-16136. -- Resolution: Fixed Fix Version/s: 2.1.0 Fixed by

[jira] [Updated] (SPARK-15963) `TaskKilledException` is not correctly caught in `Executor.TaskRunner`

[ https://issues.apache.org/jira/browse/SPARK-15963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-15963: - Description: Before this change, if either of the following cases happened to a task , the task

[jira] [Resolved] (SPARK-15963) `TaskKilledException` is not correctly caught in `Executor.TaskRunner`

[ https://issues.apache.org/jira/browse/SPARK-15963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-15963. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 13685

[jira] [Updated] (SPARK-15963) `TaskKilledException` is not correctly caught in `Executor.TaskRunner`

[ https://issues.apache.org/jira/browse/SPARK-15963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-15963: - Assignee: Liwei Lin > `TaskKilledException` is not correctly caught in `Executor.TaskRunner` >

[jira] [Created] (SPARK-16139) Audit tests for leaked threads

Imran Rashid created SPARK-16139: Summary: Audit tests for leaked threads Key: SPARK-16139 URL: https://issues.apache.org/jira/browse/SPARK-16139 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-16136) Flaky Test: TaskManagerSuite "Kill other task attempts when one attempt belonging to the same task succeeds"

Imran Rashid created SPARK-16136: Summary: Flaky Test: TaskManagerSuite "Kill other task attempts when one attempt belonging to the same task succeeds" Key: SPARK-16136 URL:

[jira] [Assigned] (SPARK-15783) Fix more flakiness: o.a.s.scheduler.BlacklistIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-15783: Assignee: Imran Rashid > Fix more flakiness: o.a.s.scheduler.BlacklistIntegrationSuite >

[jira] [Resolved] (SPARK-15783) Fix more flakiness: o.a.s.scheduler.BlacklistIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-15783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-15783. -- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request 13565

[jira] [Commented] (SPARK-16106) TaskSchedulerImpl does not correctly handle new executors on existing hosts

[ https://issues.apache.org/jira/browse/SPARK-16106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342821#comment-15342821 ] Imran Rashid commented on SPARK-16106: -- cc [~kayousterhout] After taking a closer look at this, I

[jira] [Updated] (SPARK-16106) TaskSchedulerImpl does not correctly handle new executors on existing hosts

[ https://issues.apache.org/jira/browse/SPARK-16106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-16106: - Priority: Trivial (was: Major) > TaskSchedulerImpl does not correctly handle new executors on

[jira] [Created] (SPARK-16106) TaskSchedulerImpl does not correctly handle new executors on existing hosts

Imran Rashid created SPARK-16106: Summary: TaskSchedulerImpl does not correctly handle new executors on existing hosts Key: SPARK-16106 URL: https://issues.apache.org/jira/browse/SPARK-16106 Project:

[jira] [Assigned] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-8425: --- Assignee: Imran Rashid > Add blacklist mechanism for task scheduling >

[jira] [Resolved] (SPARK-8426) Add blacklist mechanism for YARN container allocation

[ https://issues.apache.org/jira/browse/SPARK-8426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-8426. - Resolution: Duplicate yeah, agreed Kay. I don't think there is any reason for the distinction

[jira] [Resolved] (SPARK-14560) Cooperative Memory Management for Spillables

[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-14560. -- Resolution: Duplicate it was already fixed, just marking as a duplicate of SPARK-4452 to be a

[jira] [Reopened] (SPARK-14560) Cooperative Memory Management for Spillables

[ https://issues.apache.org/jira/browse/SPARK-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reopened SPARK-14560: -- > Cooperative Memory Management for Spillables > > >

[jira] [Commented] (SPARK-15976) Make Stage Numbering determinstic

[ https://issues.apache.org/jira/browse/SPARK-15976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15332686#comment-15332686 ] Imran Rashid commented on SPARK-15976: -- cc [~kayousterhout] [~markhamstra] > Make Stage Numbering

[jira] [Created] (SPARK-15976) Make Stage Numbering determinstic

Imran Rashid created SPARK-15976: Summary: Make Stage Numbering determinstic Key: SPARK-15976 URL: https://issues.apache.org/jira/browse/SPARK-15976 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331907#comment-15331907 ] Imran Rashid commented on SPARK-15815: -- yes, I agree that SPARK-15865 isn't ideal on its own.

[jira] [Commented] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331132#comment-15331132 ] Imran Rashid commented on SPARK-15815: -- [~SuYan] is this the same as

[jira] [Updated] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-15815: - Component/s: Scheduler > Hang while enable blacklistExecutor and DynamicExecutorAllocator >

[jira] [Created] (SPARK-15878) Fix test cleanup in EventLoggingListenerSuite and ReplayListenerSuite

Imran Rashid created SPARK-15878: Summary: Fix test cleanup in EventLoggingListenerSuite and ReplayListenerSuite Key: SPARK-15878 URL: https://issues.apache.org/jira/browse/SPARK-15878 Project: Spark

<    2   3   4   5   6   7   8   9   10   >