Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r89163640
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,211 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r89166279
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -17,27 +17,178 @@
package org.apache.spark.scheduler
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r89166493
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -139,29 +139,6 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r89167206
--- Diff:
mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackendSuite.scala
---
@@ -345,7 +374,17 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r89166210
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala ---
@@ -17,27 +17,178 @@
package org.apache.spark.scheduler
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r89166735
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -592,47 +579,6 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15644
thanks @kayousterhout , merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
I agree with Kay that putting in a smaller change first is better, assuming
it still has the performance gains. That doesn't preclude any further
optimizations that are bigger changes.
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15982#discussion_r90127514
--- Diff:
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
---
@@ -75,13 +75,6 @@
@Mock(answer = RETURNS_SMART_NULLS
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15982#discussion_r90126726
--- Diff:
core/src/test/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriterSuite.java
---
@@ -40,9 +41,11 @@
import org.mockito.stubbing.Answer
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15982#discussion_r90121766
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/UnsafeShuffleWriter.java ---
@@ -337,42 +340,47 @@ void forceSorterToSpill() throws IOException
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15982#discussion_r90127615
--- Diff:
core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java
---
@@ -86,14 +88,7 @@ public int compare
Github user squito commented on the issue:
https://github.com/apache/spark/pull/14079
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/15644
[SPARK-18117][CORE] Add test for TaskSetBlacklist
## What changes were proposed in this pull request?
This adds tests to verify the interaction between TaskSetBlacklist and
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r85125936
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +331,211 @@ class TaskSchedulerImplSuite extends
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15644
@kayousterhout here are the tests that should have been in SPARK-17675.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86077923
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86078546
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86078676
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15644#discussion_r86078940
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala ---
@@ -282,6 +316,114 @@ class TaskSchedulerImplSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r84380686
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +179,76 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86383782
--- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
@@ -104,10 +105,26 @@ class ShuffledRDD[K: ClassTag, V: ClassTag, C:
ClassTag
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86384061
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -22,17 +22,36 @@ import java.io.ObjectInputStream
import java.util
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86384551
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -306,18 +428,19 @@ class LongAccumulator extends AccumulatorV2[jl.Long
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86384641
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -383,16 +506,7 @@ class DoubleAccumulator extends
AccumulatorV2[jl.Double
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86381839
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -42,18 +60,43 @@ private[spark] case class AccumulatorMetadata(
* `OUT
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86483290
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -306,18 +428,19 @@ class LongAccumulator extends AccumulatorV2[jl.Long
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86484498
--- Diff:
core/src/test/scala/org/apache/spark/DataPropertyAccumulatorSuite.scala ---
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86484995
--- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
@@ -104,10 +105,26 @@ class ShuffledRDD[K: ClassTag, V: ClassTag, C:
ClassTag
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86487826
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86487362
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86486669
--- Diff: core/src/main/scala/org/apache/spark/TaskContext.scala ---
@@ -69,6 +69,20 @@ object TaskContext {
}
}
+/**
+ * Identifies
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86486810
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86486722
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86488263
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86486359
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -42,18 +60,45 @@ private[spark] case class AccumulatorMetadata(
* `OUT
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86486502
--- Diff: core/src/main/scala/org/apache/spark/TaskContextImpl.scala ---
@@ -126,4 +126,14 @@ private[spark] class TaskContextImpl
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86487664
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86486618
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86489315
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86487454
--- Diff: core/src/main/scala/org/apache/spark/TaskContext.scala ---
@@ -69,6 +69,20 @@ object TaskContext {
}
}
+/**
+ * Identifies
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86486890
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86486267
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -136,15 +181,92 @@ abstract class AccumulatorV2[IN, OUT] extends
Serializable
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16473#discussion_r96459828
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala ---
@@ -127,6 +127,14 @@ private[spark] object UIData {
def updateTaskMetrics
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16620
Thanks for pointing out this issue, and the nice description. Still
looking but sounds like a legitimate issue. I think
`SchedulerIntegrationSuite` should be able to replicate the exact scenario
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96667888
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +162,47 @@ class ExecutorsListener(storageStatusListener
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96667288
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +162,47 @@ class ExecutorsListener(storageStatusListener
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96668414
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -88,6 +91,15 @@ class BlacklistTrackerSuite extends SparkFunSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16346
the test failures look real and from the changes you've made, can you take
a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as wel
GitHub user squito opened a pull request:
https://github.com/apache/spark/pull/16639
[SPARK-19276][CORE] Fetch Failure handling robust to user error handling
## What changes were proposed in this pull request?
Fault-tolerance in spark requires special handling of shuffle
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16639
cc @kayousterhout @markhamstra @mateiz
This isn't just protecting against crazy user code -- I've seen users hit
this with spark sql (because of
https://github.com/apache/
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16346
merged to master, thanks @jsoltren!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96883996
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -244,32 +245,45 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r96889501
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -602,6 +619,20 @@ class
Github user squito commented on the issue:
https://github.com/apache/spark/pull/15505
This might be a really dumb question, but please humor me -- can you
explain to me the threads involved, both before and after this change? I must
be missing something, because it seems to me it
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16639#discussion_r96909723
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/FetchFailedException.scala ---
@@ -45,6 +45,12 @@ private[spark] class FetchFailedException
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16639#discussion_r96910555
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -405,6 +415,13 @@ private[spark] class Executor
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16639#discussion_r96910697
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -405,6 +415,13 @@ private[spark] class Executor
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16639
thanks for the feedback @mridulm , all good points. I pushed an update to
address some of the points, also have some follow up discussion
---
If your project is set up for it, you can reply to
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16473
thanks @paragpc . lgtm, but now there are merge conflicts. Can you merge
with the latest master? You might need to regenerate those test expectation
files.
---
If your project is set up for it
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11867#discussion_r96939526
--- Diff:
core/src/main/scala/org/apache/spark/status/api/v1/ApplicationListResource.scala
---
@@ -43,11 +45,24 @@ private[v1] class ApplicationListResource
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16473
ah, sorry I didn't notice that before my comment. well looks good now,
I'll merge once tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16473
merged to master, thanks @paragpc !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97162721
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -429,7 +429,7 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97162706
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -43,7 +43,7 @@ class BlacklistTrackerSuite extends SparkFunSuite
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16650
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97162877
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -719,7 +719,11 @@ private[spark] object TaskSchedulerImpl
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11867#discussion_r97392343
--- Diff: docs/monitoring.md ---
@@ -267,12 +267,23 @@ can be identified by their `[attempt-id]`. In the API
listed below, when running
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11867#discussion_r97391985
--- Diff:
core/src/main/scala/org/apache/spark/status/api/v1/ApplicationListResource.scala
---
@@ -43,11 +45,26 @@ private[v1] class ApplicationListResource
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15505#discussion_r97395812
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -602,6 +619,21 @@ class
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16620
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on the issue:
https://github.com/apache/spark/pull/11867
lgtm, will merge when tests pass
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user squito commented on the issue:
https://github.com/apache/spark/pull/11867
thanks @paragpc
merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16620#discussion_r97586026
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1193,7 +1193,14 @@ class DAGScheduler
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16620#discussion_r97417513
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala
---
@@ -648,4 +648,69 @@ class BasicSchedulerIntegrationSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16620#discussion_r97417399
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala
---
@@ -648,4 +648,69 @@ class BasicSchedulerIntegrationSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16620#discussion_r97406162
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1218,7 +1225,9 @@ class DAGScheduler(
logInfo
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16690
Jenkins, ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16690
cc @vanzin @zsxwing
(I was going to make a longer comment about removing `askWithRetry` and
whether we need another method, but then saw the comments on the referenced PR
-- I'll def
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97881374
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97881356
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97877539
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97878659
--- Diff:
core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala
---
@@ -528,6 +548,16 @@ class StandaloneDynamicAllocationSuite
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97877918
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -148,6 +153,12 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97874677
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala ---
@@ -59,6 +59,19 @@ private[spark] trait ExecutorAllocationClient
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97881401
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -148,6 +153,12 @@ class
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97875716
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -187,6 +206,22 @@ private[scheduler] class BlacklistTracker
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97875223
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -139,6 +139,11 @@ package object config {
.timeConf
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97875600
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16650
one more thought on tests -- your mock `ExecutorAllocationClient` could
make sure that when `killExecutorsOnHost` is called, the host is already in
`nodeBlacklist`, to avoid that last race you found
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98281426
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -187,6 +199,17 @@ private[scheduler] class BlacklistTracker
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98283223
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -17,7 +17,8 @@
package org.apache.spark.scheduler
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98281415
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -187,6 +199,17 @@ private[scheduler] class BlacklistTracker
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98281375
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -173,6 +174,17 @@ private[scheduler] class BlacklistTracker
Github user squito commented on the issue:
https://github.com/apache/spark/pull/16650
I think this is basically ready, perhaps one more test case. @tgravescs
any other comments?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98283147
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -456,4 +459,61 @@ class BlacklistTrackerSuite extends
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98281403
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -173,6 +174,17 @@ private[scheduler] class BlacklistTracker
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r98464730
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -102,38 +103,51 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/15237#discussion_r98464559
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala ---
@@ -102,38 +103,51 @@ private[spark] class FairSchedulableBuilder(val
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/16620#discussion_r98472355
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -718,6 +703,21 @@ private[spark] class TaskSetManager
301 - 400 of 3535 matches
Mail list logo
