Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r99661763
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -456,4 +461,69 @@ class BlacklistTrackerSuite extends
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r99660675
--- Diff:
core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala
---
@@ -489,6 +491,29 @@ class
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r99660908
--- Diff:
core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala
---
@@ -467,6 +469,51 @@ class
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r99660755
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -600,6 +603,16 @@ class
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r99661191
--- Diff:
core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala
---
@@ -467,6 +469,51 @@ class
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r99664910
--- Diff:
core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala
---
@@ -467,6 +469,52 @@ class
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/16650
@squito and I chatted about this some offline. Not only does
receiveAndReply in CoarseGrainedSchedulerBackend need to handle the situation
where it is trying to do a RegisterExecutor on an already
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97622188
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -43,7 +43,7 @@ class BlacklistTrackerSuite extends
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/16650
Working through some test failures in
org.apache.spark.deploy.StandaloneDynamicAllocationSuite and
org.apache.spark.HeartbeatReceiverSuite...
---
If your project is set up for it, you can reply
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97622417
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala ---
@@ -719,7 +719,11 @@ private[spark] object TaskSchedulerImpl
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97622210
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -429,7 +429,7 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r97625393
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -168,6 +169,21 @@ private[scheduler] class BlacklistTracker
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98079880
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -148,6 +153,12 @@ class
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98078786
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98078837
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98071059
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -139,6 +139,11 @@ package object config {
.timeConf
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98082949
--- Diff:
core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala
---
@@ -528,6 +548,16 @@ class
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98070834
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala ---
@@ -59,6 +59,19 @@ private[spark] trait ExecutorAllocationClient
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98086588
--- Diff:
core/src/main/scala/org/apache/spark/ExecutorAllocationClient.scala ---
@@ -59,6 +59,19 @@ private[spark] trait ExecutorAllocationClient
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98084608
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -139,6 +139,11 @@ package object config {
.timeConf
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16650#discussion_r98090656
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -173,6 +174,24 @@ private[scheduler] class BlacklistTracker
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/16650
My solution for the
`org.apache.spark.deploy.StandaloneDynamicAllocationSuite.kill all executors on
localhost` failure was to add a Boolean argument to killExecutorsOnHost
specifying if killed
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/16650
The error was:
`ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from
https://github.com/apache/spark.git`
I don't think that my
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96530866
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -88,6 +88,86 @@ class BlacklistTrackerSuite extends
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96525452
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +158,42 @@ class ExecutorsListener(storageStatusListener
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96525607
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +158,42 @@ class ExecutorsListener(storageStatusListener
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96526179
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -88,6 +88,86 @@ class BlacklistTrackerSuite extends
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96681232
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -88,6 +91,15 @@ class BlacklistTrackerSuite extends
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/16346
The test failures are due to s/nodeId/hostId/ in the JSON code but not the
tests. This is passing locally now with the changes I'm about to push to this
PR. Thanks.
---
If your project is set up
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96681207
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +162,47 @@ class ExecutorsListener(storageStatusListener
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r96681179
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +162,47 @@ class ExecutorsListener(storageStatusListener
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16521#discussion_r96719170
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/crypto/README.md
---
@@ -0,0 +1,158 @@
+Spark Auth Protocol and AES Encryption
GitHub user jsoltren opened a pull request:
https://github.com/apache/spark/pull/16650
[SPARK-16554][CORE] Automatically Kill Executors and Nodes when they are
Blacklisted
## What changes were proposed in this pull request?
In SPARK-8425, we introduced a mechanism
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r75392592
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -809,32 +821,65 @@ class TaskSetManagerSuite extends
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r75508919
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -90,6 +92,54 @@ package object config {
.toSequence
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/15281
There is just one change and it is to the section name:
Strike "Accumulators and Broadcast Variables"
Replace with "Accumulators, Broadcast Variables, and Checkpoints"
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14079#discussion_r75369742
--- Diff:
core/src/main/scala/org/apache/spark/internal/config/package.scala ---
@@ -90,6 +92,54 @@ package object config {
.toSequence
GitHub user jsoltren opened a pull request:
https://github.com/apache/spark/pull/15281
[DOCS] Reorganize explanation of Accumulators and Broadcast Variables
## What changes were proposed in this pull request?
The discussion of the interaction of Accumulators and Broadcast
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/14079
Great!
I've been working on some additional changes on top of this: UI
representation for blacklisted executors (SPARK-16654), and implicit killing of
blacklisted executors (SPARK-16554
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95650592
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala ---
@@ -40,7 +41,7 @@ class BlacklistTrackerSuite extends
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95646870
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -48,9 +48,14 @@ import org.apache.spark.util.{Clock, SystemClock
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95647082
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -115,6 +121,10 @@ private[scheduler] class BlacklistTracker
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95647163
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala ---
@@ -161,6 +171,9 @@ private[scheduler] class BlacklistTracker
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95647827
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -106,6 +106,28 @@ case class SparkListenerExecutorRemoved(time: Long
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95647957
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +158,44 @@ class ExecutorsListener(storageStatusListener
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95652075
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala ---
@@ -428,7 +428,7 @@ class TaskSetManagerSuite extends SparkFunSuite
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/16346#discussion_r95652263
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala
---
@@ -157,4 +158,44 @@ class ExecutorsListener(storageStatusListener
GitHub user jsoltren opened a pull request:
https://github.com/apache/spark/pull/16346
[SPARK-16654][CORE] Add UI coverage for Application Level Blacklisting
Builds on top of work in SPARK-8425 to update Application Level
Blacklisting in the scheduler.
## What changes were
GitHub user jsoltren opened a pull request:
https://github.com/apache/spark/pull/17402
[SPARK-7200] Add cleanup for memory leak tests to after() in TaskMemoâ¦
â¦ryManagerSuite
Cleans up all memory used by tests explicitly in after(). This new
function has the JUnit
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14617#discussion_r106470033
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -87,8 +87,13 @@ case class
SparkListenerEnvironmentUpdate
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14617#discussion_r106468328
--- Diff:
core/src/main/scala/org/apache/spark/storage/BlockManagerSource.scala ---
@@ -26,35 +26,39 @@ private[spark] class BlockManagerSource(val
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14617#discussion_r106467740
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -350,7 +366,12 @@ $(document).ready(function
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14617#discussion_r106467583
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage.js ---
@@ -174,8 +174,10 @@ $(document).ready(function
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14617#discussion_r106467285
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html
---
@@ -24,7 +24,10 @@ Summary
RDD
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14617#discussion_r106467368
--- Diff:
core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html
---
@@ -24,7 +24,10 @@ Summary
RDD
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14617#discussion_r106469513
--- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsPage.scala
---
@@ -116,7 +121,11 @@ private[spark] object ExecutorsPage
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/14617#discussion_r106469348
--- Diff: core/src/main/scala/org/apache/spark/storage/StorageUtils.scala
---
@@ -60,11 +63,17 @@ class StorageStatus(val blockManagerId: BlockManagerId
GitHub user jsoltren opened a pull request:
https://github.com/apache/spark/pull/17625
[SPARK-9103][WIP] Add Memory Tracking UI and track Netty memory usage
This patch resurrects https://github.com/apache/spark/pull/7753 by
liyezhang556520.
## What changes were proposed
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17140
This seems like a reasonable approach to me, so long as writing the length
of the string out doesn't break anything else (it doesn't appear to).
---
If your project is set up for it, you can
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/14617
This looks good to me. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17694
This seems reasonable to me.
There are some typos in the PR description. I think you meant "pickled"
instead of "picked" in a few places.
Using threading.Lock
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17625
This PR was closed, so, I'll create a new one focusing on just the back end
pieces. I'll create a fresh JIRA for more general memory tracking improvements
to the UI where we can hash out more
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/18395
This looks fine to me. Though, I did have to ask Marcelo a number of
questions along the way to clarify what all was going on here. Maybe some of
the points I mention below should make their way
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/18604
My preference is to backport this and other blacklisting related fixes as
far back as possible on Spark2 - meaning 2.1 and 2.0 as well, unless convinced
otherwise. So, yes, @cloud-fan, I hope we
Github user jsoltren closed the pull request at:
https://github.com/apache/spark/pull/18604
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/18604
ð
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/18594#discussion_r128871196
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -473,29 +473,36 @@ private[spark] class Executor
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/18594#discussion_r128873660
--- Diff: core/src/main/scala/org/apache/spark/executor/Executor.scala ---
@@ -473,29 +473,36 @@ private[spark] class Executor
GitHub user jsoltren opened a pull request:
https://github.com/apache/spark/pull/18604
[SPARK-21219][CORE] Task retry occurs on same executor due to race coâ¦
â¦ndition with blacklisting
There's a race condition in the current TaskSetManager where a failed task
is added
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17625
I'm still making updates here. It would be ideal if one of the admins could
please re-open this PR. Otherwise I will open a new one. Thanks!
---
If your project is set up for it, you can reply
GitHub user jsoltren opened a pull request:
https://github.com/apache/spark/pull/17762
[SPARK-9103][WIP] Track Netty memory usage - take two
This patch resurrects #7753 by liyezhang556520 and #17625 by jsoltren.
## What changes were proposed in this pull request
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17402
I'm still waiting on feedback from @joshrosen on my earlier question. I can
update this patch in response to @vanzin's review feedback if it would help to
get this checked in.
IMO
Github user jsoltren closed the pull request at:
https://github.com/apache/spark/pull/17762
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17762
To close the loop here: I'm going to rework these ideas into a new JIRA
that I'll file, to track *total* memory usage in the UI.
---
If your project is set up for it, you can reply to this email
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/17902#discussion_r118757763
--- Diff:
common/kvstore/src/test/java/org/apache/spark/kvstore/DBIteratorSuite.java ---
@@ -0,0 +1,498 @@
+/*
+ * Licensed to the Apache Software
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/17902#discussion_r118757899
--- Diff:
common/kvstore/src/test/java/org/apache/spark/kvstore/LevelDBSuite.java ---
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17902
> >For example, explaining why indices can take lists of elements
> ?
That was in the comment for 52ed2b4 earlier: "This is needed because some
UI types have comp
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17902
Thanks @vanzin. I agree that it's tough to split up a change in such a way
that it is easy to review. This approach - intermediate diffs in one PR, and a
set of PRs for the whole feature - seems
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17402
At this point I'd like to get this checked in whether or not @JoshRosen
chimes in here. @HyukjinKwon @vanzin I've updated this based on the previous
review feedback received. Thanks!
---
If your
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/17402#discussion_r116881319
--- Diff:
core/src/test/java/org/apache/spark/memory/TaskMemoryManagerSuite.java ---
@@ -25,27 +26,37 @@
public class TaskMemoryManagerSuite
Github user jsoltren commented on a diff in the pull request:
https://github.com/apache/spark/pull/17402#discussion_r116883362
--- Diff:
core/src/test/java/org/apache/spark/memory/TaskMemoryManagerSuite.java ---
@@ -25,27 +26,37 @@
public class TaskMemoryManagerSuite
Github user jsoltren commented on the issue:
https://github.com/apache/spark/pull/17402
Yes, I'll post an updated change once I wrap up a couple of other things...
> Seems like the scope of the bug was a little larger than you thought.
@vanzin The scope of the
82 matches
Mail list logo
