Github user lovexi closed the pull request at:
https://github.com/apache/spark/pull/14428
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user lovexi closed the pull request at:
https://github.com/apache/spark/pull/14338
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14254#discussion_r77283751
--- Diff: docs/monitoring.md ---
@@ -350,6 +350,7 @@ set of sinks to which metrics are reported. The
following instances are currentl
* `worker
Github user lovexi closed the pull request at:
https://github.com/apache/spark/pull/14110
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14428
I think this one is ready for reviews. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14110
Ping
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14428
Oh, my bad. Added unit test in `MetricsSystemSuite ` for this already.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14428
Reopen PR #14415 @rxin
>> DId you add a test?
I was trying to add a unit test for this feature. But the test sinks are
not visible for MetricsSystem when it tries to registe
GitHub user lovexi opened a pull request:
https://github.com/apache/spark/pull/14428
[SPARK-16810] Refactor registerSinks with multiple constructos
## What changes were proposed in this pull request?
For some metrics, it may require some **app detailed information** from
Github user lovexi closed the pull request at:
https://github.com/apache/spark/pull/14415
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14415
Oops. I kind of messed up here. Going to open another PR for this. Close
this for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14415
@rxin I was trying to add a unit test for this feature. But the test sinks
are not visible for `MetricsSystem` when it tries to register test sink classes
since test sinks are hidden in test
GitHub user lovexi opened a pull request:
https://github.com/apache/spark/pull/14415
[SPARK-16810] Refactor registerSinks with multiple constructos
## What changes were proposed in this pull request?
For some metrics, it may require some **app detailed information** from
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14110#discussion_r72828906
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
---
@@ -345,6 +351,8 @@ class
GitHub user lovexi opened a pull request:
https://github.com/apache/spark/pull/14338
Make parameters configurable in BlockManager
## What changes were proposed in this pull request?
Make parameters configurable in BlockManager class, such as max_attempts
and sleep_time
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14254
Test it please, Jenkins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
GitHub user lovexi opened a pull request:
https://github.com/apache/spark/pull/14254
Add shuffle service metrics entry in monitoring docs
## What changes were proposed in this pull request?
Add shuffle service metrics entry in currently supporting metrics list
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14080
@rxin Thank you for mentioning that. I even didn't notice that. Haha.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14080
Thank you for reminding me this. Already updated PR description and add
more details.
cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14080
Restart test, please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14080
@rxin Thank you for letting me know this. That saves me a lot of time on
testing..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14080
test this, please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14080
test it, please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70171996
--- Diff:
core/src/main/scala/org/apache/spark/deploy/ExternalShuffleServiceSource.scala
---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70171933
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -79,32 +89,59 @@ protected void
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70171938
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -143,4 +180,29 @@ private void
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70171931
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -52,6 +59,8
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70171929
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -79,32 +89,59 @@ protected void
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70171924
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -143,4 +180,29 @@ private void
GitHub user lovexi opened a pull request:
https://github.com/apache/spark/pull/14110
[SPARK-16455] Add isClusterAvailableForNewOffers hook to check cluster
availability
## What changes were proposed in this pull request?
In our case, we are implementing restartable
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14080
Added some test cases to verify expected metrics values @ericl
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70115043
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -143,4 +179,26 @@ private void
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70114491
--- Diff:
core/src/main/scala/org/apache/spark/deploy/ExternalShuffleServiceSource.scala
---
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70112425
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -93,18 +113,34 @@ protected void
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70111868
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -64,6 +75,10 @@ public
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70110638
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -93,18 +113,34 @@ protected void
Github user lovexi commented on a diff in the pull request:
https://github.com/apache/spark/pull/14080#discussion_r70109683
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockHandler.java
---
@@ -143,4 +179,26 @@ private void
Github user lovexi commented on the issue:
https://github.com/apache/spark/pull/14080
@rxin Sure. Get a cleaner title instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user lovexi opened a pull request:
https://github.com/apache/spark/pull/14080
[SPARK-16405] [ShuffleService] [Metrics] Add metrics and source for
external shuffle service
## What changes were proposed in this pull request?
ExternalShuffleService is essential
39 matches
Mail list logo
