Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/23242#discussion_r239994508
--- Diff:
examples/src/main/scala/org/apache/spark/examples/AccumulatorMetricsTest.scala
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237652232
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -708,16 +709,36 @@ private[spark] class PythonBroadcast(@transient var
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237250388
--- Diff: python/pyspark/broadcast.py ---
@@ -134,7 +137,15 @@ def value(self):
""" Return the br
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/23166#discussion_r237231750
--- Diff: python/pyspark/broadcast.py ---
@@ -118,8 +121,16 @@ def dump(self, value, f):
f.close()
def load_from_path(self
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/23166
@squito yeah looking at the code we could maintain a soft reference so that
we dont have to keep requesting it from the disk. It is definitely an
optimization
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/23166
[SPARK-26201] Fix python broadcast with encryption
## What changes were proposed in this pull request?
Python with rpc and disk encryption enabled along with a python broadcast
variable
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r227118510
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
---
@@ -77,17 +82,54 @@
private static final
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/21402
@cloud-fan yes we can close this
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22628
Thanks @tgravescs for explaining the issue
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22628
@tgravescs @abellina plz take a look thanks
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/22628
Change the spark.shuffle.server.chunkFetchHandlerThreadsPercent default to
100
## What changes were proposed in this pull request?
We want to change the default percentage to 100
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22173
closes #21402
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22173
test this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218942035
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
---
@@ -77,17 +82,54 @@
private static final
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218939333
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
---
@@ -98,21 +98,32 @@ public TransportContext
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218930135
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
---
@@ -77,17 +82,43 @@
private static final
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218621553
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
---
@@ -0,0 +1,136
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218590278
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
---
@@ -0,0 +1,136
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218559203
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
---
@@ -281,4 +282,31 @@ public Properties cryptoConf
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218163618
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
---
@@ -0,0 +1,136
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216074852
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
---
@@ -0,0 +1,136
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216069578
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
---
@@ -281,4 +282,31 @@ public Properties cryptoConf
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216068689
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
---
@@ -281,4 +282,31 @@ public Properties cryptoConf
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22173
thanks @vanzin, also @tgravescs gentle ping...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r212639516
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/TransportContext.java
---
@@ -144,14 +161,17 @@ public TransportChannelHandler
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22173
@tgravescs @vanzin @Victsm please chime in thanks
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/22173
[SPARK-24335] Spark external shuffle server improvement to better handle
block fetch requests.
## What changes were proposed in this pull request?
This is a continuation PR from https
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/21402
@Victsm @vanzin i want to get this going, is it better if I have a PR up
with the requested changes and concerns
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/21636
Spark 24533
## What changes were proposed in this pull request?
Typesafe has rebranded to lightbend. Just changing the downloads path to
avoid redirection
## How was this patch
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/21475#discussion_r194760044
--- Diff: docs/configuration.md ---
@@ -1629,8 +1629,10 @@ Apart from these, the following properties are also
available, and may be useful
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/21475
@tgravescs @squito fixed the description hope you can take a look thanks
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/21475
That description might work too @squito If that is ok with @tgravescs I can
rephrase it... Just wanted it to be explicit
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/21468
LGTM @pgandhi999 Hope @tgravescs can confirm it
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/21476#discussion_r192804713
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
---
@@ -1485,6 +1486,22 @@ private object Client extends
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/21475
@tgravescs plz review thanks
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/21475
[SPARK-24416] Fix configuration specification for killBlacklisted executors
## What changes were proposed in this pull request?
spark.blacklist.killBlacklistedExecutors is defined
Github user redsanket closed the pull request at:
https://github.com/apache/spark/pull/19103
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/19140
Added principal check back and tested in secure hadoop env. Let me know if
this looks fine with you @jerryshao @vanzin @tgravescs
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/19140
@jerryshao yes will do no issues thanks
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/19140
Previous discussion on this PR is here
https://github.com/apache/spark/pull/19103
---
-
To unsubscribe, e-mail: reviews
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/19140#discussion_r137096611
--- Diff:
core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala
---
@@ -103,15 +103,17 @@ private[deploy] class
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/19103
@vanzin @tgravescs sorry for the delay, will put up a PR against master, we
can move further discussion there, about the suggested improvements, I put up a
PR against master just for workaround
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/19140
[SPARK-21890] Credentials not being passed to add the tokens
I observed this while running a oozie job trying to connect to hbase via
spark.
It look like the creds are not being passed
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/19103
[SPARK-21890]
## What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-21890
Modified to pass creds to get delegation tokens
## How
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/18940
@vanzin addressed the config comment thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/18940
Thanks @vanzin @kiszk will do, makes sense to me now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/18940
@kiszk wouldn't the updated release notes/docs take care of that, which
configs can no longer be used and which are not. I don't mind adding a warning
msg saying please use another cache.size
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/18940
@kiszk I dont think that would be ideal, it is better to backport the
feature itself to a desired version or branch, having two conflicting configs
for the same task is not ideal, if that is what
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/18940
@dbolshak there were no unit tests for google cache implementation here
before, I could add a simple test to check for cache behavior if it is
necessary but ideally a scale test is necessary
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/18940#discussion_r133220047
--- Diff:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
---
@@ -104,15 +105,22 @@ public
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/18940
YSPARK-734 Change CacheLoader to limit entries based on memory footprint
Right now the spark shuffle service has a cache for index files. It is
based on a # of files cached
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
@tgravescs @vanzin ready for merge?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r115097978
--- Diff:
core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
---
@@ -22,6 +23,7 @@
"duration"
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r115097526
--- Diff:
core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
---
@@ -22,6 +23,7 @@
"duration"
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r115096536
--- Diff:
core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
---
@@ -22,6 +23,7 @@
"duration"
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r114924015
--- Diff:
core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
---
@@ -22,6 +23,7 @@
"duration"
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r114921697
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala ---
@@ -283,10 +283,15 @@ private[spark] object EventLoggingListener
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r114921013
--- Diff:
core/src/test/resources/HistoryServerExpectations/completed_app_list_json_expectation.json
---
@@ -22,6 +23,7 @@
"duration"
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
I think I should set up my IDE would be nice to have something like
checkstyle.xml instead of configuring these things, sorry abt that
---
If your project is set up for it, you can reply
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
SparkContext was not able to read SparkListenerLogStart event as it is not
a part of it and the subsequent replay listener suite tries to compare
eventLogs and original events emitted via
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r114647969
--- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala ---
@@ -60,6 +60,8 @@ private[spark] class SparkUI private (
var appId
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
The issue before was that I was replaying the events before adding the
listeners in the getSparkUI.
That caused a bunch of tests to fail and other files were to do with the
appSparkVersion
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
ok looks like I did not modify the expectation.json files will fix them
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
ok will add it to either RuntimeInfo or ApplicationInfo. I thought it might
break the contract underneath but if it doesn't then I should add it here
thanks for the input. Will do
---
If your
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
@vanzin Can I add this to SparkConf.scala
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkConf.scala#L58
just to have the application info here it will have
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r113299115
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ApplicationEventListener.scala
---
@@ -57,4 +58,10 @@ private[spark] class
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r113298757
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/ApplicationEventListener.scala
---
@@ -57,4 +58,10 @@ private[spark] class
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r112726055
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala ---
@@ -160,7 +160,6 @@ case class SparkListenerApplicationEnd(time: Long
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/17658
@vanzin sure will address the concerns thanks for the review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r112268780
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala ---
@@ -71,7 +71,6 @@ private[spark] trait SparkListenerBus
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/17658#discussion_r112067747
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala ---
@@ -71,7 +71,6 @@ private[spark] trait SparkListenerBus
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/17658
[SPARK-20355] Add per application spark version on the history server
headerpage
## What changes were proposed in this pull request?
Spark Version for a specific application
Github user redsanket closed the pull request at:
https://github.com/apache/spark/pull/12152
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user redsanket commented on the pull request:
https://github.com/apache/spark/pull/12152#issuecomment-205611695
@vanzin The idea was to make it pluggable to be used by other projects if
neccessary. If we do not want to have additional information regarding the
build it can
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/12152
[SPARK-14279] Spark Version will be picked from pom.version
## What changes were proposed in this pull request?
This PR proposes to pick the version information for SPARK from pom.version
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/10838#discussion_r53812741
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -328,7 +345,9 @@ final class ShuffleBlockFetcherIterator
Github user redsanket commented on the pull request:
https://github.com/apache/spark/pull/10838#issuecomment-182911828
@zsxwing rebased and changed ArrayBuffer to HashSet
@tgravescs might want to take a look at it one more time
---
If your project is set up for it, you can reply
Github user redsanket commented on the pull request:
https://github.com/apache/spark/pull/10838#issuecomment-183063914
@zsxwing addressed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/10838#discussion_r52518306
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -143,9 +148,11 @@ final class ShuffleBlockFetcherIterator
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/10838#discussion_r52564828
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -143,9 +148,11 @@ final class ShuffleBlockFetcherIterator
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/10838#discussion_r52542439
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -143,9 +148,11 @@ final class ShuffleBlockFetcherIterator
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/10838#discussion_r52541897
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -143,9 +148,11 @@ final class ShuffleBlockFetcherIterator
Github user redsanket commented on the pull request:
https://github.com/apache/spark/pull/10838#issuecomment-178209754
@zsxwing Updated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/10838#discussion_r50576595
--- Diff:
core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala
---
@@ -258,6 +268,9 @@ final class ShuffleBlockFetcherIterator
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/10838
[SPARK-6166] Limit number of concurrent outbound connections
This JIRA is related to
https://github.com/apache/spark/pull/5852
Had to do some minor rework and test to make sure
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/10838#discussion_r50208257
--- Diff: docs/configuration.md ---
@@ -392,6 +392,17 @@ Apart from these, the following properties are also
available, and may be useful
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/10838#discussion_r50208175
--- Diff: core/src/main/scala/org/apache/spark/SecurityManager.scala ---
@@ -81,7 +81,7 @@ import org.apache.spark.util.Utils
* - HTTP for broadcast
89 matches
Mail list logo
