[GitHub] spark pull request #19227: [SPARK-20060][CORE] Support accessing secure Hado...
GitHub user jerryshao opened a pull request:
https://github.com/apache/spark/pull/19227
[SPARK-20060][CORE] Support accessing secure Hadoop cluster in standalone
client mode
## What changes were proposed in this pull request?
This PR leverages the facility of SPARK-16742 to support accessing security
Hadoop cluster in standalone client mode. The key idea is brought from
SPARK-16742, add a simple logics to support `HadoopDelegationTokenManager` in
`StandaloneSchedulerBackend`. Besides this PR also address related
docs/configuration issue:
1. Rename "spark.yarn.principal", "spark.yarn.keytab" to
"spark.security.principal" and "spark.security.keytab", since principal/keytab
is not only used in yarn mode, so rename it to be more meaningful.
2. Move the related docs from `running-on-yarn` page to `security` page,
since it is not a yarn only feature.
3. Change `spark-submit --help` output to be more precise about
`--principal` and `--keytab`.
## How was this patch tested?
The change is verified in local security cluster.
@tgravescs @vanzin please help to review, thanks! I think the change is
quite straightforward, mainly on the doc/configuration things, update them to
be more accurate.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jerryshao/apache-spark SPARK-20060
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19227.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19227
commit 2b3d2f24f94a1cee63fff9733b27f479673d7a90
Author: jerryshao
Date: 2017-09-14T02:15:41Z
Support accessing secure Hadoop cluster in standalone client mode
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19135: [SPARK-21923][CORE]Avoid calling reserveUnrollMemoryForT...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19135 So it somehow reflects that CPU core contention is the main issue for memory pre-occupation , am I right? AFAIK from our customer, we usually don't allocate so many cores to one executor, also avoid big executor (many cores + large heap memory) to reduce GC and other memory/cpu contentions. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19160: [SPARK-21934][CORE] Expose Shuffle Netty memory usage to...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19160 @squito would you please help to review this PR, thanks a lot. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19210: Fix Graphite re-connects for Graphite instances behind E...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19210 You should also update the the files under `dev/deps`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19132: [SPARK-21922] Fix duration always updating when t...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19132#discussion_r138533547
--- Diff:
core/src/main/scala/org/apache/spark/status/api/v1/AllStagesResource.scala ---
@@ -47,7 +47,8 @@ private[v1] class AllStagesResource(ui: SparkUI) {
listener.stageIdToData.get((stageInfo.stageId,
stageInfo.attemptId))
}
} yield {
- AllStagesResource.stageUiToStageData(status, stageInfo, stageUiData,
includeDetails = false)
+ AllStagesResource.stageUiToStageData(
+status, stageInfo, stageUiData, includeDetails = false, Some(ui))
--- End diff --
It's not a good idea to pass in `SparkUI` to only get `lastUpdate`, the API
looks weird to add this `SparkUI` argument, the fix here only just make it
work. It is better to add one more field in `StageUIData` or `TaskUIData` if
possible.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19132: [SPARK-21922] Fix duration always updating when t...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19132#discussion_r138526050
--- Diff:
core/src/main/scala/org/apache/spark/status/api/v1/AllStagesResource.scala ---
@@ -142,7 +142,7 @@ private[v1] object AllStagesResource {
index = uiData.taskInfo.index,
attempt = uiData.taskInfo.attemptNumber,
launchTime = new Date(uiData.taskInfo.launchTime),
- duration = uiData.taskDuration,
+ duration = uiData.taskDuration(),
--- End diff --
Yes, if it is not a big change I think it should be fixed here. Because
currently with this fix UI and REST API are inconsistent.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19132: [SPARK-21922] Fix duration always updating when t...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19132#discussion_r138510683
--- Diff:
core/src/main/scala/org/apache/spark/status/api/v1/AllStagesResource.scala ---
@@ -142,7 +142,7 @@ private[v1] object AllStagesResource {
index = uiData.taskInfo.index,
attempt = uiData.taskInfo.attemptNumber,
launchTime = new Date(uiData.taskInfo.launchTime),
- duration = uiData.taskDuration,
+ duration = uiData.taskDuration(),
--- End diff --
Here what if we call the REST API on history server to get stage info?
Looks like we may still have this issue since we don't have last update time
here, what do you think @ajbozarth ?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17387: [SPARK-20060][Deploy][Kerberos]Support Standalone visiti...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17387 @yaooqinn I think the patch here is quite old and cannot be merged anymore, can you please close it. If you still want to address this issue, can you please create a new PR, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Overall LGTM, @ajbozarth can you please review again? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19141: [SPARK-21384] [YARN] Spark + YARN fails with Loca...
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19141#discussion_r138505323 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -565,7 +565,6 @@ private[spark] class Client( distribute(jarsArchive.toURI.getPath, resType = LocalResourceType.ARCHIVE, destName = Some(LOCALIZED_LIB_DIR)) - jarsArchive.delete() --- End diff -- Think about this again, I think you're right. But I'm not sure if the program will be crashed or not if we delete the dependencies in the run-time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Thanks @HyukjinKwon , I will ping Josh about this thing ð . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19130: [SPARK-21917][CORE][YARN] Supporting adding http(s) reso...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19130 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19141: [SPARK-21384] [YARN] Spark + YARN fails with Loca...
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19141#discussion_r138262309 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -565,7 +565,6 @@ private[spark] class Client( distribute(jarsArchive.toURI.getPath, resType = LocalResourceType.ARCHIVE, destName = Some(LOCALIZED_LIB_DIR)) - jarsArchive.delete() --- End diff -- What if your scenario and SPARK-20741's scenario are both encountered? Looks like your approach above cannot be worked. I'm wondering if we can copy or move this __spark_libs__.zip temp file to another non-temp file and add that file to the dist cache. That non-temp file will not be deleted and can be overwritten during another launching, so we will always have only one copy. Besides, I think we have several workarounds to handle this issue like spark.yarn.jars or spark.yarn.archive, so looks like this corner case is not so necessary to fix (just my thinking, normally people will not use local FS in a real cluster). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19130: [SPARK-21917][CORE][YARN] Supporting adding http(s) reso...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19130 @tgravescs , thanks for your comments, can you review again, if it is what you expected. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Looks like I don't have the Jenkins permission to trigger UT ð . Let me ping @srowen to trigger the test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19132: [SPARK-21922] Fix duration always updating when t...
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19132#discussion_r138240053 --- Diff: core/src/main/scala/org/apache/spark/ui/SparkUI.scala --- @@ -50,6 +50,7 @@ private[spark] class SparkUI private ( val operationGraphListener: RDDOperationGraphListener, var appName: String, val basePath: String, +val lastUpdateTime: Long = -1L, --- End diff -- I would like to user `Option[Long] = None` as default value to reflect there's no update time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 ok to test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19174: [SPARK-21963][CORE][TEST]Create temp file should be dele...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19174 This seems not a big problem, all the temp files are created under `target/tmp`, this can be cleaned by `mvn clean` or `sbt clean`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Still I have a question about history server, is you event log an incomplete event log or completed when you met such issue? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19184: [SPARK-21971][CORE] Too many open files in Spark ...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19184#discussion_r137981999
--- Diff:
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java
---
@@ -104,6 +124,10 @@ public void loadNext() throws IOException {
if (taskContext != null) {
taskContext.killTaskIfInterrupted();
}
+if (this.din == null) {
+ // Good time to init (if all files are opened, we can get Too Many
files exception)
+ initStreams();
+}
--- End diff --
I agree with @viirya , we're using priority queue to do merge sort, this
will turn out to be all the readers in the priority queue is opened, so still
cannot solve this issue.
I think a valid fix is to control the number of concurrent merged files,
like MR's `io.sort.factor`.
Also we still need to address similar issue in `ExternalSorter` and other
places in Shuffle.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 So your PR description is quite confusing, would you please elaborate your problem in detail and describe how to reproduce the issue. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19142: When the number of attempting to restart receiver greate...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19142 NVM, I mean in the Spark code there're some intended empty "else" branch, are you going to add trace log to them all? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18948: Add the validation of spark.cores.max under Streaming
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18948 The patch here is not solid, we will not merge it unless you have better solution. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19142: When the number of attempting to restart receiver greate...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19142 I'm -1 on this PR. This PR actually fix nothing instead of adding one trace log, also usually user will not enable trace log, so this one line fix is not so helpful. You can find many place in Spark where it has no "else" branch, are you going to fix them all? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19160: [SPARK-21934][CORE] Expose Shuffle Netty memory usage to...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19160 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19132: [SPARK-21922] Fix duration always updating when task fai...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19132 Is this a problem only in History UI, or it also has issues in Live UI? From my understanding you only pass a last update time for history UI, so is it intended? Also you mentioned "When executor failed and task metrics have not send to driver,the status will always be 'RUNNING'", is this a bug in scheduler? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19160: [SPARK-21934][CORE] Expose Shuffle Netty memory u...
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/19160 [SPARK-21934][CORE] Expose Shuffle Netty memory usage to MetricsSystem ## What changes were proposed in this pull request? This is a followup work of SPARK-9104 to expose the Netty memory usage to MetricsSystem. Current the shuffle Netty memory usage of `NettyBlockTransferService` will be exposed, if using external shuffle, then the Netty memory usage of `ExternalShuffleClient` and `ExternalShuffleService` will be exposed instead. Currently I don't expose Netty memory usage of `YarnShuffleService`, because `YarnShuffleService` doesn't have `MetricsSystem` itself, and is better to connect to Hadoop's MetricsSystem. ## How was this patch tested? Manually verified in local cluster. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerryshao/apache-spark SPARK-21934 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19160.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19160 commit 04a7ec944b3273fbe9b9bdb6e217814452a1a12c Author: jerryshao Date: 2017-09-07T13:25:39Z Expose Shuffle Netty memory usage to MetricsSystem --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19150: [SPARK-21939][TEST] Use TimeLimits instead of Timeouts
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19150 Merging to master, thanks @dongjoon-hyun . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19130: [SPARK-21917][CORE][YARN] Supporting adding http(s) reso...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19130 @vanzin @tgravescs , would you please help to review, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19131: [MINOR][SQL]remove unuse import class
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19131 Jenkins, test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19150: [SPARK-21939][TEST] Use TimeLimits instead of Timeouts
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19150 LGTM, there's a typo in PR description, "Timeouts is deprecated." not "TimeLimits". --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19141: [SPARK-21384] [YARN] Spark + YARN fails with Loca...
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/19141#discussion_r137427105 --- Diff: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -565,7 +565,6 @@ private[spark] class Client( distribute(jarsArchive.toURI.getPath, resType = LocalResourceType.ARCHIVE, destName = Some(LOCALIZED_LIB_DIR)) - jarsArchive.delete() --- End diff -- Agree with Marcelo, this is a valid concern, we should not avoid such regression here. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19141: [SPARK-21384] [YARN] Spark + YARN fails with LocalFileSy...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19141 OK to test. (I may not have the permission to trigger Jenkins test ð ) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19141: [SPARK-21384] [YARN] Spark + YARN fails with LocalFileSy...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19141 I see, thanks for the explanation. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19131: [MINOR][SQL]remove unuse import class
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19131 Personally I'm not fond of such fix, this will break lots of existing PRs and force them to rebase again. Usually this could be addressed when fixing other issues. IMHO I don't encourage such PRs. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19141: [SPARK-21384] [YARN] Spark 2.2 + YARN without spark.yarn...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19141 Also looks like this is not a Spark 2.2 issue, would you please fix the PR title be more accurate about the problem? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19141: [SPARK-21384] [YARN] Spark 2.2 + YARN without spark.yarn...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19141 Can you please describe your usage scenario and steps to reproduce your issue, from my understanding. Did you configure your default FS to a local FS? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19131: [MINOR][SQL]remove unuse import class
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19131 What about other components, here you only fixed sql and core module. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19134 I see @felixcheung . Since we have a solution to turn off Python Kafka unit test as mentioned by @vanzin , so it is fine to just mark as deprecated, not remove the code. Another thing is that we also have some Kafka related examples, maybe we should also handle them. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19131: [MINOR][SQL]remove unuse import class
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19131 Jenkins, test this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19140: [SPARK-21890] Credentials not being passed to add the to...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19140 @redsanket can you please test this with a secure Hadoop environment using spark-submit (not Oozie), I don't want to bring in any regression here. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18628: [SPARK-18061][ThriftServer] Add spnego auth support for ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18628 Thanks @jiangxb1987 , let me merge it to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19135: [SPARK-21923][CORE]Avoid call reserveUnrollMemoryForThis...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19135 Sorry I'm not so familiar with this part, but from the test result seems that the performance just improved a little. I would doubt the way you generate RDD `0 until Integer.MAX_VALUE` might take most of the time (since a large integer array needs to be serialized with tasks and ship to executor). Also I see you use 1 executor with 20 cores to do test. In the normal usage case we will not allocate so many cores to 1 executor, can you please test with 2-4 cores per executor, I guess with less cores, the contention of MemoryManager lock should be alleviated, and the performance might be close. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19134 Yes, Python kafka.py itself is OK to leave without calling it, but the UT will involve Scala Kafka module to do the test. Currently I don't know how to address this issue. Ideally it would be better to put Python Kafka related codes to this python module, and only load in when specifically opt-in. But I'm not sure how to achieve this. Another option is to remove this python Kafka code. Previously we stopped supporting Python Spark Streaming work (https://github.com/apache/spark/pull/14340#issuecomment-235457517), so it might be OK to remove it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19134: [SPARK-21893][BUILD][STREAMING][WIP] Put Kafka 0.8 behin...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19134 @srowen how do you handle python kafka.py, should it also be opt-in? As far as I understand looks like you don't address it in this PR. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19130: [SPARK-21917][CORE][YARN] Supporting adding http(s) reso...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19130 Jenkins, retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUser to s...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19121 Sorry I didn't clearly say the problem. But IMO the changes you made is really not so necessary. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18519: [SPARK-16742] Mesos Kerberos Support
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18519 @ArtRand @vanzin , is it only worked in client deploy mode, am I understanding correctly? I don't see a code to ship tokens from local client to remote driver. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19130: [SPARK-21917][CORE][YARN] Supporting Download htt...
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/19130 [SPARK-21917][CORE][YARN] Supporting Download http(s) resources in yarn mode ## What changes were proposed in this pull request? In the current Spark, when submitting application on YARN with remote resources `./bin/spark-shell --jars http://central.maven.org/maven2/com/github/swagger-akka-http/swagger-akka-http_2.11/0.10.1/swagger-akka-http_2.11-0.10.1.jar --master yarn-client -v`, Spark will be failed with: ``` java.io.IOException: No FileSystem for scheme: http at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2586) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2593) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:354) at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:478) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11$$anonfun$apply$6.apply(Client.scala:600) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11$$anonfun$apply$6.apply(Client.scala:599) at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:74) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11.apply(Client.scala:599) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$11.apply(Client.scala:598) at scala.collection.immutable.List.foreach(List.scala:381) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:598) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:848) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:173) ``` This is because `YARN#client` assumes resources are on the Hadoop compatible FS. To fix this problem, here propose to download remote http(s) resources to local and add this local downloaded resources to dist cache. This solution has one downside: remote resources are downloaded and uploaded again, but it only restricted to only remote http(s) resources, also the overhead is not so big. The advantages of this solution is that it is simple and the code changes restricts to only `SparkSubmit`. ## How was this patch tested? Unit test added, also verified in local cluster. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerryshao/apache-spark SPARK-21917 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19130.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19130 commit 42a79ab5dab8f956eb923a2260b5aed7ae2660cf Author: jerryshao Date: 2017-09-05T08:18:36Z Download remote http(s) resources to local in yarn mode Change-Id: I7897817ceaaafecd779a6e085c96d2a28363d7d6 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUser to s...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19121 No, I don't agree with you. SPARK_USER is set in SparkContext with driver's current UGI and this env variable will be propagated to executors to create executor's UGI with the same user in driver. For example, if your standalone cluster is started with user "spark", and you submit a Spark application with user "foo" in the gateway, so all the executors should use "foo" to access Hadoop, but with your changes, executors will use "spark" to communicate with Hadoop, since executor process is forked with user "spark" by worker, it is not correct. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUser to s...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19121 UGI is only used for security, normally it is used for Spark application to communicate with Hadoop using correct user. doAs already wraps the whole `CoarseGrainedExecutorBackend` process, all the task threads forked in this process will honor this UGI, don't need to wrap again on each task. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19121: [SPARK-21906][YARN][Spark Core]Don't runAsSparkUser to s...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19121 Can you please elaborate the problem you met, did you meet any unexpected behavior? The changes here get rid of env variable "SPARK_USER", this might be OK for yarn application, but what if user runs on standalone mode and explicitly set this "SPARK_USER", your changes seems break the semantics. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18628: [SPARK-18061][ThriftServer] Add spnego auth support for ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18628 @cloud-fan @jiangxb1987, what do you think about this PR, I think it mostly copies from HS2, and it is quite isolated unless we enabled spnego, so it should be safe to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19115: [SPARK-21882][CORE] OutputMetrics doesn't count written ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19115 Please create a new pr against master branch and close this one. If the issue doesn't exist in master branch, then consider backporting that fix to 2.2 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19115: Update PairRDDFunctions.scala
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19115 @awarrior please follow the [doc](https://spark.apache.org/contributing.html) to submit patch. You need to change the PR title like other PRs by adding JIRA id and component tag. Add the details of your problem in PR description, not just a simple JIRA link. Submit PR against master branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19079: [SPARK-21859][CORE] Fix SparkFiles.get failed on driver ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19079 Please close this PR @lgrcyanny thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19103: [SPARK-21890] Credentials not being passed to add the to...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19103 @vanzin From my understanding seems like it is a workaround to avoid issuing new HDFS tokens (since this user credential we already has HDFS tokens). But how to handle HBase/Hive thing without TGT? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19103: [SPARK-21890] Credentials not being passed to add the to...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19103 @tgravescs , I think it is in `AMCredentialRenewer` we explicitly create a new `Credential` every time when issuing new tokens. ``` // HACK: // HDFS will not issue new delegation tokens, if the Credentials object // passed in already has tokens for that FS even if the tokens are expired (it really only // checks if there are tokens for the service, and not if they are valid). So the only real // way to get new tokens is to make sure a different Credentials object is used each time to // get new tokens and then the new tokens are copied over the current user's Credentials. // So: // - we login as a different user and get the UGI // - use that UGI to get the tokens (see doAs block below) // - copy the tokens over to the current user's credentials (this will overwrite the tokens // in the current user's Credentials object for this FS). ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19103: [SPARK-21890] Credentials not being passed to add the to...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19103 >Oozie client gets the necessary tokens the application needs before launching. It passes those tokens along to the oozie launcher job (MR job) which will then actually call the Spark client to launch the spark app and pass the tokens along. >The oozie launcher job cannot get anymore tokens because all it has is tokens ( you can't get tokens with tokens, you need tgt or keytab). >The error here is because the launcher job runs the Spark Client to submit the spark job but the spark client doesn't see that it already has the hdfs tokens so it tries to get more, which ends with the exception. So the problem is that Oozie will get tokens for Spark instead of letting Spark do itself, and in Oozie launcher we should not let Spark `Yarn#client` to get tokens itself since there might not have tgt available in Oozie launcher. From my understanding of your issue, this seems like a more general issue regarding Oozie launcher and Spark token manage stuff. With the patch, looks like it only address the HDFS issue, how do we handle hive/hbase, looks like still have issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19077: [SPARK-21860][core]Improve memory reuse for heap memory ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19077 This PR generally looks fine to me, my concern is that will this change bring in subtle impact on the code which leverage it. CC @JoshRosen to take a review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18935: [SPARK-9104][CORE] Expose Netty memory metrics in Spark
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18935 @squito can you please review again? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19077: [SPARK-21860][core]Improve memory reuse for heap ...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19077#discussion_r136332974
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java
---
@@ -47,23 +47,29 @@ private boolean shouldPool(long size) {
@Override
public MemoryBlock allocate(long size) throws OutOfMemoryError {
-if (shouldPool(size)) {
+int arraySize = (int)((size + 7) / 8);
+long alignedSize = arraySize * 8;
+if (shouldPool(alignedSize)) {
synchronized (this) {
-final LinkedList> pool =
bufferPoolsBySize.get(size);
+final LinkedList> pool =
bufferPoolsBySize.get(alignedSize);
if (pool != null) {
while (!pool.isEmpty()) {
final WeakReference blockReference = pool.pop();
final MemoryBlock memory = blockReference.get();
if (memory != null) {
- assert (memory.size() == size);
+ assert ((int)((memory.size() + 7) / 8) == arraySize);
+ memory.resetSize(size);
--- End diff --
I got it, thanks for the explanation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16803: [SPARK-19458][BUILD]load hive jars from local repo which...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/16803 @windpiger can you please rebase the code, it seems too old to review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19032: [SPARK-17321][YARN] Avoid writing shuffle metadata to di...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19032 Merge to master branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19074: [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-upl...
Github user jerryshao closed the pull request at: https://github.com/apache/spark/pull/19074 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19074: [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19074 Thanks @vanzin , it should be passed now ð . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9518: [SPARK-11574][Core] Add metrics StatsD sink
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/9518 Merge to master branch, thanks @xflin ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19032: [SPARK-17321][YARN] Avoid writing shuffle metadat...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19032#discussion_r136223332
--- Diff:
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -321,6 +326,7 @@ public ByteBuffer getMetaData() {
* overrode and called when Hadoop version is 2.5+ and NM recovery is
enabled, otherwise we
--- End diff --
Sure I will.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9518: [SPARK-11574][Core] Add metrics StatsD sink
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/9518 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9518: [SPARK-11574][Core] Add metrics StatsD sink
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/9518 This LGTM, since it is a quite independent PR and doesn't bring in other dependencies, I think it is good to be merged. Ping others do you have further comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19077: [SPARK-21860][core]Improve memory reuse for heap memory ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19077 Can you please add some unit test to verify your changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19077: [SPARK-21860][core]Improve memory reuse for heap ...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19077#discussion_r136069800
--- Diff:
common/unsafe/src/main/java/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java
---
@@ -47,23 +47,29 @@ private boolean shouldPool(long size) {
@Override
public MemoryBlock allocate(long size) throws OutOfMemoryError {
-if (shouldPool(size)) {
+int arraySize = (int)((size + 7) / 8);
+long alignedSize = arraySize * 8;
+if (shouldPool(alignedSize)) {
synchronized (this) {
-final LinkedList> pool =
bufferPoolsBySize.get(size);
+final LinkedList> pool =
bufferPoolsBySize.get(alignedSize);
if (pool != null) {
while (!pool.isEmpty()) {
final WeakReference blockReference = pool.pop();
final MemoryBlock memory = blockReference.get();
if (memory != null) {
- assert (memory.size() == size);
+ assert ((int)((memory.size() + 7) / 8) == arraySize);
+ memory.resetSize(size);
--- End diff --
Hmm, from my understanding the size of `MemoryBlock` is always the actual
size, not the aligned size, so looks like we dont need to reset the size here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19079: [SPARK-21859][CORE] Fix SparkFiles.get failed on ...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19079#discussion_r135973949
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -481,7 +481,7 @@ object SparkSubmit extends CommandLineUtils {
sysProp = "spark.executor.memory"),
OptionAssigner(args.totalExecutorCores, STANDALONE | MESOS,
ALL_DEPLOY_MODES,
sysProp = "spark.cores.max"),
- OptionAssigner(args.files, LOCAL | STANDALONE | MESOS,
ALL_DEPLOY_MODES,
+ OptionAssigner(args.files, ALL_CLUSTER_MGRS, ALL_DEPLOY_MODES,
--- End diff --
Also as @vanzin mentioned, also the description of "Yarn cluster mode
`SparkFiles.get` is working" is not a design purpose. So to fix this issue I
think you should have a more solid patch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19079: [SPARK-21859][CORE] Fix SparkFiles.get failed on ...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19079#discussion_r135973543
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -481,7 +481,7 @@ object SparkSubmit extends CommandLineUtils {
sysProp = "spark.executor.memory"),
OptionAssigner(args.totalExecutorCores, STANDALONE | MESOS,
ALL_DEPLOY_MODES,
sysProp = "spark.cores.max"),
- OptionAssigner(args.files, LOCAL | STANDALONE | MESOS,
ALL_DEPLOY_MODES,
+ OptionAssigner(args.files, ALL_CLUSTER_MGRS, ALL_DEPLOY_MODES,
--- End diff --
I don't say `SparkFiles.get` is not useful, I'm saying your fix is not
correct, the changes here will break the original semantics. Also we recently
support remote files, to handle this scenario we should think how to address
this problem for all the cluster managers, not only in yarn client mode.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19079: [SPARK-21859][CORE] Fix SparkFiles.get failed on ...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19079#discussion_r135958924
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -481,7 +481,7 @@ object SparkSubmit extends CommandLineUtils {
sysProp = "spark.executor.memory"),
OptionAssigner(args.totalExecutorCores, STANDALONE | MESOS,
ALL_DEPLOY_MODES,
sysProp = "spark.cores.max"),
- OptionAssigner(args.files, LOCAL | STANDALONE | MESOS,
ALL_DEPLOY_MODES,
+ OptionAssigner(args.files, ALL_CLUSTER_MGRS, ALL_DEPLOY_MODES,
--- End diff --
The changes here is not correct. For yarn application, we use
`spark.yarn.dist.files` to handle files, and this will be added to distributed
cache. Without your change it breaks the current code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19079: [SPARK-21859][CORE] Fix SparkFiles.get failed on driver ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19079 Currently for Spark yarn-client application, we don't support fetching files using above `SparkFiles.get` API. Since you already know where the file is in client mode, so may be you don't need to call `SparkFiles.get`. Here actually requires several changes including supporting remote files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19074: [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19074 @vanzin @srowen pushed another commit to change 2.10 repl code, I tested locally with 2.10 code, please review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19074: [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19074 OK, so I will do the test locally. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19032: [SPARK-17321][YARN] Avoid writing shuffle metadata to di...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19032 @vanzin @tgravescs do you have any further comment? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19032: [SPARK-17321][YARN] Avoid writing shuffle metadata to di...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19032 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19074: [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19074 Ohh, sorry about it, I forgot to fix it in 2.10 repl code. I will push a fix soon. BTW how do we trigger scala 2.10 build on Jenkins? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 Yes, maybe. You can take a try locally. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19074: [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19074 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 I think it is already in master branch @caneGuy . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 @caneGuy can you please check this JIRA (https://issues.apache.org/jira/browse/SPARK-14423), I remembered I fixed this issue before. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 I tried locally with same name jar uploaded twice, the spark application can be started. Can you please paste your exception here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 But as I remembered the same name file will be ignored when met again. This should not be a fatal issue, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19073: [SPARK-21855][YARN] Should print error when upload same ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19073 @caneGuy why do you think it is misleading? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19074: [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19074 CC @vanzin @tgravescs please review. Since 2.2 and master are quite different in this part of code, so backporting changes a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19074: [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-upl...
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/19074 [SPARK-21714][CORE][BACKPORT-2.2] Avoiding re-uploading remote resources in yarn client mode ## What changes were proposed in this pull request? This is a backport PR to fix issue of re-uploading remote resource in yarn client mode. The original PR is #18962. ## How was this patch tested? Tested in local UT. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerryshao/apache-spark SPARK-21714-2.2-backport Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19074.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19074 commit 9c5b56247ebc997cf7e8e64cbe95e07ff74bbbe6 Author: jerryshao Date: 2017-08-25T16:57:53Z [SPARK-21714][CORE][YARN] Avoiding re-uploading remote resources in yarn client mode With SPARK-10643, Spark supports download resources from remote in client deploy mode. But the implementation overrides variables which representing added resources (like `args.jars`, `args.pyFiles`) to local path, And yarn client leverage this local path to re-upload resources to distributed cache. This is unnecessary to break the semantics of putting resources in a shared FS. So here proposed to fix it. This is manually verified with jars, pyFiles in local and remote storage, both in client and cluster mode. Author: jerryshao Closes #18962 from jerryshao/SPARK-21714. (cherry picked from commit 1813c4a8dd4388fe76a4ec772c9be151be0f60a1) Signed-off-by: jerryshao Change-Id: Ib2e8cb056707b362bc1c496002bac1472dc78ea7 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19061: [SPARK-21568][CORE] ConsoleProgressBar should only be en...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19061 >If such a thing as a non-Spark repl-like application exists, it wouldn't be getting the progress bar by default, for example, because its default log level is "INFO" in Spark, something that disables the progress bar. That makes sense! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18962: [SPARK-21714][CORE][YARN] Avoiding re-uploading remote r...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18962 Sorry I missed the comments, I will file another PR against branch 2.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19032: [SPARK-17321][YARN] Avoid writing shuffle metadat...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19032#discussion_r135466826
--- Diff:
common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java
---
@@ -73,6 +75,8 @@
public class YarnShuffleService extends AuxiliaryService {
private static final Logger logger =
LoggerFactory.getLogger(YarnShuffleService.class);
+ private static final boolean DEFAULT_NM_RECOVERY_ENABLED = false;
--- End diff --
Let me check the yarn code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19061: [SPARK-21568][CORE] ConsoleProgressBar should only be en...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19061 @dongjoon-hyun I'm just thinking if other repl-like projects may actually require this, you changes here make them fail to leverage this feature. Did you see any issue with this feature on in non-shell apps? Also I think you should do the check in SparkConf because user can still set this with SparkConf in run-time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19012: [SPARK-17742][core] Fail launcher app handle if child pr...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19012 LGTM, I tried locally. Looks like now the NPE is gone in yarn UT, thanks for the fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19047: [SPARK-21798]: No config to replace deprecated SP...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/19047#discussion_r135247004
--- Diff:
launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
@@ -136,7 +136,8 @@ void addOptionString(List cmd, String options) {
Set cp = new LinkedHashSet<>();
addToClassPath(cp, appClassPath);
-
+addToClassPath(cp, getenv("SPARK_DAEMON_CLASSPATH"));
--- End diff --
I think it would be better to only support this env variable for daemon
process like HistoryServer, ExternalShuffleService and others, like what we did
for `SPARK_DAEMON_JAVA_OPTS`. Currently with your fix normal Spark application
will also honor this env variable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19034: [SPARK-21821][CORE]Support to force kill the CoarseGrain...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19034 Agree with @vanzin , if you really want to fix this issue I think you should find out the root cause and fix the code in Spark. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19039: [SPARK-21829][CORE] Enable config to permanently blackli...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19039 The changes you made in `BlacklistTracker` seems break the design purpose of backlist. The blacklist in Spark as well as in MR/TEZ assumes bad nodes/executors will be back to normal in several hours, so it always has a timeout for blacklist. In your case, the problem is not bad nodes/executors, it is that you don't what to start executors on some nodes (like slow nodes). This is more like a cluster manager problem rather than Spark problem. To summarize your problem, you want your Spark application runs on some specific nodes. To solve your problem, for YARN you could use node label and Spark on YARN already support node label. You could google node label to know the details. For standalone, simply you should not start worker on such nodes you don't want. For Mesos I'm not sure, I guess it should also has similar approaches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19034: [SPARK-21821][CORE]Support to force kill the CoarseGrain...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19034 My thinking is that different users may have different deploys and different environment, so usually they will maintain their own scripts for such purpose, this seems not a Spark problem from my understanding. Maybe @srowen can weigh in this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19034: [SPARK-21821][CORE]Support to force kill the CoarseGrain...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19034 Should it be better to maintain it in your in-house env rather than in community Spark? IMHO such script seem not so proper to maintain Spark codebase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
