Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13042
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219995467
merged to master. I'm not against merging to 2.0.0 but wasn't sure how much
of a perf problem this is in normal usage.
---
If your project is set up for it, you can
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219797992
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219797991
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219797583
**[Test build #58701 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58701/consoleFull)**
for PR 13042 at commit
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219761677
Ok. Changed TestShuffleDataContext to use the method in JavaUtil
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219761359
**[Test build #58701 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58701/consoleFull)**
for PR 13042 at commit
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219659234
I'm OK with just the first change, for tidiness.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219506107
- I am fine with changing `TestShuffleDataContext` to use the method in
JavaUtil.
- For `Utils.deleteRecursively`, I am not sure if that will have impact on
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219208629
This seems OK to me. There is actually another delete-recursively method in
TestShuffleDataContext in network-shuffle which should be able to use this
method rather
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219128637
There are no outstanding review comments left to address here. If
everything looks good, can someone merge this PR ? Thanks !!
---
If your project is set up for
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-219128249
@vanzin : I haven't gathered numbers with `Files.walkFileTree`. Based on
its source code, it will have the same quirks wrt race conditions and speed. I
believe
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218872339
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218872344
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218872100
**[Test build #58511 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58511/consoleFull)**
for PR 13042 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218844388
**[Test build #58511 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58511/consoleFull)**
for PR 13042 at commit
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218843566
Out of curiosity, I wonder what the performance would be if you use
`java.nio.file.Files.walkFileTree`... assuming it can be used in this case.
---
If your project is
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218843364
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218843259
Jenkins add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218835975
Not sure of the build failure was related from my recent change(s). Retest
please.
---
If your project is set up for it, you can reply to this email and have your
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r63054715
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +81,32 @@ public static String
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218814903
**[Test build #58499 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58499/consoleFull)**
for PR 13042 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218814935
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218814932
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218812762
**[Test build #58499 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58499/consoleFull)**
for PR 13042 at commit
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r63051771
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +81,32 @@ public static String
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r63046042
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +81,32 @@ public static String
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218799080
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r63044500
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +129,40 @@ public static void
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62997972
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +129,40 @@ public static void
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218659846
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218659848
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218659786
**[Test build #58439 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58439/consoleFull)**
for PR 13042 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218649334
**[Test build #58439 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58439/consoleFull)**
for PR 13042 at commit
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62960246
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +128,27 @@ public static void
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62939281
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +128,27 @@ public static void
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62939126
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +128,27 @@ public static void
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218540631
@podwhitehawk I agree with @srowen about single argument being passed.
Generally speaking, there is not a maximum path length in Unix but there
might be
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218420285
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218420282
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218420055
**[Test build #58352 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58352/consoleFull)**
for PR 13042 at commit
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218413926
That LGTM; @podwhitehawk I don't think that will be a problem? the command
is always `rm -rf [dir]`
---
If your project is set up for it, you can reply to this email
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62820647
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +80,32 @@ public static String
Github user podwhitehawk commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218410934
as a linux adm, JFYI, you may encounter "too many arguments" error someday
as described here:
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218394679
**[Test build #58352 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58352/consoleFull)**
for PR 13042 at commit
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62808325
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +128,27 @@ public static void
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62808334
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +128,27 @@ public static void
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62808363
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +128,27 @@ public static void
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62808294
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +80,32 @@ public static String
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218387086
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218387087
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218386867
**[Test build #58332 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58332/consoleFull)**
for PR 13042 at commit
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62799243
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +128,27 @@ public static void
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62799170
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +128,27 @@ public static void
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62799116
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -109,6 +128,27 @@ public static void
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218376555
Seems fairly reasonable; is the speedup measurable? wouldn't have thought
so, but maybe for tens of thousands of files, sure.
---
If your project is set up for it, you
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62798961
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +80,32 @@ public static String
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218368903
**[Test build #58332 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58332/consoleFull)**
for PR 13042 at commit
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62794873
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +80,27 @@ public static String
Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62794845
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +80,27 @@ public static String
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62793386
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +80,27 @@ public static String
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/13042#discussion_r62793337
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
---
@@ -79,14 +80,27 @@ public static String
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218356728
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218356727
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218356644
**[Test build #58312 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58312/consoleFull)**
for PR 13042 at commit
Github user tejasapatil commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218344802
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13042#issuecomment-218344768
**[Test build #58312 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58312/consoleFull)**
for PR 13042 at commit
GitHub user tejasapatil opened a pull request:
https://github.com/apache/spark/pull/13042
[SPARK-15263][Core] Make shuffle service dir cleanup faster by using `rm
-rf`
## What changes were proposed in this pull request?
Jira:
68 matches
Mail list logo
