Github user trystanleftwich commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77208611
So to confirm, i think this function needs to be able to handle 5 states:
Path is a dir which has subdirs
(structure is hdfs://foo/foo1/foo2.jar)
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77209251
I tested with this version of `fetchHcfsFile` and my tests pass:
/**
* Fetch a file or directory from a Hadoop-compatible filesystem.
*
*
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77223185
Hi @trystanleftwich , just tested my code again with more strict checks,
and files show up as files, directories show up as directories.
---
If your project is set up
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4881#discussion_r25805285
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -389,16 +389,30 @@ class UtilsSuite extends FunSuite with
ResetSystemProperties {
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77224961
Leaving a link to an alternate fix in #4894
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77221581
The code will create a directory local_dir/foo/foo.jar and not a file
Hmm. Let me check that.
---
If your project is set up for it, you can reply to this email
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77245736
Let's close this PR in favor of #4894, which I just merged. Thanks for
reporting this blocker.
---
If your project is set up for it, you can reply to this email and
Github user trystanleftwich closed the pull request at:
https://github.com/apache/spark/pull/4881
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77216017
@trystanleftwich I believe that's correct. To summarize:
- Before this patch, adding `hdfs://single/file.jar` doesn't work (a
regression from Spark 1.2)
-
Github user trystanleftwich commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77221033
Ok, i've pushed my changes, I've added tests that should cover all the
states, I was getting errors with @vanzin code snippet, if you pass in dir i.e
path =
Github user trystanleftwich commented on a diff in the pull request:
https://github.com/apache/spark/pull/4881#discussion_r25806270
--- Diff: core/src/test/scala/org/apache/spark/util/UtilsSuite.scala ---
@@ -389,16 +389,30 @@ class UtilsSuite extends FunSuite with
GitHub user trystanleftwich opened a pull request:
https://github.com/apache/spark/pull/4880
[SPARK-6144]When in cluster mode using ADD JAR with a hdfs:// sourced ja...
...r will fail
While in cluster mode if you use ADD JAR with a HDFS sourced jar it will
fail trying to source
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4880#discussion_r25737673
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -632,8 +633,9 @@ private[spark] object Utils extends Logging {
fs:
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4880#discussion_r25741212
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -643,7 +645,12 @@ private[spark] object Utils extends Logging {
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77079868
Jenkins, this is ok to test.
By the way in the future it would be good to open this against the master
branch. In this case it's fine because the 1.3 branch
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77056801
[Test build #28246 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28246/consoleFull)
for PR 4880 at commit
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4880#discussion_r25740726
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -643,7 +645,12 @@ private[spark] object Utils extends Logging {
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77054200
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4880#discussion_r25738862
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -632,8 +633,9 @@ private[spark] object Utils extends Logging {
fs:
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77068164
@trystanleftwich thanks for fixing this. I believe given the current way we
call `fetchHcfsFile` your existing patch is sufficient in fixing the problem.
However, at
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77071494
I tried this patch locally and while it works for `addFile(String)`, it
seems to not work for `addFile(String, boolean)` (i.e. the directory version).
Here's the error I
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4880#discussion_r25743304
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -643,7 +645,12 @@ private[spark] object Utils extends Logging {
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77062488
@pwendell adding to your radar.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77072245
Ah, I didn't realize `addFile` also supports directories for Hadoop file
systems. Then this does seem to a correctness problem.
---
If your project is set up for it,
Github user trystanleftwich commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77079677
I fat fingered and accidentally closed this ticket, And for some reason its
not picking up that the branch has changes in it. I reopened here:
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77079619
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4880#discussion_r25743237
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -643,7 +645,12 @@ private[spark] object Utils extends Logging {
GitHub user trystanleftwich opened a pull request:
https://github.com/apache/spark/pull/4881
[SPARK-6144]When in cluster mode using ADD JAR with a hdfs:// sourced ja...
...r will fail
You can merge this pull request into a Git repository by running:
$ git pull
GitHub user trystanleftwich reopened a pull request:
https://github.com/apache/spark/pull/4881
[SPARK-6144]When in cluster mode using ADD JAR with a hdfs:// sourced ja...
...r will fail
You can merge this pull request into a Git repository by running:
$ git pull
Github user trystanleftwich closed the pull request at:
https://github.com/apache/spark/pull/4881
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77056435
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77060524
LGTM aside from minor style issue. I also think this should really go into
1.3...
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77073312
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4880#issuecomment-77073304
[Test build #28246 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28246/consoleFull)
for PR 4880 at commit
Github user trystanleftwich closed the pull request at:
https://github.com/apache/spark/pull/4880
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4881#discussion_r25748314
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -632,8 +633,9 @@ private[spark] object Utils extends Logging {
fs:
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77085167
[Test build #625 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/625/consoleFull)
for PR 4881 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4881#issuecomment-77093891
[Test build #625 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/625/consoleFull)
for PR 4881 at commit
38 matches
Mail list logo
