[
https://issues.apache.org/jira/browse/SPARK-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157263#comment-14157263
]
Tom Weber commented on SPARK-3769:
----------------------------------
Thanks for the quick turnaround!
I can see that it wouldn't necessarily make sense that a fully qualified path
(relative to the driver programs filesystem)
would be what the .get method would take on the worker node systems. But, at
the same time, the .get seems like it
just takes what you give it and blindly concatenates it to the
.getRootDirectory result w/out even validating it or failing
if that file doesn't exist.
I appreciate the File object methods for pulling the path name apart; I'll use
that and that will work just fine. First time
playing around with all of this, so sometimes what you expect it to do is just
a matter of thinking about it a particular way :)
You can close this ticket out, as I'm sure I'll be able to work fine by using
the full path on the driver side and only the file
name on the worker side. Seems like it might be convenient though if these
matched set of routines did this themselves
since the driver side needs a qualified path to find the file, and the worker
side, by definition, strips that off and only put's
the file in the designated work directory (which make sense of course). No big
deal though.
Thanks again,
Tom
> SparkFiles.get gives me the wrong fully qualified path
> ------------------------------------------------------
>
> Key: SPARK-3769
> URL: https://issues.apache.org/jira/browse/SPARK-3769
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 1.0.2, 1.1.0
> Environment: linux host, and linux grid.
> Reporter: Tom Weber
> Priority: Minor
>
> My spark pgm running on my host, (submitting work to my grid).
> JavaSparkContext sc =new JavaSparkContext(conf);
> final String path = args[1];
> sc.addFile(path); /* args[1] = /opt/tom/SparkFiles.sas */
> The log shows:
> 14/10/02 16:07:14 INFO Utils: Copying /opt/tom/SparkFiles.sas to
> /tmp/spark-4c661c3f-cb57-4c9f-a0e9-c2162a89db77/SparkFiles.sas
> 14/10/02 16:07:15 INFO SparkContext: Added file /opt/tom/SparkFiles.sas at
> http://10.20.xx.xx:49587/files/SparkFiles.sas with timestamp 1412280434986
> those are paths on my host machine. The location that this file gets on grid
> nodes is:
> /opt/tom/spark-1.1.0-bin-hadoop2.4/work/app-20141002160704-0002/1/SparkFiles.sas
> While the call to get the path in my code that runs in my mapPartitions
> function on the grid nodes is:
> String pgm = SparkFiles.get(path);
> And this returns the following string:
> /opt/tom/spark-1.1.0-bin-hadoop2.4/work/app-20141002160704-0002/1/./opt/tom/SparkFiles.sas
> So, am I expected to take the qualified path that was given to me and parse
> it to get only the file name at the end, and then concatenate that to what I
> get from the SparkFiles.getRootDirectory() call in order to get this to work?
> Or pass only the parsed file name to the SparkFiles.get method? Seems as
> though I should be able to pass the same file specification to both
> sc.addFile() and SparkFiles.get() and get the correct location of the file.
> Thanks,
> Tom
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
