Re: Accessing a file that was passed via --files to spark submit

[Marcelo Vanzin]

>From spark-submit -h:

  --files FILES   Comma-separated list of files to be
placed in the working
 directory of each executor. File paths of
these files
 in executors can be accessed via
SparkFiles.get(fileName).

On Sun, Mar 18, 2018 at 1:06 AM, Vitaliy Pisarev
 wrote:
> I am submitting a script to spark-submit and passing it a file using --files
> property. Later on I need to read it in a worker.
>
> I don't understand what API I should use to do that. I figured I'd try just:
>
> with open('myfile'):
>
> but this did not work.
>
> I am able to pass the file using the addFile mechanism but it may not be
> good enough for me.
>
> This may seem like a very simple question but I did not find any
> comprehensive documentation on spark-submit. The docs sure doen't cover it.
>
>



-- 
Marcelo

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Accessing a file that was passed via --files to spark submit

[Vitaliy Pisarev]

I am submitting a script to spark-submit and passing it a file using
--files property. Later on I need to read it in a worker.

I don't understand what API I should use to do that. I figured I'd try just:

with open('myfile'):

but this did not work.

I am able to pass the file using the addFile mechanism but it may not be
good enough for me.

This may seem like a very simple question but I did not find any
comprehensive documentation on spark-submit. The docs
 sure
doen't cover it.