Re: Adding h5 files in a zip to use with PySpark

[Ashwin Raaghav]

Thanks! That worked! :)

And to read the files, I used pyspark.SparkFiles module.


On Thu, Jun 16, 2016 at 7:12 AM, Sun Rui  wrote:

> have you tried
> --files ?
> > On Jun 15, 2016, at 18:50, ar7  wrote:
> >
> > I am using PySpark 1.6.1 for my spark application. I have additional
> modules
> > which I am loading using the argument --py-files. I also have a h5 file
> > which I need to access from one of the modules for initializing the
> > ApolloNet.
> >
> > Is there any way I could access those files from the modules if I put
> them
> > in the same archive? I tried this approach but it was throwing an error
> > because the files are not there in every worker. I can think of one
> solution
> > which is copying the file to each of the workers but I want to know if
> there
> > are better ways to do it?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Adding-h5-files-in-a-zip-to-use-with-PySpark-tp27173.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> > For additional commands, e-mail: user-h...@spark.apache.org
> >
>
>
>


-- 
Regards,

Ashwin Raaghav

Re: Adding h5 files in a zip to use with PySpark

[Sun Rui]

have you tried
--files ?
> On Jun 15, 2016, at 18:50, ar7  wrote:
> 
> I am using PySpark 1.6.1 for my spark application. I have additional modules
> which I am loading using the argument --py-files. I also have a h5 file
> which I need to access from one of the modules for initializing the
> ApolloNet.
> 
> Is there any way I could access those files from the modules if I put them
> in the same archive? I tried this approach but it was throwing an error
> because the files are not there in every worker. I can think of one solution
> which is copying the file to each of the workers but I want to know if there
> are better ways to do it?
> 
> 
> 
> --
> View this message in context: 
> http://apache-spark-user-list.1001560.n3.nabble.com/Adding-h5-files-in-a-zip-to-use-with-PySpark-tp27173.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
> 



-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Adding h5 files in a zip to use with PySpark

[ar7]

I am using PySpark 1.6.1 for my spark application. I have additional modules
which I am loading using the argument --py-files. I also have a h5 file
which I need to access from one of the modules for initializing the
ApolloNet.

Is there any way I could access those files from the modules if I put them
in the same archive? I tried this approach but it was throwing an error
because the files are not there in every worker. I can think of one solution
which is copying the file to each of the workers but I want to know if there
are better ways to do it?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Adding-h5-files-in-a-zip-to-use-with-PySpark-tp27173.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org