Re: Using SPARK packages in Spark Cluster

Monday, February 15, 2016 at 10:03 To: Jorge Machado <jom...@me.com> Cc: Spark Group <user@spark.apache.org> Subject: Re: Using SPARK packages in Spark Cluster Hi Jorge/ All, Please please please go through this link http://spark.apache.org/docs/latest/spark-standalone.html. The

Re: Using SPARK packages in Spark Cluster

Hi Jorge/ All, Please please please go through this link http://spark.apache.org/docs/latest/spark-standalone.html. The link tells you how to start a SPARK cluster in local mode. If you have not started or worked in SPARK cluster in

Re: Using SPARK packages in Spark Cluster

Hi Gourav, I did not unterstand your problem… the - - packages command should not make any difference if you are running standalone or in YARN for example. Give us an example what packages are you trying to load, and what error are you getting… If you want to use the libraries in

Re: Using SPARK packages in Spark Cluster

Hi, I am grateful for everyone's response, but sadly no one here actually has read the question before responding. Has anyone yet tried starting a SPARK cluster as mentioned in the link in my email? :) Regards, Gourav On Mon, Feb 15, 2016 at 11:16 AM, Jorge Machado wrote: >

Re: Using SPARK packages in Spark Cluster

$SPARK_HOME/bin/spark-shell --packages com.databricks:spark-csv_2.10:1.3.0 It will download everything for you and register into your JVM. If you want to use it in your Prod just package it with maven. > On 15/02/2016, at 12:14, Gourav Sengupta wrote: > > Hi, >

Re: Using SPARK packages in Spark Cluster

Hi, How to we include the following package: https://github.com/databricks/spark-csv while starting a SPARK standalone cluster as mentioned here: http://spark.apache.org/docs/latest/spark-standalone.html Thanks and Regards, Gourav Sengupta On Mon, Feb 15, 2016 at 10:32 AM, Ramanathan R

Re: Using SPARK packages in Spark Cluster

Hi Gourav, If your question is how to distribute python package dependencies across the Spark cluster programmatically? ...here is an example - $ export PYTHONPATH='path/to/thrift.zip:path/to/happybase.zip:path/to/your/py/application' And in code:

Re: Using SPARK packages in Spark Cluster

Hi, So far no one is able to get my question at all. I know what it takes to load packages via SPARK shell or SPARK submit. How do I load packages when starting a SPARK cluster, as mentioned here http://spark.apache.org/docs/latest/spark-standalone.html ? Regards, Gourav Sengupta On Mon,

Re: Using SPARK packages in Spark Cluster

Hi, I was interested in knowing how to load the packages into SPARK cluster started locally. Can someone pass me on the links to set the conf file so that the packages can be loaded? Regards, Gourav On Fri, Feb 12, 2016 at 6:52 PM, Burak Yavuz wrote: > Hello Gourav, > > The

Using SPARK packages in Spark Cluster

Hi, I am creating sparkcontext in a SPARK standalone cluster as mentioned here: http://spark.apache.org/docs/latest/spark-standalone.html using the following code: -- sc.stop()

Re: Using SPARK packages in Spark Cluster

Hello Gourav, The packages need to be loaded BEFORE you start the JVM, therefore you won't be able to add packages dynamically in code. You should use the --packages with pyspark before you start your application. One option is to add a `conf` that will load some packages if you are constantly