Thanks! These aren't in the docs, I will make a JIRA to add them. On Monday, April 4, 2016, Saisai Shao <[email protected]> wrote:
> spark.jars.ivy, spark.jars.packages, spark.jars.excludes is the > configurations you can use. > > Thanks > Saisai > > On Sun, Apr 3, 2016 at 1:59 AM, Russell Jurney <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > >> Thanks, Andy! >> >> On Mon, Mar 28, 2016 at 8:44 AM, Andy Davidson < >> [email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >> >>> Hi Russell >>> >>> I use Jupyter python notebooks a lot. Here is how I start the server >>> >>> set -x # turn debugging on >>> >>> #set +x # turn debugging off >>> >>> >>> # https://github.com/databricks/spark-csv >>> >>> # http://spark-packages.org/package/datastax/spark-cassandra-connector >>> >>> # >>> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md >>> >>> # >>> https://github.com/datastax/spark-cassandra-connector/blob/master/doc/15_python.md#pyspark-with-data-frames >>> >>> >>> # packages are ',' seperate with no white space >>> >>> extraPkgs="--packages >>> com.databricks:spark-csv_2.11:1.3.0,datastax:spark-cassandra-connector:1.6.0-M1-s_2.10" >>> >>> >>> export PYSPARK_PYTHON=python3 >>> >>> export PYSPARK_DRIVER_PYTHON=python3 >>> >>> IPYTHON_OPTS=notebook $SPARK_ROOT/bin/pyspark $extraPkgs --conf >>> spark.cassandra.connection.host= >>> ec2-54-153-102-232.us-west-1.compute.amazonaws.com $* >>> >>> >>> >>> From: Russell Jurney <[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>> Date: Sunday, March 27, 2016 at 7:22 PM >>> To: "user @spark" <[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>> Subject: --packages configuration equivalent item name? >>> >>> I run PySpark with CSV support like so: IPYTHON=1 pyspark --packages >>> com.databricks:spark-csv_2.10:1.4.0 >>> >>> I don't want to type this --packages argument each time. Is there a >>> config item for --packages? I can't find one in the reference at >>> http://spark.apache.org/docs/latest/configuration.html >>> >>> If there is no way to do this, please let me know so I can make a JIRA >>> for this feature. >>> >>> Thanks! >>> -- >>> Russell Jurney twitter.com/rjurney [email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');> relato.io >>> >>> >> >> >> -- >> Russell Jurney twitter.com/rjurney [email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');> relato.io >> > > -- Russell Jurney twitter.com/rjurney [email protected] relato.io
