You can specify spark properties as mentioned here http://spark.apache.org/docs/latest/configuration.html They will be passed to spark via --conf
Manuel Sopena Ballesteros <manuel...@garvan.org.au> 于2019年11月15日周五 上午11:19写道： > Thank you very much, that worked > > > > What about passing –conf flag to pyspark? > > > > Manuel > > > > *From:* Jeff Zhang [mailto:zjf...@gmail.com] > *Sent:* Friday, November 15, 2019 12:35 PM > *To:* users > *Subject:* Re: send parameters to pyspark > > > > you can set property spark.jars > > > > Manuel Sopena Ballesteros <manuel...@garvan.org.au> 于2019年11月15日周五 上午9:30 > 写道： > > Dear zeppelin community, > > > > I need to send some parameters to pyspark so it can find extra jars. > > > > This is an example of the parameters I need to send to pyspark: > > > > pyspark \ > > --jars > /share/ClusterShare/anaconda3/envs/python37/lib/python3.7/site-packages/hail/hail-all-spark.jar > \ > > --conf > spark.driver.extraClassPath=/share/ClusterShare/anaconda3/envs/python37/lib/python3.7/site-packages/hail/hail-all-spark.jar > \ > > --conf > spark.executor.extraClassPath=/share/ClusterShare/anaconda3/envs/python37/lib/python3.7/site-packages/hail/hail-all-spark.jar > \ > > --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ > > --conf spark.kryo.registrator=is.hail.kryo.HailKryoRegistrator > > > > How could I configure my spark interpreter to do this? > > > > Thank you very much > > NOTICE > > Please consider the environment before printing this email. This message > and any attachments are intended for the addressee named and may contain > legally privileged/confidential/copyright information. If you are not the > intended recipient, you should not read, use, disclose, copy or distribute > this communication. If you have received this message in error please > notify us at once by return email and then delete both messages. We accept > no liability for the distribution of viruses or similar in electronic > communications. This notice should not be removed. > > > > > -- > > Best Regards > > Jeff Zhang > NOTICE > Please consider the environment before printing this email. This message > and any attachments are intended for the addressee named and may contain > legally privileged/confidential/copyright information. If you are not the > intended recipient, you should not read, use, disclose, copy or distribute > this communication. If you have received this message in error please > notify us at once by return email and then delete both messages. We accept > no liability for the distribution of viruses or similar in electronic > communications. This notice should not be removed. > -- Best Regards Jeff Zhang
