One option we have used in the past is to expose spark application functionality via REST, this would enable python or any other client that is capable of doing a HTTP request to integrate with your Spark application.
To get you started, this might be a useful reference http://blog.michaelhamrah.com/2013/06/scala-web-apis-up-and-running-with-spray-and-akka/ On Sun, Feb 28, 2016 at 10:38 AM, moshir mikael <moshir.mik...@gmail.com> wrote: > Ok, > but what do I need for the program to run. > In python sparkcontext = SparkContext(conf) only works when you have > spark installed locally. > AFAIK there is no *pyspark *package for python that you can install doing > pip install pyspark. > You actually need to install spark to get it running (e.g : > https://github.com/KristianHolsheimer/pyspark-setup-guide). > > Does it mean you need to install spark on the box your applications runs > to benefit from pyspark and this is required to connect to another remote > spark cluster ? > Am I missing something obvious ? > > > Le dim. 28 févr. 2016 à 19:01, Todd Nist <tsind...@gmail.com> a écrit : > >> Define your SparkConfig to set the master: >> >> val conf = new SparkConf().setAppName(AppName) >> .setMaster(SparkMaster) >> .set(....) >> >> Where SparkMaster = "spark://SparkServerHost:7077". So if your spark >> server hostname it "RADTech" then it would be "spark://RADTech:7077". >> >> Then when you create the SparkContext, pass the SparkConf to it: >> >> val sparkContext = new SparkContext(conf) >> >> Then use the sparkContext for interact with the SparkMaster / Cluster. >> Your program basically becomes the driver. >> >> HTH. >> >> -Todd >> >> On Sun, Feb 28, 2016 at 9:25 AM, mms <moshir.mik...@gmail.com> wrote: >> >>> Hi, I cannot find a simple example showing how a typical application can >>> 'connect' to a remote spark cluster and interact with it. Let's say I have >>> a Python web application hosted somewhere *outside *a spark cluster, >>> with just python installed on it. How can I talk to Spark without using a >>> notebook, or using ssh to connect to a cluster master node ? I know of >>> spark-submit and spark-shell, however forking a process on a remote host to >>> execute a shell script seems like a lot of effort What are the recommended >>> ways to connect and query Spark from a remote client ? Thanks Thx ! >>> ------------------------------ >>> View this message in context: Spark Integration Patterns >>> <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Integration-Patterns-tp26354.html> >>> Sent from the Apache Spark User List mailing list archive >>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >>> >> >> -- Luciano Resende http://people.apache.org/~lresende http://twitter.com/lresende1975 http://lresende.blogspot.com/
