I believe you are looking for something like Spark Jobserver for running jobs & JDBC server for accessing data? I am curious to know more about it, any further discussion will be very helpful
On Mon, Feb 29, 2016 at 6:06 AM, Luciano Resende <luckbr1...@gmail.com> wrote: > One option we have used in the past is to expose spark application > functionality via REST, this would enable python or any other client that > is capable of doing a HTTP request to integrate with your Spark application. > > To get you started, this might be a useful reference > > > http://blog.michaelhamrah.com/2013/06/scala-web-apis-up-and-running-with-spray-and-akka/ > > > On Sun, Feb 28, 2016 at 10:38 AM, moshir mikael <moshir.mik...@gmail.com> > wrote: > >> Ok, >> but what do I need for the program to run. >> In python sparkcontext = SparkContext(conf) only works when you have >> spark installed locally. >> AFAIK there is no *pyspark *package for python that you can install >> doing pip install pyspark. >> You actually need to install spark to get it running (e.g : >> https://github.com/KristianHolsheimer/pyspark-setup-guide). >> >> Does it mean you need to install spark on the box your applications runs >> to benefit from pyspark and this is required to connect to another remote >> spark cluster ? >> Am I missing something obvious ? >> >> >> Le dim. 28 févr. 2016 à 19:01, Todd Nist <tsind...@gmail.com> a écrit : >> >>> Define your SparkConfig to set the master: >>> >>> val conf = new SparkConf().setAppName(AppName) >>> .setMaster(SparkMaster) >>> .set(....) >>> >>> Where SparkMaster = "spark://SparkServerHost:7077". So if your spark >>> server hostname it "RADTech" then it would be "spark://RADTech:7077". >>> >>> Then when you create the SparkContext, pass the SparkConf to it: >>> >>> val sparkContext = new SparkContext(conf) >>> >>> Then use the sparkContext for interact with the SparkMaster / Cluster. >>> Your program basically becomes the driver. >>> >>> HTH. >>> >>> -Todd >>> >>> On Sun, Feb 28, 2016 at 9:25 AM, mms <moshir.mik...@gmail.com> wrote: >>> >>>> Hi, I cannot find a simple example showing how a typical application >>>> can 'connect' to a remote spark cluster and interact with it. Let's say I >>>> have a Python web application hosted somewhere *outside *a spark >>>> cluster, with just python installed on it. How can I talk to Spark without >>>> using a notebook, or using ssh to connect to a cluster master node ? I know >>>> of spark-submit and spark-shell, however forking a process on a remote host >>>> to execute a shell script seems like a lot of effort What are the >>>> recommended ways to connect and query Spark from a remote client ? Thanks >>>> Thx ! >>>> ------------------------------ >>>> View this message in context: Spark Integration Patterns >>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Integration-Patterns-tp26354.html> >>>> Sent from the Apache Spark User List mailing list archive >>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >>>> >>> >>> > > > -- > Luciano Resende > http://people.apache.org/~lresende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ > -- Best Regards, Ayan Guha
