unsubscribe
Re: Scala from Jupyter
Hello, As andy mentioned spark notebook works great out of the box and also you can create your own custom spark config (memory stuff, injecting custom jars into notebook, etc) to run for each notebook. I have been using it to write and test my production level spark applications using a huge chunk of production data in my local machine. It works really well and also you can highly customise your spark notebook package for various versions of big data components (like different hadoop, hive versions) on their website. Disclaimer: This is just my views and experience based over using zeppelin and spark notebook over a period of 5-6 months On Tue, Feb 16, 2016 at 8:36 PM, andy petrella <andy.petre...@gmail.com> wrote: > That's great man. > > Note, if you don't want to struggle too much that you just have to > download the version you need here http://spark-notebook.io/ > It can be whatever you want zip, tgz, docker or even deb ^^. So pick the > flavor you like the most. > > I'd recommend you two things: > 1. build on master to have access to coool features including the > variables panels (a la RStudio) > 2. go over the existing examples (around 40 atm), which are including > dynamic charts for streaming, graph, geospatial, machine learning, etc. > Some are funny :-P (the US History for instance). Oh, the charts are all > scala based but there are some implicits that will save your life coz they > will render what you were thinking of without asking... > > Have fun! > cheers, > > > > On Tue, Feb 16, 2016 at 1:43 PM Aleksandr Modestov < > aleksandrmodes...@gmail.com> wrote: > >> Thank you! >> I will test Spark Notebook. >> >> On Tue, Feb 16, 2016 at 3:37 PM, andy petrella <andy.petre...@gmail.com> >> wrote: >> >>> Hello Alex! >>> >>> Rajeev is right, come over the spark notebook gitter room, you'll be >>> helped by many experienced people if you have some troubles: >>> https://gitter.im/andypetrella/spark-notebook >>> >>> The spark notebook has many integrated, reactive (scala) and extendable >>> (scala) plotting capabilities. >>> >>> cheers and have fun! >>> andy >>> >>> On Tue, Feb 16, 2016 at 1:04 PM Rajeev Reddy <rajeev.redd...@gmail.com> >>> wrote: >>> >>>> Hello, >>>> >>>> Let me understand your query correctly. >>>> >>>> Case 1. You have a jupyter installation for python and you want to use >>>> it for scala. >>>> Solution: You can install kernels other than python Ref >>>> <https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages> >>>> >>>> Case 2. You want to use spark scala >>>> Solution: You can create a notebook config in which you create spark >>>> context and inject it back to your notebook OR Install other kind of >>>> notebooks like spark-notebook >>>> <https://github.com/andypetrella/spark-notebook/> or apache zeppelin >>>> <https://zeppelin.incubator.apache.org/> >>>> >>>> >>>> According to my experience for case 2. I have been using and prefer >>>> spark notebook over zeppelin >>>> >>>> >>>> On Tue, Feb 16, 2016 at 4:49 PM, AlexModestov < >>>> aleksandrmodes...@gmail.com> wrote: >>>> >>>>> Hello! >>>>> I want to use Scala from Jupyter (or may be something else if you could >>>>> recomend anything. I mean an IDE). Does anyone know how I can do this? >>>>> Thank you! >>>>> >>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Scala-from-Jupyter-tp26234.html >>>>> Sent from the Apache Spark User List mailing list archive at >>>>> Nabble.com. >>>>> >>>>> - >>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks, >>>> Rajeev Reddy >>>> Software Development Engineer 1 >>>> IXP - Information Intelligence (I2) Team >>>> Flipkart Internet Pvt. Ltd (flipkart.com) >>>> http://rajeev-reddy.com >>>> +91-8001618957 >>>> >>> -- >>> andy >>> >> >> -- > andy > -- Thanks, Rajeev Reddy Software Development Engineer 1 IXP - Information Intelligence (I2) Team http://rajeev-reddy.com +91-8001618957
Re: Scala from Jupyter
Hello, Let me understand your query correctly. Case 1. You have a jupyter installation for python and you want to use it for scala. Solution: You can install kernels other than python Ref <https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages> Case 2. You want to use spark scala Solution: You can create a notebook config in which you create spark context and inject it back to your notebook OR Install other kind of notebooks like spark-notebook <https://github.com/andypetrella/spark-notebook/> or apache zeppelin <https://zeppelin.incubator.apache.org/> According to my experience for case 2. I have been using and prefer spark notebook over zeppelin On Tue, Feb 16, 2016 at 4:49 PM, AlexModestov <aleksandrmodes...@gmail.com> wrote: > Hello! > I want to use Scala from Jupyter (or may be something else if you could > recomend anything. I mean an IDE). Does anyone know how I can do this? > Thank you! > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Scala-from-Jupyter-tp26234.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Thanks, Rajeev Reddy Software Development Engineer 1 IXP - Information Intelligence (I2) Team Flipkart Internet Pvt. Ltd (flipkart.com) http://rajeev-reddy.com +91-8001618957
Spark UI keeps redirecting to /null and returns 500
I am having problem in accessing spark UI while running in spark-client mode. It works fine in local mode. It keeps redirecting back to itself by adding /null at the end and ultimately run out of size limit for url and returns 500. Look at response below. I have a feeling that I might be missing some config, I played with various config settings for yarn with no success. I am using spark version 1.3.1 Any help will be greatly appreciated. --2015-09-09 11:22:17-- http://192.168.13.37:4040/ Connecting to 192.168.13.37:4040... connected. HTTP request sent, awaiting response... HTTP/1.1 302 Found Location: http://192.168.13.37:4040/null/ Content-Length: 0 Server: Jetty(8.y.z-SNAPSHOT) Location: http://192.168.13.37:4040/null/ [following] --2015-09-09 11:22:17-- http://192.168.13.37:4040/null/ Reusing existing connection to 192.168.13.37:4040. HTTP request sent, awaiting response... HTTP/1.1 302 Found Location: http://192.168.13.37:4040/null/null/null/ Content-Length: 0 Server: Jetty(8.y.z-SNAPSHOT) Location: http://192.168.13.37:4040/null/null/null/ [following] --2015-09-09 11:22:17-- http://192.168.13.37:4040/null/null/null/ Reusing existing connection to 192.168.13.37:4040. HTTP request sent, awaiting response... HTTP/1.1 302 Found Location: http://192.168.13.37:4040/null/null/null/null/null/null/null/ Content-Length: 0 Server: Jetty(8.y.z-SNAPSHOT) Location: http://192.168.13.37:4040/null/null/null/null/null/null/null/ [following] --2015-09-09 11:22:17-- http://192.168.13.37:4040/null/null/null/null/null/null/null/ Reusing existing connection to 192.168.13.37:4040. HTTP request sent, awaiting response... HTTP/1.1 302 Found Location: http://192.168.13.37:4040/null/null/null/null/null/null/null/null/null/null/null/null/null/null/null/ Content-Length: 0 Server: Jetty(8.y.z-SNAPSHOT) Location: http://192.168.13.37:4040/null/null/null/null/null/null/null/null/null/null/null/null/null/null/null/ [following] --2015-09-09 11:22:17-- http://192.168.13.37:4040/null/null/null/null/null/null/null/null/null/null/null/null/null/null/null/ Reusing existing connection to 192.168.13.37:4040. Here is stack dump: 15/09/09 11:22:18 WARN server.Response: Committed before 500 null 15/09/09 11:22:18 WARN server.AbstractHttpConnection:
Spark UI keep redirecting to /null and returns 500
Hi All, I am having problem in accessing spark UI while running in spark-client mode. It works fine in local mode. It keeps redirecting back to itself by adding /null at the end and ultimately run out of size limit for url and returns 500. Look at following below. I have a feeling that I might be missing some config, I played with various config setting for yarn with no success. I am using spark version 1.3.1 Any help will be greatly appreciated. --2015-09-09 11:22:17-- http://192.168.13.37:4040/ Connecting to 192.168.13.37:4040... connected. HTTP request sent, awaiting response... HTTP/1.1 302 Found Location: http://192.168.13.37:4040/null/ Content-Length: 0 Server: Jetty(8.y.z-SNAPSHOT) Location: http://192.168.13.37:4040/null/ [following] --2015-09-09 11:22:17-- http://192.168.13.37:4040/null/ Reusing existing connection to 192.168.13.37:4040. HTTP request sent, awaiting response... HTTP/1.1 302 Found Location: http://192.168.13.37:4040/null/null/null/ Content-Length: 0 Server: Jetty(8.y.z-SNAPSHOT) Location: http://192.168.13.37:4040/null/null/null/ [following] --2015-09-09 11:22:17-- http://192.168.13.37:4040/null/null/null/ Reusing existing connection to 192.168.13.37:4040. HTTP request sent, awaiting response... HTTP/1.1 302 Found Location: http://192.168.13.37:4040/null/null/null/null/null/null/null/ Content-Length: 0 Server: Jetty(8.y.z-SNAPSHOT) Location: http://192.168.13.37:4040/null/null/null/null/null/null/null/ [following] --2015-09-09 11:22:17-- http://192.168.13.37:4040/null/null/null/null/null/null/null/ Reusing existing connection to 192.168.13.37:4040. HTTP request sent, awaiting response... HTTP/1.1 302 Found Location: http://192.168.13.37:4040/null/null/null/null/null/null/null/null/null/null/null/null/null/null/null/ Content-Length: 0 Server: Jetty(8.y.z-SNAPSHOT) Location: http://192.168.13.37:4040/null/null/null/null/null/null/null/null/null/null/null/null/null/null/null/ [following] --2015-09-09 11:22:17-- http://192.168.13.37:4040/null/null/null/null/null/null/null/null/null/null/null/null/null/null/null/ Reusing existing connection to 192.168.13.37:4040. Here is stack dump: 15/09/09 11:22:18 WARN server.Response: Committed before 500 null 15/09/09 11:22:18 WARN server.AbstractHttpConnection:
