Hi there I am moving this discussion to the dev list.
I was able to reproduce up to a certain extent the first video except that the view is not updated following a modification of the resource (the html file in my case). As far as helium is concerned, I would be happy if you could share the helium.conf file to test your 3 apps located in https://github.com/zeppelin-project/helium-packages. I found the possibility to run external application very interesting - and this is from my perspective the breakthrough proposed here. As far as the transfer of data across interpreters is concerned (as this was the initial request), I was wondering if we could have something more platform independent - Doan suggested thrift. I believe that this would allow more polyglot applications. Currently Zeppelin does offer a thrift interface but to my knowledge only Java interpreter application have shown up. Has anyone considered implementing a plain Python or Julia interpreter ? Cheers Guillaume On 29 October 2015 at 20:01, DuyHai Doan <doanduy...@gmail.com> wrote: > Look like the > https://github.com/Leemoonsoo/incubator-zeppelin/blob/helium/zeppelin-interpreter/src/main/java/org/apache/zeppelin/resource/ResourcePool.java > is very similar to my idea :) > > On Thu, Oct 29, 2015 at 8:57 PM, DuyHai Doan <doanduy...@gmail.com> wrote: > >> Thanks for the pointer on resource pool Moon. >> >> My first idea was: >> >> 1. create another Zeppelin infrastructure interpreter who will embed an >> in-memory DB (Apache xxx) >> 2. extends the Interpreter base class to create a connection pool and >> offer base methods like put(String key, Object value) and get(String key). >> 3. communication between each interpreter and the in-memory DB will be >> done using Thrift >> 4. optionally the in-memory DB can persist data on local storage (or >> implementable extension) for durability >> >> I don't know how resource pool is implemented, I will throw an eye to it >> >> On Thu, Oct 29, 2015 at 8:40 PM, moon soo Lee <m...@apache.org> wrote: >> >>> Yes it is. >>> >>> ResourcePool class in PoC implements sharing. >>> But ofcourse we can discuss better design and implementation. >>> >>> Thanks, >>> moon >>> >>> On 2015년 10월 29일 (목) at 오후 7:06 tog <guillaume.all...@gmail.com> wrote: >>> >>>> Moon >>>> >>>> Is the POC described in the wiki already implementing that sharing? >>>> >>>> Cheers >>>> Guillaume >>>> On Oct 29, 2015 5:26 PM, "moon soo Lee" <leemoon...@gmail.com> wrote: >>>> >>>>> Actually, Helium proposal ( >>>>> https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal) >>>>> includes one "Resource pool" . Which is basically distributed map across >>>>> interpreter processes. Please take a look if you're interested. >>>>> >>>>> Thanks, >>>>> moon >>>>> On 2015년 10월 29일 (목) at 오후 6:17 tog <guillaume.all...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Doan, >>>>>> What if not all interpreters will share the same memory/machine? I >>>>>> guess we should propose something that should work also in the case of >>>>>> remote interpreters. >>>>>> But may be this is already something you have in mind. >>>>>> >>>>>> Cheers >>>>>> Guillaume >>>>>> >>>>>> On 29 October 2015 at 16:58, DuyHai Doan <doanduy...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I am thinking about a new idea to propose a common in-memory shared >>>>>>> storage so that all interpreters can pass around variables. I'll create >>>>>>> a >>>>>>> JIRA soon to submit the idea and the architecture >>>>>>> >>>>>>> On Thu, Oct 29, 2015 at 5:40 PM, moon soo Lee <m...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> If your custom interpreter is in the same interpreter group >>>>>>>> 'spark', you can exchange data between SparkInterpreter and your custom >>>>>>>> interpreter. (because of interpreters in the same group runs in the >>>>>>>> same >>>>>>>> process) >>>>>>>> >>>>>>>> But if your custom interpreter is in the different interpreter >>>>>>>> group, then only way at the moment is persist data from >>>>>>>> SparkInterpreter >>>>>>>> and read data in your custom interpreter. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> moon >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 29, 2015 at 11:07 AM Miyuru Dayarathna < >>>>>>>> miyu...@yahoo.co.uk> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I am trying to access the Spark data frame defined in the Zeppelin >>>>>>>>> Tutorial notebook from a separate paragraph using a custom written >>>>>>>>> Zeppelin >>>>>>>>> Interpreter. To make it more clear given below is the code snippet >>>>>>>>> from >>>>>>>>> "Load data into table" paragraph of the Zeppelin Tutorial notebook. >>>>>>>>> When >>>>>>>>> this is run the data frame called "bank" gets initialized in a Spark >>>>>>>>> Interpreter process. I want to use the bank data frame from my custom >>>>>>>>> Zeppelin interpreter. Can you please let me know how to do this? Is >>>>>>>>> there a >>>>>>>>> Zeppelin API which provides me the access to such variables running >>>>>>>>> in a >>>>>>>>> different Interpreter than where they were instantiated? >>>>>>>>> >>>>>>>>> //---------------------------------- >>>>>>>>> >>>>>>>>> val bank = bankText.map(s => s.split(";")).filter(s => s(0) != >>>>>>>>> "\"age\"").map( >>>>>>>>> s => Bank(s(0).toInt, >>>>>>>>> s(1).replaceAll("\"", ""), >>>>>>>>> s(2).replaceAll("\"", ""), >>>>>>>>> s(3).replaceAll("\"", ""), >>>>>>>>> s(5).replaceAll("\"", "").toInt >>>>>>>>> ) >>>>>>>>> ).toDF() >>>>>>>>> >>>>>>>>> bank.registerTempTable("bank") >>>>>>>>> >>>>>>>>> //---------------------------------- >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Miyuru >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net >>>>>> >>>>> >> > -- PGP KeyID: 2048R/EA31CFC9 subkeys.pgp.net
