Great, thanks. So here some suggestions: 1. We could create one new layer between Toolbox and the Engines, this would be Docker client. 2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL
It's not clear to me if the commands listed on General Commands will need to be written for every language, from the image I believe not, but how it would work, since starting a notebook in Python is different than starting in Scala. - Lucas On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi < daniel.takabaya...@gmail.com> wrote: > > https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello < > rafa.reis.nove...@gmail.com> escreveu: > > > Hi Taka, > > > > I can't see the image! Maybe the Apache email list don't support > > attachments. > > > > Best! > > Rafael J. R. Novello > > > > Skype: rafael.novello > > Blog: http://rafanovello.blogspot.com.br/ > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi < > > daniel.takabaya...@gmail.com> escreveu: > > > > > Hi all, > > > > > > To try to give a little more details about this "new concept" that we > > want > > > to bring to the marvin toolbox, I did this simple architecture draw. > > > > > > [image: marvin-architecture-views (5).png] > > > > > > The general idea here is try to transform the toolbox something > > > disconnected with the "language", something more agnostic. Also in this > > new > > > architecture we could use remote resource to process engines and make > > easy > > > the support for new languages. > > > > > > This "new toolbox" will be the only thing that a Marvin user must to > > > install and also we could start to support multiples O.S (once the REPL > > is > > > a dummy application that only interprets and by pass commands). > > > > > > Regards, > > > Taka > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello < > > > rafa.reis.nove...@gmail.com> escreveu: > > > > > >> Alan, > > >> > > >> Yes! We are using the Docker SDK and it's possible to use the API to > > >> automate, but it's a little bit harder than automate CLI calls. > > >> > > >> Atenciosamente, > > >> Rafael J. R. Novello > > >> > > >> Skype: rafael.novello > > >> Blog: http://rafanovello.blogspot.com.br/ > > >> > > >> > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org> > > >> escreveu: > > >> > > >> > Hi, > > >> > > > >> > One question here, I understand that we start to use with this PoC > the > > >> > Docker SDK API, right? > > >> > > > >> > Why not use the API to expose some endpoints to permit this kind of > > >> > automation by devops? > > >> > > > >> > I think it is possible and it solves the CLI problem, right? > > >> > > > >> > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello < > > >> > rafa.reis.nove...@gmail.com> > > >> > wrote: > > >> > > > >> > > Lucas, > > >> > > > > >> > > Sorry, I didn't understood your question bellow. > > >> > > > > >> > > "Would it make sense to use the same solution that we will use for > > >> > having a > > >> > > single-language REPL to have a single-language CLI?" > > >> > > > > >> > > For DevOps purposes, maybe this new toolbox concept is not ideal. > I > > >> think > > >> > > we can keep the CLI inside the docker container but it will not > easy > > >> to > > >> > > automate > > >> > > jobs by this way. > > >> > > > > >> > > How to deal with this issue? Voting? > > >> > > Atenciosamente, > > >> > > Rafael J. R. Novello > > >> > > > > >> > > Skype: rafael.novello > > >> > > Blog: http://rafanovello.blogspot.com.br/ > > >> > > > > >> > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel < > > >> > > lucasb...@apache.org> escreveu: > > >> > > > > >> > > > Got it! Thanks for clarifying. > > >> > > > > > >> > > > Would it make sense to use the same solution that we will use > for > > >> > having > > >> > > a > > >> > > > single-language REPL to have a single-language CLI? My only > > concern > > >> > with > > >> > > > killing the CLI is that you remove an essential feature for > > DevOps. > > >> > > > > > >> > > > - Lucas > > >> > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello < > > >> > > > rafa.reis.nove...@gmail.com> > > >> > > > wrote: > > >> > > > > > >> > > > > Lucas, > > >> > > > > > > >> > > > > The idea is that REPL will substitute the actual CLI. It's > > because > > >> > with > > >> > > > the > > >> > > > > actual concept (using language specific CLI) we will need to > > >> > > re-implement > > >> > > > > the same features many times and probably each CLI will have a > > >> > > different > > >> > > > > behavior because some language specific restrictions and/or > > >> > > limitations. > > >> > > > > > > >> > > > > With this new concept, all users will have the same experience > > >> > > > interacting > > >> > > > > with Marvin REPL and they will use the bests tools to develop > > >> their > > >> > > > engines > > >> > > > > (Jupyter Notebook and/or Lab with different languages support > or > > >> even > > >> > > > > Apache Zeppelin). All the interact will occur through REPL and > > >> docker > > >> > > > > protocol. > > >> > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the same but we can > use > > >> > docker > > >> > > > to > > >> > > > > do the same job. > > >> > > > > > > >> > > > > Best! > > >> > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel < > > >> > > > > lucasb...@apache.org> escreveu: > > >> > > > > > > >> > > > > > Thanks for the clarifications Rafael, so from what I > > understood, > > >> > > > Marvin's > > >> > > > > > developers would not use the REPL to explore data and test > > >> models, > > >> > > i.e. > > >> > > > > > develop the engine. Is the idea to build something more like > > an > > >> > > > > interactive > > >> > > > > > CLI? I think an interactive CLI would be useful for the > > >> developer > > >> > > > > > experience, however, it's important to keep the unattended > > >> > (current) > > >> > > > CLI > > >> > > > > in > > >> > > > > > place for scenarios where users want to automate Marvin > tasks. > > >> > > > > > > > >> > > > > > Alan, I believe the idea is still the same as you started, > but > > >> > using > > >> > > > > > docker SDK now. > > >> > > > > > > > >> > > > > > - Lucas > > >> > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello < > > >> > > > > > rafa.reis.nove...@gmail.com> > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > Hi Lucas! > > >> > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache! > > >> > > > > > > > > >> > > > > > > Let me help with some comments: > > >> > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and the > easiest > > >> one > > >> > was > > >> > > > > > Python. > > >> > > > > > > We have found a small project [1] that have all features > we > > >> > desired > > >> > > > for > > >> > > > > > > REPL: > > >> > > > > > > - Autocomplete > > >> > > > > > > - Python commands disabled (the user have only the > commands > > we > > >> > > > > provide). > > >> > > > > > > The other languages REPL options don't have this feature. > > >> > > > > > > - Easy to show output > > >> > > > > > > - Etc > > >> > > > > > > So, I think the language chosen will not be important here > > >> > because > > >> > > > the > > >> > > > > > user > > >> > > > > > > will only interact with the commands that we create. > > >> > > > > > > > > >> > > > > > > 2 - The "engine-generate" command will download a docker > > image > > >> > that > > >> > > > we > > >> > > > > > > create for that language and start a container with the > > basic > > >> > > project > > >> > > > > > > structure for that language. > > >> > > > > > > > > >> > > > > > > 3 - The REPL client will use the "docker protocol" to run > > >> > command, > > >> > > > > > > start/stop services and etc inside the container and it > will > > >> > > receive > > >> > > > > the > > >> > > > > > > log stream to show. No, the REPL will no pass code > snippets > > >> for > > >> > > > docker > > >> > > > > > > container (I think it will not be necessary) > > >> > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first item. > > >> > > > > > > > > >> > > > > > > [1] - https://github.com/italorossi/ishell > > >> > > > > > > > > >> > > > > > > Let me know if there is any other question! > > >> > > > > > > Best! > > >> > > > > > > Rafael J. R. Novello > > >> > > > > > > > > >> > > > > > > Skype: rafael.novello > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/ > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel < > > >> > > > > > > lucasb...@apache.org> escreveu: > > >> > > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache > > >> > > > > > > > > > >> > > > > > > > I have a few questions about the previous explanation: > > >> > > > > > > > 1) The REPL itself, it would be an application in which > > >> > > language? > > >> > > > > > > Remember > > >> > > > > > > > that the main idea is to allow the user to program on > his > > >> > > preferred > > >> > > > > > > > language in the REPL. > > >> > > > > > > > 2) Should the engine-generate command also generate a > > >> docker > > >> > > image > > >> > > > > > with > > >> > > > > > > > the user's application? > > >> > > > > > > > 3) What type of communication would happen between the > > REPL > > >> > and > > >> > > > the > > >> > > > > > > engine > > >> > > > > > > > via Docker SDK? Would the REPL pass snippets of code to > be > > >> > > executed > > >> > > > > by > > >> > > > > > > the > > >> > > > > > > > docker container? > > >> > > > > > > > 4) Have you considered code completion in the REPL? > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei < > > >> > > > yifei.z.l...@gmail.com> > > >> > > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > Ok guys, > > >> > > > > > > > > > > >> > > > > > > > > The basic idea is to provide only one Toolbox for > > multiple > > >> > > > > languages. > > >> > > > > > > > > We are looking for possibility to build a single > Marvin > > >> Repl, > > >> > > > > > > > > instead of severals toolboxes with differentes > > interfaces > > >> or > > >> > > > > > commands. > > >> > > > > > > > > > > >> > > > > > > > > In this case, the engine-generate command will > download > > >> and > > >> > > > start a > > >> > > > > > > > Docker > > >> > > > > > > > > container with basic engine structure corresponding > the > > >> > choosed > > >> > > > > > > language. > > >> > > > > > > > > this means we don't need to build Toolboxes of > > differents > > >> > > > > languages, > > >> > > > > > we > > >> > > > > > > > > just > > >> > > > > > > > > build the engine template of all languages that we > want > > to > > >> > > > support > > >> > > > > > and > > >> > > > > > > > > provide it > > >> > > > > > > > > as Docker containers > > >> > > > > > > > > > > >> > > > > > > > > We have started researches around our basic > requirements > > >> > like: > > >> > > > > > > > > - Repl interface > > >> > > > > > > > > - System communication > > >> > > > > > > > > - Connection security > > >> > > > > > > > > - Tool popularity > > >> > > > > > > > > - Update complexity > > >> > > > > > > > > - Languages support > > >> > > > > > > > > - ...... > > >> > > > > > > > > > > >> > > > > > > > > And we did some POC with code here: > > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl > > >> > > > > > > > > > > >> > > > > > > > > There is POC testing gRPC using Scala and Python, > > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS, > > >> > > > > > > > > Repl interface and Docker SDK with Python. > > >> > > > > > > > > > > >> > > > > > > > > At this moment we prefer the Repl interface + Docker > SDK > > >> way, > > >> > > > > because > > >> > > > > > > > good > > >> > > > > > > > > part of the requirements > > >> > > > > > > > > will be guaranteed by Docker. > > >> > > > > > > > > > > >> > > > > > > > > With this informations, what do you think? Should we > > >> submit > > >> > all > > >> > > > > this > > >> > > > > > > POCs > > >> > > > > > > > > to Apache Repo? > > >> > > > > > > > > Please feel free to opine. > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > Thats all, thanks!!! > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel > Takabayashi < > > >> > > > > > > > > daniel.takabaya...@gmail.com> escreveu: > > >> > > > > > > > > > > >> > > > > > > > > > Zhang, > > >> > > > > > > > > > > > >> > > > > > > > > > I think the best approach is give us a better > > >> explanation > > >> > > about > > >> > > > > > this > > >> > > > > > > > new > > >> > > > > > > > > > feature and how this can help us to archive the > > desired > > >> > > feature > > >> > > > > > > > (support > > >> > > > > > > > > > multiple languages). Than if the majority agree > than I > > >> can > > >> > do > > >> > > > the > > >> > > > > > > > merge, > > >> > > > > > > > > > just like I did before, but in another branch. > > >> > > > > > > > > > > > >> > > > > > > > > > Let me know if makes sense, ok? > > >> > > > > > > > > > > > >> > > > > > > > > > Taka > > >> > > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende > < > > >> > > > > > > > > luckbr1...@gmail.com> > > >> > > > > > > > > > escreveu: > > >> > > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a refactoring of the > > >> existing > > >> > > > repo? > > >> > > > > Or > > >> > > > > > > is > > >> > > > > > > > > > > it a new rewrite of the repo? > > >> > > > > > > > > > > > > >> > > > > > > > > > > Just asking as it might make sense to make it a > > branch > > >> > then > > >> > > > > > > actually > > >> > > > > > > > a > > >> > > > > > > > > > > thing in parallel, as this will have an effect for > > >> > > releases, > > >> > > > > etc. > > >> > > > > > > but > > >> > > > > > > > > > > you guys know more here than I do. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Also, it's probably good to have a write up of the > > >> main > > >> > > > > direction > > >> > > > > > > of > > >> > > > > > > > > > > the design that can help people get familiar with > > the > > >> new > > >> > > > > > approach. > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei < > > >> > > > > > > yifei.z.l...@gmail.com > > >> > > > > > > > > > > >> > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Hey guys, > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > We have reorganized the Poc repo, and want to > > merge > > >> it > > >> > to > > >> > > > our > > >> > > > > > > > Apache > > >> > > > > > > > > > > repo. > > >> > > > > > > > > > > > Just making sure here before we do the merge, > > >> because > > >> > im > > >> > > > not > > >> > > > > > sure > > >> > > > > > > > if > > >> > > > > > > > > > > doing > > >> > > > > > > > > > > > this will perserve the Git commit history. > > >> > > > > > > > > > > > What we are planning to do is: > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > git filter-branch --subdirectory-filter <git > > >> > > > > origin_repository > > >> > > > > > > > > > directory> > > >> > > > > > > > > > > > -- --all > > >> > > > > > > > > > > > mkdir POCs/ > > >> > > > > > > > > > > > git mv * POCs/ > > >> > > > > > > > > > > > git commit -m "colleted the data to move" > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > git clone <git destination_repository_url> cd > <git > > >> > > > > > > > > > > > destination_repository_directory> git remote add > > >> > > > > > > > > > origin_repository_branch > > >> > > > > > > > > > > > <git origin_repository directory> git pull > > >> > > > > > > origin_repository_branch > > >> > > > > > > > > > > master > > >> > > > > > > > > > > > --allow-unrelated-histories git remote rm > > >> > > > > > > origin_repository_branch > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance ! > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > -------------------------------------------------------------- > > >> > > > > > > > > > > > Zhang Yifei > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > -- > > >> > > > > > > > > > > Luciano Resende > > >> > > > > > > > > > > http://twitter.com/lresende1975 > > >> > > > > > > > > > > http://lresende.blogspot.com/ > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > -- > > >> > > > > > > > > > > >> > -------------------------------------------------------------- > > >> > > > > > > > > Zhang Yifei > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
