https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
Em qui, 25 de out de 2018 às 07:38, Rafael Novello < rafa.reis.nove...@gmail.com> escreveu: > Hi Taka, > > I can't see the image! Maybe the Apache email list don't support > attachments. > > Best! > Rafael J. R. Novello > > Skype: rafael.novello > Blog: http://rafanovello.blogspot.com.br/ > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi < > daniel.takabaya...@gmail.com> escreveu: > > > Hi all, > > > > To try to give a little more details about this "new concept" that we > want > > to bring to the marvin toolbox, I did this simple architecture draw. > > > > [image: marvin-architecture-views (5).png] > > > > The general idea here is try to transform the toolbox something > > disconnected with the "language", something more agnostic. Also in this > new > > architecture we could use remote resource to process engines and make > easy > > the support for new languages. > > > > This "new toolbox" will be the only thing that a Marvin user must to > > install and also we could start to support multiples O.S (once the REPL > is > > a dummy application that only interprets and by pass commands). > > > > Regards, > > Taka > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello < > > rafa.reis.nove...@gmail.com> escreveu: > > > >> Alan, > >> > >> Yes! We are using the Docker SDK and it's possible to use the API to > >> automate, but it's a little bit harder than automate CLI calls. > >> > >> Atenciosamente, > >> Rafael J. R. Novello > >> > >> Skype: rafael.novello > >> Blog: http://rafanovello.blogspot.com.br/ > >> > >> > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org> > >> escreveu: > >> > >> > Hi, > >> > > >> > One question here, I understand that we start to use with this PoC the > >> > Docker SDK API, right? > >> > > >> > Why not use the API to expose some endpoints to permit this kind of > >> > automation by devops? > >> > > >> > I think it is possible and it solves the CLI problem, right? > >> > > >> > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello < > >> > rafa.reis.nove...@gmail.com> > >> > wrote: > >> > > >> > > Lucas, > >> > > > >> > > Sorry, I didn't understood your question bellow. > >> > > > >> > > "Would it make sense to use the same solution that we will use for > >> > having a > >> > > single-language REPL to have a single-language CLI?" > >> > > > >> > > For DevOps purposes, maybe this new toolbox concept is not ideal. I > >> think > >> > > we can keep the CLI inside the docker container but it will not easy > >> to > >> > > automate > >> > > jobs by this way. > >> > > > >> > > How to deal with this issue? Voting? > >> > > Atenciosamente, > >> > > Rafael J. R. Novello > >> > > > >> > > Skype: rafael.novello > >> > > Blog: http://rafanovello.blogspot.com.br/ > >> > > > >> > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel < > >> > > lucasb...@apache.org> escreveu: > >> > > > >> > > > Got it! Thanks for clarifying. > >> > > > > >> > > > Would it make sense to use the same solution that we will use for > >> > having > >> > > a > >> > > > single-language REPL to have a single-language CLI? My only > concern > >> > with > >> > > > killing the CLI is that you remove an essential feature for > DevOps. > >> > > > > >> > > > - Lucas > >> > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello < > >> > > > rafa.reis.nove...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > Lucas, > >> > > > > > >> > > > > The idea is that REPL will substitute the actual CLI. It's > because > >> > with > >> > > > the > >> > > > > actual concept (using language specific CLI) we will need to > >> > > re-implement > >> > > > > the same features many times and probably each CLI will have a > >> > > different > >> > > > > behavior because some language specific restrictions and/or > >> > > limitations. > >> > > > > > >> > > > > With this new concept, all users will have the same experience > >> > > > interacting > >> > > > > with Marvin REPL and they will use the bests tools to develop > >> their > >> > > > engines > >> > > > > (Jupyter Notebook and/or Lab with different languages support or > >> even > >> > > > > Apache Zeppelin). All the interact will occur through REPL and > >> docker > >> > > > > protocol. > >> > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the same but we can use > >> > docker > >> > > > to > >> > > > > do the same job. > >> > > > > > >> > > > > Best! > >> > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel < > >> > > > > lucasb...@apache.org> escreveu: > >> > > > > > >> > > > > > Thanks for the clarifications Rafael, so from what I > understood, > >> > > > Marvin's > >> > > > > > developers would not use the REPL to explore data and test > >> models, > >> > > i.e. > >> > > > > > develop the engine. Is the idea to build something more like > an > >> > > > > interactive > >> > > > > > CLI? I think an interactive CLI would be useful for the > >> developer > >> > > > > > experience, however, it's important to keep the unattended > >> > (current) > >> > > > CLI > >> > > > > in > >> > > > > > place for scenarios where users want to automate Marvin tasks. > >> > > > > > > >> > > > > > Alan, I believe the idea is still the same as you started, but > >> > using > >> > > > > > docker SDK now. > >> > > > > > > >> > > > > > - Lucas > >> > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello < > >> > > > > > rafa.reis.nove...@gmail.com> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hi Lucas! > >> > > > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache! > >> > > > > > > > >> > > > > > > Let me help with some comments: > >> > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and the easiest > >> one > >> > was > >> > > > > > Python. > >> > > > > > > We have found a small project [1] that have all features we > >> > desired > >> > > > for > >> > > > > > > REPL: > >> > > > > > > - Autocomplete > >> > > > > > > - Python commands disabled (the user have only the commands > we > >> > > > > provide). > >> > > > > > > The other languages REPL options don't have this feature. > >> > > > > > > - Easy to show output > >> > > > > > > - Etc > >> > > > > > > So, I think the language chosen will not be important here > >> > because > >> > > > the > >> > > > > > user > >> > > > > > > will only interact with the commands that we create. > >> > > > > > > > >> > > > > > > 2 - The "engine-generate" command will download a docker > image > >> > that > >> > > > we > >> > > > > > > create for that language and start a container with the > basic > >> > > project > >> > > > > > > structure for that language. > >> > > > > > > > >> > > > > > > 3 - The REPL client will use the "docker protocol" to run > >> > command, > >> > > > > > > start/stop services and etc inside the container and it will > >> > > receive > >> > > > > the > >> > > > > > > log stream to show. No, the REPL will no pass code snippets > >> for > >> > > > docker > >> > > > > > > container (I think it will not be necessary) > >> > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first item. > >> > > > > > > > >> > > > > > > [1] - https://github.com/italorossi/ishell > >> > > > > > > > >> > > > > > > Let me know if there is any other question! > >> > > > > > > Best! > >> > > > > > > Rafael J. R. Novello > >> > > > > > > > >> > > > > > > Skype: rafael.novello > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/ > >> > > > > > > > >> > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel < > >> > > > > > > lucasb...@apache.org> escreveu: > >> > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache > >> > > > > > > > > >> > > > > > > > I have a few questions about the previous explanation: > >> > > > > > > > 1) The REPL itself, it would be an application in which > >> > > language? > >> > > > > > > Remember > >> > > > > > > > that the main idea is to allow the user to program on his > >> > > preferred > >> > > > > > > > language in the REPL. > >> > > > > > > > 2) Should the engine-generate command also generate a > >> docker > >> > > image > >> > > > > > with > >> > > > > > > > the user's application? > >> > > > > > > > 3) What type of communication would happen between the > REPL > >> > and > >> > > > the > >> > > > > > > engine > >> > > > > > > > via Docker SDK? Would the REPL pass snippets of code to be > >> > > executed > >> > > > > by > >> > > > > > > the > >> > > > > > > > docker container? > >> > > > > > > > 4) Have you considered code completion in the REPL? > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei < > >> > > > yifei.z.l...@gmail.com> > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Ok guys, > >> > > > > > > > > > >> > > > > > > > > The basic idea is to provide only one Toolbox for > multiple > >> > > > > languages. > >> > > > > > > > > We are looking for possibility to build a single Marvin > >> Repl, > >> > > > > > > > > instead of severals toolboxes with differentes > interfaces > >> or > >> > > > > > commands. > >> > > > > > > > > > >> > > > > > > > > In this case, the engine-generate command will download > >> and > >> > > > start a > >> > > > > > > > Docker > >> > > > > > > > > container with basic engine structure corresponding the > >> > choosed > >> > > > > > > language. > >> > > > > > > > > this means we don't need to build Toolboxes of > differents > >> > > > > languages, > >> > > > > > we > >> > > > > > > > > just > >> > > > > > > > > build the engine template of all languages that we want > to > >> > > > support > >> > > > > > and > >> > > > > > > > > provide it > >> > > > > > > > > as Docker containers > >> > > > > > > > > > >> > > > > > > > > We have started researches around our basic requirements > >> > like: > >> > > > > > > > > - Repl interface > >> > > > > > > > > - System communication > >> > > > > > > > > - Connection security > >> > > > > > > > > - Tool popularity > >> > > > > > > > > - Update complexity > >> > > > > > > > > - Languages support > >> > > > > > > > > - ...... > >> > > > > > > > > > >> > > > > > > > > And we did some POC with code here: > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl > >> > > > > > > > > > >> > > > > > > > > There is POC testing gRPC using Scala and Python, > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS, > >> > > > > > > > > Repl interface and Docker SDK with Python. > >> > > > > > > > > > >> > > > > > > > > At this moment we prefer the Repl interface + Docker SDK > >> way, > >> > > > > because > >> > > > > > > > good > >> > > > > > > > > part of the requirements > >> > > > > > > > > will be guaranteed by Docker. > >> > > > > > > > > > >> > > > > > > > > With this informations, what do you think? Should we > >> submit > >> > all > >> > > > > this > >> > > > > > > POCs > >> > > > > > > > > to Apache Repo? > >> > > > > > > > > Please feel free to opine. > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > Thats all, thanks!!! > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi < > >> > > > > > > > > daniel.takabaya...@gmail.com> escreveu: > >> > > > > > > > > > >> > > > > > > > > > Zhang, > >> > > > > > > > > > > >> > > > > > > > > > I think the best approach is give us a better > >> explanation > >> > > about > >> > > > > > this > >> > > > > > > > new > >> > > > > > > > > > feature and how this can help us to archive the > desired > >> > > feature > >> > > > > > > > (support > >> > > > > > > > > > multiple languages). Than if the majority agree than I > >> can > >> > do > >> > > > the > >> > > > > > > > merge, > >> > > > > > > > > > just like I did before, but in another branch. > >> > > > > > > > > > > >> > > > > > > > > > Let me know if makes sense, ok? > >> > > > > > > > > > > >> > > > > > > > > > Taka > >> > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende < > >> > > > > > > > > luckbr1...@gmail.com> > >> > > > > > > > > > escreveu: > >> > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a refactoring of the > >> existing > >> > > > repo? > >> > > > > Or > >> > > > > > > is > >> > > > > > > > > > > it a new rewrite of the repo? > >> > > > > > > > > > > > >> > > > > > > > > > > Just asking as it might make sense to make it a > branch > >> > then > >> > > > > > > actually > >> > > > > > > > a > >> > > > > > > > > > > thing in parallel, as this will have an effect for > >> > > releases, > >> > > > > etc. > >> > > > > > > but > >> > > > > > > > > > > you guys know more here than I do. > >> > > > > > > > > > > > >> > > > > > > > > > > Also, it's probably good to have a write up of the > >> main > >> > > > > direction > >> > > > > > > of > >> > > > > > > > > > > the design that can help people get familiar with > the > >> new > >> > > > > > approach. > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei < > >> > > > > > > yifei.z.l...@gmail.com > >> > > > > > > > > > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > Hey guys, > >> > > > > > > > > > > > > >> > > > > > > > > > > > We have reorganized the Poc repo, and want to > merge > >> it > >> > to > >> > > > our > >> > > > > > > > Apache > >> > > > > > > > > > > repo. > >> > > > > > > > > > > > Just making sure here before we do the merge, > >> because > >> > im > >> > > > not > >> > > > > > sure > >> > > > > > > > if > >> > > > > > > > > > > doing > >> > > > > > > > > > > > this will perserve the Git commit history. > >> > > > > > > > > > > > What we are planning to do is: > >> > > > > > > > > > > > > >> > > > > > > > > > > > git filter-branch --subdirectory-filter <git > >> > > > > origin_repository > >> > > > > > > > > > directory> > >> > > > > > > > > > > > -- --all > >> > > > > > > > > > > > mkdir POCs/ > >> > > > > > > > > > > > git mv * POCs/ > >> > > > > > > > > > > > git commit -m "colleted the data to move" > >> > > > > > > > > > > > > >> > > > > > > > > > > > git clone <git destination_repository_url> cd <git > >> > > > > > > > > > > > destination_repository_directory> git remote add > >> > > > > > > > > > origin_repository_branch > >> > > > > > > > > > > > <git origin_repository directory> git pull > >> > > > > > > origin_repository_branch > >> > > > > > > > > > > master > >> > > > > > > > > > > > --allow-unrelated-histories git remote rm > >> > > > > > > origin_repository_branch > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance ! > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > -------------------------------------------------------------- > >> > > > > > > > > > > > Zhang Yifei > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > -- > >> > > > > > > > > > > Luciano Resende > >> > > > > > > > > > > http://twitter.com/lresende1975 > >> > > > > > > > > > > http://lresende.blogspot.com/ > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > -- > >> > > > > > > > > > >> > -------------------------------------------------------------- > >> > > > > > > > > Zhang Yifei > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
