I see... what about this new version? https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
I changed the toolbox ...from 1 component to 3... docker client, cli and repl. Both (cli and repl) now are using the docker client to control the engines. Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel < lucasb...@apache.org> escreveu: > For sure, both suggestions are to make sure we're not killing the CLI and > that we have a component (the SDK client) that is shared by CLI and REPL. > > Thanks for clarifying where the system calls will live. > > - Lucas > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi < > daniel.takabaya...@gmail.com> wrote: > > > Hi Lucas, could you please justified your two suggestions? > > > > The general ideia is to use the Docker DSK to execute local commands, to > > consume outputs and also to execute Marvin "specific language" commands. > > Today almost all CLIs commands in Marvin is a mixed of systems commands > and > > library method calls. The plan is to put all "specific language calls" > > inside the marvin-common-libs and group the systems call on the toolbox > > (using the docker SDK in this task). > > > > > > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel < > > lucasb...@apache.org> escreveu: > > > > > Great, thanks. > > > > > > So here some suggestions: > > > 1. We could create one new layer between Toolbox and the Engines, this > > > would be Docker client. > > > 2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL > > > > > > It's not clear to me if the commands listed on General Commands will > need > > > to be written for every language, from the image I believe not, but how > > it > > > would work, since starting a notebook in Python is different than > > starting > > > in Scala. > > > > > > - Lucas > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi < > > > daniel.takabaya...@gmail.com> wrote: > > > > > > > > > > > > > > > > > https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY > > > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello < > > > > rafa.reis.nove...@gmail.com> escreveu: > > > > > > > > > Hi Taka, > > > > > > > > > > I can't see the image! Maybe the Apache email list don't support > > > > > attachments. > > > > > > > > > > Best! > > > > > Rafael J. R. Novello > > > > > > > > > > Skype: rafael.novello > > > > > Blog: http://rafanovello.blogspot.com.br/ > > > > > > > > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi < > > > > > daniel.takabaya...@gmail.com> escreveu: > > > > > > > > > > > Hi all, > > > > > > > > > > > > To try to give a little more details about this "new concept" > that > > we > > > > > want > > > > > > to bring to the marvin toolbox, I did this simple architecture > > draw. > > > > > > > > > > > > [image: marvin-architecture-views (5).png] > > > > > > > > > > > > The general idea here is try to transform the toolbox something > > > > > > disconnected with the "language", something more agnostic. Also > in > > > this > > > > > new > > > > > > architecture we could use remote resource to process engines and > > make > > > > > easy > > > > > > the support for new languages. > > > > > > > > > > > > This "new toolbox" will be the only thing that a Marvin user must > > to > > > > > > install and also we could start to support multiples O.S (once > the > > > REPL > > > > > is > > > > > > a dummy application that only interprets and by pass commands). > > > > > > > > > > > > Regards, > > > > > > Taka > > > > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello < > > > > > > rafa.reis.nove...@gmail.com> escreveu: > > > > > > > > > > > >> Alan, > > > > > >> > > > > > >> Yes! We are using the Docker SDK and it's possible to use the > API > > to > > > > > >> automate, but it's a little bit harder than automate CLI calls. > > > > > >> > > > > > >> Atenciosamente, > > > > > >> Rafael J. R. Novello > > > > > >> > > > > > >> Skype: rafael.novello > > > > > >> Blog: http://rafanovello.blogspot.com.br/ > > > > > >> > > > > > >> > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva < > ju...@apache.org> > > > > > >> escreveu: > > > > > >> > > > > > >> > Hi, > > > > > >> > > > > > > >> > One question here, I understand that we start to use with this > > PoC > > > > the > > > > > >> > Docker SDK API, right? > > > > > >> > > > > > > >> > Why not use the API to expose some endpoints to permit this > kind > > > of > > > > > >> > automation by devops? > > > > > >> > > > > > > >> > I think it is possible and it solves the CLI problem, right? > > > > > >> > > > > > > >> > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello < > > > > > >> > rafa.reis.nove...@gmail.com> > > > > > >> > wrote: > > > > > >> > > > > > > >> > > Lucas, > > > > > >> > > > > > > > >> > > Sorry, I didn't understood your question bellow. > > > > > >> > > > > > > > >> > > "Would it make sense to use the same solution that we will > use > > > for > > > > > >> > having a > > > > > >> > > single-language REPL to have a single-language CLI?" > > > > > >> > > > > > > > >> > > For DevOps purposes, maybe this new toolbox concept is not > > > ideal. > > > > I > > > > > >> think > > > > > >> > > we can keep the CLI inside the docker container but it will > > not > > > > easy > > > > > >> to > > > > > >> > > automate > > > > > >> > > jobs by this way. > > > > > >> > > > > > > > >> > > How to deal with this issue? Voting? > > > > > >> > > Atenciosamente, > > > > > >> > > Rafael J. R. Novello > > > > > >> > > > > > > > >> > > Skype: rafael.novello > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/ > > > > > >> > > > > > > > >> > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel < > > > > > >> > > lucasb...@apache.org> escreveu: > > > > > >> > > > > > > > >> > > > Got it! Thanks for clarifying. > > > > > >> > > > > > > > > >> > > > Would it make sense to use the same solution that we will > > use > > > > for > > > > > >> > having > > > > > >> > > a > > > > > >> > > > single-language REPL to have a single-language CLI? My > only > > > > > concern > > > > > >> > with > > > > > >> > > > killing the CLI is that you remove an essential feature > for > > > > > DevOps. > > > > > >> > > > > > > > > >> > > > - Lucas > > > > > >> > > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello < > > > > > >> > > > rafa.reis.nove...@gmail.com> > > > > > >> > > > wrote: > > > > > >> > > > > > > > > >> > > > > Lucas, > > > > > >> > > > > > > > > > >> > > > > The idea is that REPL will substitute the actual CLI. > It's > > > > > because > > > > > >> > with > > > > > >> > > > the > > > > > >> > > > > actual concept (using language specific CLI) we will > need > > to > > > > > >> > > re-implement > > > > > >> > > > > the same features many times and probably each CLI will > > > have a > > > > > >> > > different > > > > > >> > > > > behavior because some language specific restrictions > > and/or > > > > > >> > > limitations. > > > > > >> > > > > > > > > > >> > > > > With this new concept, all users will have the same > > > experience > > > > > >> > > > interacting > > > > > >> > > > > with Marvin REPL and they will use the bests tools to > > > develop > > > > > >> their > > > > > >> > > > engines > > > > > >> > > > > (Jupyter Notebook and/or Lab with different languages > > > support > > > > or > > > > > >> even > > > > > >> > > > > Apache Zeppelin). All the interact will occur through > REPL > > > and > > > > > >> docker > > > > > >> > > > > protocol. > > > > > >> > > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the same but we > > can > > > > use > > > > > >> > docker > > > > > >> > > > to > > > > > >> > > > > do the same job. > > > > > >> > > > > > > > > > >> > > > > Best! > > > > > >> > > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto > Miguel < > > > > > >> > > > > lucasb...@apache.org> escreveu: > > > > > >> > > > > > > > > > >> > > > > > Thanks for the clarifications Rafael, so from what I > > > > > understood, > > > > > >> > > > Marvin's > > > > > >> > > > > > developers would not use the REPL to explore data and > > test > > > > > >> models, > > > > > >> > > i.e. > > > > > >> > > > > > develop the engine. Is the idea to build something > more > > > like > > > > > an > > > > > >> > > > > interactive > > > > > >> > > > > > CLI? I think an interactive CLI would be useful for > the > > > > > >> developer > > > > > >> > > > > > experience, however, it's important to keep the > > unattended > > > > > >> > (current) > > > > > >> > > > CLI > > > > > >> > > > > in > > > > > >> > > > > > place for scenarios where users want to automate > Marvin > > > > tasks. > > > > > >> > > > > > > > > > > >> > > > > > Alan, I believe the idea is still the same as you > > started, > > > > but > > > > > >> > using > > > > > >> > > > > > docker SDK now. > > > > > >> > > > > > > > > > > >> > > > > > - Lucas > > > > > >> > > > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello < > > > > > >> > > > > > rafa.reis.nove...@gmail.com> > > > > > >> > > > > > wrote: > > > > > >> > > > > > > > > > > >> > > > > > > Hi Lucas! > > > > > >> > > > > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache! > > > > > >> > > > > > > > > > > > >> > > > > > > Let me help with some comments: > > > > > >> > > > > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and the > > > > easiest > > > > > >> one > > > > > >> > was > > > > > >> > > > > > Python. > > > > > >> > > > > > > We have found a small project [1] that have all > > features > > > > we > > > > > >> > desired > > > > > >> > > > for > > > > > >> > > > > > > REPL: > > > > > >> > > > > > > - Autocomplete > > > > > >> > > > > > > - Python commands disabled (the user have only the > > > > commands > > > > > we > > > > > >> > > > > provide). > > > > > >> > > > > > > The other languages REPL options don't have this > > > feature. > > > > > >> > > > > > > - Easy to show output > > > > > >> > > > > > > - Etc > > > > > >> > > > > > > So, I think the language chosen will not be > important > > > here > > > > > >> > because > > > > > >> > > > the > > > > > >> > > > > > user > > > > > >> > > > > > > will only interact with the commands that we create. > > > > > >> > > > > > > > > > > > >> > > > > > > 2 - The "engine-generate" command will download a > > docker > > > > > image > > > > > >> > that > > > > > >> > > > we > > > > > >> > > > > > > create for that language and start a container with > > the > > > > > basic > > > > > >> > > project > > > > > >> > > > > > > structure for that language. > > > > > >> > > > > > > > > > > > >> > > > > > > 3 - The REPL client will use the "docker protocol" > to > > > run > > > > > >> > command, > > > > > >> > > > > > > start/stop services and etc inside the container and > > it > > > > will > > > > > >> > > receive > > > > > >> > > > > the > > > > > >> > > > > > > log stream to show. No, the REPL will no pass code > > > > snippets > > > > > >> for > > > > > >> > > > docker > > > > > >> > > > > > > container (I think it will not be necessary) > > > > > >> > > > > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first item. > > > > > >> > > > > > > > > > > > >> > > > > > > [1] - https://github.com/italorossi/ishell > > > > > >> > > > > > > > > > > > >> > > > > > > Let me know if there is any other question! > > > > > >> > > > > > > Best! > > > > > >> > > > > > > Rafael J. R. Novello > > > > > >> > > > > > > > > > > > >> > > > > > > Skype: rafael.novello > > > > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/ > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto > > > Miguel < > > > > > >> > > > > > > lucasb...@apache.org> escreveu: > > > > > >> > > > > > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache > > > > > >> > > > > > > > > > > > > >> > > > > > > > I have a few questions about the previous > > explanation: > > > > > >> > > > > > > > 1) The REPL itself, it would be an application in > > > which > > > > > >> > > language? > > > > > >> > > > > > > Remember > > > > > >> > > > > > > > that the main idea is to allow the user to program > > on > > > > his > > > > > >> > > preferred > > > > > >> > > > > > > > language in the REPL. > > > > > >> > > > > > > > 2) Should the engine-generate command also > > generate a > > > > > >> docker > > > > > >> > > image > > > > > >> > > > > > with > > > > > >> > > > > > > > the user's application? > > > > > >> > > > > > > > 3) What type of communication would happen > between > > > the > > > > > REPL > > > > > >> > and > > > > > >> > > > the > > > > > >> > > > > > > engine > > > > > >> > > > > > > > via Docker SDK? Would the REPL pass snippets of > code > > > to > > > > be > > > > > >> > > executed > > > > > >> > > > > by > > > > > >> > > > > > > the > > > > > >> > > > > > > > docker container? > > > > > >> > > > > > > > 4) Have you considered code completion in the > REPL? > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei < > > > > > >> > > > yifei.z.l...@gmail.com> > > > > > >> > > > > > > > wrote: > > > > > >> > > > > > > > > > > > > >> > > > > > > > > Ok guys, > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > The basic idea is to provide only one Toolbox > for > > > > > multiple > > > > > >> > > > > languages. > > > > > >> > > > > > > > > We are looking for possibility to build a single > > > > Marvin > > > > > >> Repl, > > > > > >> > > > > > > > > instead of severals toolboxes with differentes > > > > > interfaces > > > > > >> or > > > > > >> > > > > > commands. > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > In this case, the engine-generate command will > > > > download > > > > > >> and > > > > > >> > > > start a > > > > > >> > > > > > > > Docker > > > > > >> > > > > > > > > container with basic engine structure > > corresponding > > > > the > > > > > >> > choosed > > > > > >> > > > > > > language. > > > > > >> > > > > > > > > this means we don't need to build Toolboxes of > > > > > differents > > > > > >> > > > > languages, > > > > > >> > > > > > we > > > > > >> > > > > > > > > just > > > > > >> > > > > > > > > build the engine template of all languages that > we > > > > want > > > > > to > > > > > >> > > > support > > > > > >> > > > > > and > > > > > >> > > > > > > > > provide it > > > > > >> > > > > > > > > as Docker containers > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > We have started researches around our basic > > > > requirements > > > > > >> > like: > > > > > >> > > > > > > > > - Repl interface > > > > > >> > > > > > > > > - System communication > > > > > >> > > > > > > > > - Connection security > > > > > >> > > > > > > > > - Tool popularity > > > > > >> > > > > > > > > - Update complexity > > > > > >> > > > > > > > > - Languages support > > > > > >> > > > > > > > > - ...... > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > And we did some POC with code here: > > > > > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > There is POC testing gRPC using Scala and > Python, > > > > > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS, > > > > > >> > > > > > > > > Repl interface and Docker SDK with Python. > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > At this moment we prefer the Repl interface + > > Docker > > > > SDK > > > > > >> way, > > > > > >> > > > > because > > > > > >> > > > > > > > good > > > > > >> > > > > > > > > part of the requirements > > > > > >> > > > > > > > > will be guaranteed by Docker. > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > With this informations, what do you think? > Should > > we > > > > > >> submit > > > > > >> > all > > > > > >> > > > > this > > > > > >> > > > > > > POCs > > > > > >> > > > > > > > > to Apache Repo? > > > > > >> > > > > > > > > Please feel free to opine. > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Thats all, thanks!!! > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel > > > > Takabayashi < > > > > > >> > > > > > > > > daniel.takabaya...@gmail.com> escreveu: > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > Zhang, > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > I think the best approach is give us a better > > > > > >> explanation > > > > > >> > > about > > > > > >> > > > > > this > > > > > >> > > > > > > > new > > > > > >> > > > > > > > > > feature and how this can help us to archive > the > > > > > desired > > > > > >> > > feature > > > > > >> > > > > > > > (support > > > > > >> > > > > > > > > > multiple languages). Than if the majority > agree > > > > than I > > > > > >> can > > > > > >> > do > > > > > >> > > > the > > > > > >> > > > > > > > merge, > > > > > >> > > > > > > > > > just like I did before, but in another branch. > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > Let me know if makes sense, ok? > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > Taka > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano > > > Resende > > > > < > > > > > >> > > > > > > > > luckbr1...@gmail.com> > > > > > >> > > > > > > > > > escreveu: > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a refactoring of > > the > > > > > >> existing > > > > > >> > > > repo? > > > > > >> > > > > Or > > > > > >> > > > > > > is > > > > > >> > > > > > > > > > > it a new rewrite of the repo? > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > Just asking as it might make sense to make > it > > a > > > > > branch > > > > > >> > then > > > > > >> > > > > > > actually > > > > > >> > > > > > > > a > > > > > >> > > > > > > > > > > thing in parallel, as this will have an > effect > > > for > > > > > >> > > releases, > > > > > >> > > > > etc. > > > > > >> > > > > > > but > > > > > >> > > > > > > > > > > you guys know more here than I do. > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > Also, it's probably good to have a write up > of > > > the > > > > > >> main > > > > > >> > > > > direction > > > > > >> > > > > > > of > > > > > >> > > > > > > > > > > the design that can help people get familiar > > > with > > > > > the > > > > > >> new > > > > > >> > > > > > approach. > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang > Yifei < > > > > > >> > > > > > > yifei.z.l...@gmail.com > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > wrote: > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > Hey guys, > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > We have reorganized the Poc repo, and want > > to > > > > > merge > > > > > >> it > > > > > >> > to > > > > > >> > > > our > > > > > >> > > > > > > > Apache > > > > > >> > > > > > > > > > > repo. > > > > > >> > > > > > > > > > > > Just making sure here before we do the > > merge, > > > > > >> because > > > > > >> > im > > > > > >> > > > not > > > > > >> > > > > > sure > > > > > >> > > > > > > > if > > > > > >> > > > > > > > > > > doing > > > > > >> > > > > > > > > > > > this will perserve the Git commit history. > > > > > >> > > > > > > > > > > > What we are planning to do is: > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > git filter-branch --subdirectory-filter > <git > > > > > >> > > > > origin_repository > > > > > >> > > > > > > > > > directory> > > > > > >> > > > > > > > > > > > -- --all > > > > > >> > > > > > > > > > > > mkdir POCs/ > > > > > >> > > > > > > > > > > > git mv * POCs/ > > > > > >> > > > > > > > > > > > git commit -m "colleted the data to move" > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > git clone <git destination_repository_url> > > cd > > > > <git > > > > > >> > > > > > > > > > > > destination_repository_directory> git > remote > > > add > > > > > >> > > > > > > > > > origin_repository_branch > > > > > >> > > > > > > > > > > > <git origin_repository directory> git pull > > > > > >> > > > > > > origin_repository_branch > > > > > >> > > > > > > > > > > master > > > > > >> > > > > > > > > > > > --allow-unrelated-histories git remote rm > > > > > >> > > > > > > origin_repository_branch > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance ! > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > -------------------------------------------------------------- > > > > > >> > > > > > > > > > > > Zhang Yifei > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > -- > > > > > >> > > > > > > > > > > Luciano Resende > > > > > >> > > > > > > > > > > http://twitter.com/lresende1975 > > > > > >> > > > > > > > > > > http://lresende.blogspot.com/ > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > -- > > > > > >> > > > > > > > > > > > > > >> > -------------------------------------------------------------- > > > > > >> > > > > > > > > Zhang Yifei > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >
