+1 Em sex., 17 de abr. de 2020 às 11:15, Bruno Sette <brunosilvase...@gmail.com> escreveu:
> Standardizing with gRPC seems better to me. It is a good agnostic and safe > language solution. > > Em sex., 17 de abr. de 2020 às 14:56, Lucas Cardoso Silva < > cardosolucas61....@gmail.com> escreveu: > > > Standardize with gRPC seems like a better idea. It has all the security > > features we need and maintains the code's agnosticism. What do the rest > of > > you guys think? > > > > Lucas Cardoso > > > > Em sex., 3 de abr. de 2020 às 14:45, Daniel Lucredio < > > daniel.lucre...@ufscar.br> escreveu: > > > > > If we are going for a server to maintain the daemon, instead of a > simple > > > process just to keep the container alive, why not go for a gRPC > solution? > > > In this way, both the engine's endpoints and the toolbox commands would > > be > > > accessible in the same way. And it would also support other languages > for > > > the clients. What do you think? > > > > > > Daniel Lucrédio > > > > > > Em seg., 30 de mar. de 2020 às 14:38, Lucas Cardoso Silva < > > > cardosolucas61....@gmail.com> escreveu: > > > > > > > What do you think about using the Pyro4 library to maintain the > daemon? > > > The > > > > license is compatible and it deals with security issues. We can use > the > > > > docker API to provision containers and interact at a high level while > > > > remote method calls are made through Pyro lib. > > > > > > > > Pyro4 github: https://github.com/irmen/Pyro4 > > > > > > > > Best regards, > > > > Lucas Cardoso > > > > > > > > Em qui., 19 de mar. de 2020 às 16:24, Lucas Cardoso Silva < > > > > cardosolucas61....@gmail.com> escreveu: > > > > > > > > > Great! I will do some tests with the "daemon" approach and report > any > > > > > updates. Thanks! > > > > > > > > > > Em ter., 17 de mar. de 2020 às 17:40, Daniel Takabayashi < > > > > > daniel.takabaya...@gmail.com> escreveu: > > > > > > > > > >> No, I think everything is here. The "daemon" idea is the best way > to > > > > >> handle > > > > >> this problem, basically, we need to have a Marvin agent running > > inside > > > > the > > > > >> container responsible to receive and run the commands from > outside. > > > But > > > > I > > > > >> didn't get the part related to Jupyter. > > > > >> > > > > >> > > > > >> Em sex., 13 de mar. de 2020 às 15:15, Lucas Cardoso Silva < > > > > >> cardosolucas61....@gmail.com> escreveu: > > > > >> > > > > >> > I did some testing with the Docker API. The only problem is > that a > > > > >> > container needs a process running to stay active, if it does not > > > exist > > > > >> the > > > > >> > instance stop before receiving the command through the API. > There > > is > > > > the > > > > >> > possibility to maintain an infinite and inexpensive process > like: > > > tail > > > > >> -f / > > > > >> > dev / null (not a good idea). Another possibility would be > > refactor > > > > the > > > > >> > part of the marvin that would remain in the container to become > a > > > > >> daemon or > > > > >> > leave an instance of Jupyter running (the easy way). Is there > any > > > > other > > > > >> > possibility that I'm not seeing? > > > > >> > > > > > >> > Em sex., 13 de mar. de 2020 às 15:46, Daniel Takabayashi < > > > > >> > daniel.takabaya...@gmail.com> escreveu: > > > > >> > > > > > >> > > Take a look at the Docker API > > > > >> https://docs.docker.com/engine/api/v1.24/, > > > > >> > I > > > > >> > > believe this will simplify the solution. Cause if you add > > another > > > > >> layer > > > > >> > > (Flask application) you going to need to control/manage this > > layer > > > > as > > > > >> > well, > > > > >> > > creating a chicken and egg problem. > > > > >> > > > > > > >> > > I believe the Docker API is a simple solution that solves the > > > > problem > > > > >> > > (communication between toolbox and containers). > > > > >> > > > > > > >> > > Take a look here > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY > > > > >> > > > > > > >> > > > > > > >> > > Em sex., 13 de mar. de 2020 às 11:19, Lucas Cardoso Silva < > > > > >> > > cardosolucas61....@gmail.com> escreveu: > > > > >> > > > > > > >> > > > I thought about using a flask application, which can become > > very > > > > >> fast > > > > >> > and > > > > >> > > > scalable with a Python WSGI server for production like > > Gunicorn. > > > > We > > > > >> can > > > > >> > > > make the communication establish by HTTPS protocol and use a > > > > public > > > > >> and > > > > >> > > > private key generation system for each engine generated. > > > Another, > > > > >> > > althougt > > > > >> > > > complex, alternative would be use OAuth 2.0. > > > > >> > > > > > > > >> > > > Regards, > > > > >> > > > Lucas Cardoso > > > > >> > > > > > > > >> > > > Em sex., 13 de mar. de 2020 às 13:58, Daniel Takabayashi < > > > > >> > > > daniel.takabaya...@gmail.com> escreveu: > > > > >> > > > > > > > >> > > > > @Daniel The biggest challenge is to run commands in a > remote > > > > >> > container > > > > >> > > > > without having access to the whole O.S. Basically the idea > > is > > > to > > > > >> > create > > > > >> > > > an > > > > >> > > > > interface to make possible, in a secure way the > > communication > > > > >> > > > > between Toolbox and remote engines. > > > > >> > > > > > > > > >> > > > > Creating this we could start to run our engines in clouds > > and > > > > >> > services > > > > >> > > > like > > > > >> > > > > Google Run, Kubernetes, Lambda Functions and etc in the > same > > > way > > > > >> > Marvin > > > > >> > > > > runs locally. > > > > >> > > > > > > > > >> > > > > @Lucas Lets talk about the details of these APIs > > (interfaces, > > > > >> > > technology) > > > > >> > > > > > > > > >> > > > > Thanks, > > > > >> > > > > Taka > > > > >> > > > > > > > > >> > > > > Em sex., 13 de mar. de 2020 às 07:21, Daniel Lucredio < > > > > >> > > > > daniel.lucre...@ufscar.br> escreveu: > > > > >> > > > > > > > > >> > > > > > Hi Lucas and everyone, > > > > >> > > > > > > > > > >> > > > > > Couldn't the developer just run the CLI from inside the > > > > >> container, > > > > >> > > > > opening > > > > >> > > > > > a shell inside it? > > > > >> > > > > > > > > > >> > > > > > []s > > > > >> > > > > > > > > > >> > > > > > Daniel > > > > >> > > > > > > > > > >> > > > > > Em qui., 5 de mar. de 2020 às 13:31, Lucas Cardoso > Silva < > > > > >> > > > > > cardosolucas61....@gmail.com> escreveu: > > > > >> > > > > > > > > > >> > > > > > > I think we could define better how we will make the > > > > >> communication > > > > >> > > CLI > > > > >> > > > > > with > > > > >> > > > > > > the Docker development instance in the new > architecture. > > > The > > > > >> > > > container > > > > >> > > > > > > needs a running process to stay active. We could make > a > > > > >> > > communication > > > > >> > > > > via > > > > >> > > > > > > API endpoints to keep a web service running on the > > > container > > > > >> > > > receiving > > > > >> > > > > > > information from the CLI for the execution of tasks in > > > > >> Marvin. I > > > > >> > > > > created > > > > >> > > > > > a > > > > >> > > > > > > use-case scenario to facilitate understanding and > > further > > > > >> > > discussion. > > > > >> > > > > > > > > > > >> > > > > > > Scenario: A developer created an engine on marvin > > through > > > > the > > > > >> > CLI, > > > > >> > > he > > > > >> > > > > > > configures his engine in order to describe all the > > > > >> dependencies > > > > >> > of > > > > >> > > > the > > > > >> > > > > > > operating system that will be used during development. > > > After > > > > >> the > > > > >> > > > engine > > > > >> > > > > > > configuration process, the developer uses the CLI to > > > upload > > > > a > > > > >> > > > > development > > > > >> > > > > > > environment on a Docker container, that development > > > > >> environment > > > > >> > > will > > > > >> > > > > > > contain the refined Marvin toolbox. The CLI then > changes > > > its > > > > >> > > > interface > > > > >> > > > > in > > > > >> > > > > > > order to contain the standard commands for using > > engines. > > > He > > > > >> uses > > > > >> > > the > > > > >> > > > > CLI > > > > >> > > > > > > to communicate with the toolbox endpoints and inform > > that > > > he > > > > >> > wants > > > > >> > > to > > > > >> > > > > run > > > > >> > > > > > > an instance of the notebook. After building the model, > > the > > > > >> > > developer > > > > >> > > > > can, > > > > >> > > > > > > through the CLI, perform a dryrun and upload the > > > http-server > > > > >> for > > > > >> > > > > testing. > > > > >> > > > > > > All of these procedures will be done using the > > > communication > > > > >> > > through > > > > >> > > > > API > > > > >> > > > > > > endpoints. > > > > >> > > > > > > > > > > >> > > > > > > Does that sound like a good strategy? Do you have any > > > > >> > suggestions, > > > > >> > > or > > > > >> > > > > > > something that was already foreseen in the original > > > project? > > > > >> > > > > > > > > > > >> > > > > > > Best regards, > > > > >> > > > > > > Lucas Cardoso > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > -- > Att., > > *Bruno Silva Sette* >
