Re: AutoML Engine

[Lucas Cardoso Silva]

Thank you for your interest!
For now we focus only on ease of use for the user, so we have implemented a
setup and configuration script for 3 widely used AutoML tools:
auto-sklearn, H2O and TPOT.
Auto-sklearn is fully functional and we have an example in public engines.
The others presented some problems regarding the model serialization, but
we are already looking for a solution. For future works, we thought it
would be interesting to include autokeras (for deep learning) and to work
with other parts of the pipeline such as data preprocessing, as well as
improving the support and usability of these tools for Marvin users.
I think the mailing list server doesn't allow attachments, so I uploaded
the images to imgur:

https://imgur.com/a/ltEi1XB

Thanks,
Lucas

Em ter, 10 de set de 2019 às 19:29, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> Hi Lucas,
>
> Awesome that you guys are trying to bring autoML inside Marvin.
> Could you explain the implementation and tools did you used to get the
> automl solution working inside Marvin?
>
> Thanks,
> Taka
>
> Em ter, 10 de set de 2019 às 13:28, Wei Chen 
> escreveu:
>
> > That's will be nice.
> > I am not able to see your attached images btw.
> >
> > Best Regards
> > Wei
> >
> > On Wed, Sep 11, 2019 at 2:29 AM Lucas Cardoso Silva <
> > cardosolucas61@gmail.com> wrote:
> >
> > > Hi guys.
> > >
> > > We at B2W-UFSCar Lab have been studying AutoML and have developed a
> > > prototype AutoML engine for Marvin, trying to understand it better.
> > >
> > > What do you think about integrating this engine into Marvin? There is
> an
> > > open issue about this:
> > >
> >
> https://issues.apache.org/jira/projects/MARVIN/issues/MARVIN-46?filter=allopenissues
> > >
> > > Here are some print screens to show you the basic idea:
> > >
> > > - marvin1.png: CLI command to create a new AutoML project
> > > - marvin2.png: Dialog to select the desired library to install
> > > - marvin3.png: Notebook running a sample AutoML trainer
> > >
> > > If you want to take a look in the current prototype, it is on a local
> > fork
> > > of GitHub "dev" branch:
> https://github.com/cardosolucas/incubator-marvin
> > >
> > > Iris example engine:
> > >
> >
> https://github.com/cardosolucas/incubator-marvin/tree/develop/public-engines/iris-automl-engine
> > >
> > > Thanks,
> > > Lucas.
> > >
> > >
> > >
> >
>

AutoML Engine

[Lucas Cardoso Silva]

Hi guys.

We at B2W-UFSCar Lab have been studying AutoML and have developed a
prototype AutoML engine for Marvin, trying to understand it better.

What do you think about integrating this engine into Marvin? There is an
open issue about this:
https://issues.apache.org/jira/projects/MARVIN/issues/MARVIN-46?filter=allopenissues

Here are some print screens to show you the basic idea:

- marvin1.png: CLI command to create a new AutoML project
- marvin2.png: Dialog to select the desired library to install
- marvin3.png: Notebook running a sample AutoML trainer

If you want to take a look in the current prototype, it is on a local fork
of GitHub "dev" branch: https://github.com/cardosolucas/incubator-marvin

Iris example engine:
https://github.com/cardosolucas/incubator-marvin/tree/develop/public-engines/iris-automl-engine

Thanks,
Lucas.

Marvin Architecture Validation

[Lucas Cardoso Silva]

Hi guys,
I am considering using Marvin as an artifact for my master's dissertation.
The idea is to conduct a formal architectural design and validation process
in an Open Source environment. While conducting the process we should
produce sufficient documentation to guide developers through the
refactoring of architecture (architectural views, scenarios, requirements
...) and validate the architecture by adapting to the scenarios listed by
contributors.

To make this happen, I ask your collaboration to validate Marvin's current
architectural views as we will use them as the starting point of the
process. Marvin UML diagrams have also been requested at:
https://issues.apache.org/jira/projects/MARVIN/issues/MARVIN-67.

Figure 1 is a class diagram of the base engine:
https://docs.google.com/drawings/d/183GbZHci4vIAqZDLP0jcUL231wdQ5cNAO2rhyTKCUCY/edit?usp=sharing
Figure 2 is the component diagram:
https://docs.google.com/drawings/d/1qSd1eAksJ_mqn4Tu7ZCnGnEWuJaL7hfPQGyCPhW2fdk/edit?usp=sharing
Figure 3 is the deploy diagram:
https://docs.google.com/drawings/d/1CdtjHa-QR9SwZbBqkalUHN9cL2sHiP75NAFN7bRt8-0/edit?usp=sharing
Figure 4 is a generic architectural view inspired by the new architecture
figure:
https://docs.google.com/drawings/d/1UOR8Bk0fpLAOnotdeAn4Ww1GBbRxhdtM5mfoTJKQNVo/edit?usp=sharing

Thanks in advance!

Re: report

[Lucas Cardoso Silva]

Looks fine to me.

Regards,
Lucas Cardoso

Em qua, 4 de mar de 2020 12:20, Zhang Yifei 
escreveu:

> Hello guys,
>
> We just updated our report at
> https://cwiki.apache.org/confluence/display/INCUBATOR/March2020
> Please feel free to suggest any changes.
>
> Many thanks.
>
> --
> --
> Zhang Yifei
>

Communication between CLI and Docker container in the new architecture.

[Lucas Cardoso Silva]

I think we could define better how we will make the communication CLI with
the Docker development instance in the new architecture. The container
needs a running process to stay active. We could make a communication via
API endpoints to keep a web service running on the container receiving
information from the CLI for the execution of tasks in Marvin. I created a
use-case scenario to facilitate understanding and further discussion.

Scenario: A developer created an engine on marvin through the CLI, he
configures his engine in order to describe all the dependencies of the
operating system that will be used during development. After the engine
configuration process, the developer uses the CLI to upload a development
environment on a Docker container, that development environment will
contain the refined Marvin toolbox. The CLI then changes its interface in
order to contain the standard commands for using engines. He uses the CLI
to communicate with the toolbox endpoints and inform that he wants to run
an instance of the notebook. After building the model, the developer can,
through the CLI, perform a dryrun and upload the http-server for testing.
All of these procedures will be done using the communication through API
endpoints.

Does that sound like a good strategy? Do you have any suggestions, or
something that was already foreseen in the original project?

Best regards,
Lucas Cardoso

Re: Communication between CLI and Docker container in the new architecture.

[Lucas Cardoso Silva]

Standardize with gRPC seems like a better idea. It has all the security
features we need and maintains the code's agnosticism. What do the rest of
you guys think?

Lucas Cardoso

Em sex., 3 de abr. de 2020 às 14:45, Daniel Lucredio <
daniel.lucre...@ufscar.br> escreveu:

> If we are going for a server to maintain the daemon, instead of a simple
> process just to keep the container alive, why not go for a gRPC solution?
> In this way, both the engine's endpoints and the toolbox commands would be
> accessible in the same way. And it would also support other languages for
> the clients. What do you think?
>
> Daniel Lucrédio
>
> Em seg., 30 de mar. de 2020 às 14:38, Lucas Cardoso Silva <
> cardosolucas61@gmail.com> escreveu:
>
> > What do you think about using the Pyro4 library to maintain the daemon?
> The
> > license is compatible and it deals with security issues. We can use the
> > docker API to provision containers and interact at a high level while
> > remote method calls are made through Pyro lib.
> >
> > Pyro4 github: https://github.com/irmen/Pyro4
> >
> > Best regards,
> > Lucas Cardoso
> >
> > Em qui., 19 de mar. de 2020 às 16:24, Lucas Cardoso Silva <
> > cardosolucas61@gmail.com> escreveu:
> >
> > > Great! I will do some tests with the "daemon" approach and report any
> > > updates. Thanks!
> > >
> > > Em ter., 17 de mar. de 2020 às 17:40, Daniel Takabayashi <
> > > daniel.takabaya...@gmail.com> escreveu:
> > >
> > >> No, I think everything is here. The "daemon" idea is the best way to
> > >> handle
> > >> this problem, basically, we need to have a Marvin agent running inside
> > the
> > >> container responsible to receive and run the commands from outside.
> But
> > I
> > >> didn't get the part related to Jupyter.
> > >>
> > >>
> > >> Em sex., 13 de mar. de 2020 às 15:15, Lucas Cardoso Silva <
> > >> cardosolucas61@gmail.com> escreveu:
> > >>
> > >> > I did some testing with the Docker API. The only problem is that a
> > >> > container needs a process running to stay active, if it does not
> exist
> > >> the
> > >> > instance stop before receiving the command through the API. There is
> > the
> > >> > possibility to maintain an infinite and inexpensive process like:
> tail
> > >> -f /
> > >> > dev / null (not a good idea). Another possibility would be refactor
> > the
> > >> > part of the marvin that would remain in the container to become a
> > >> daemon or
> > >> > leave an instance of Jupyter running (the easy way). Is there any
> > other
> > >> > possibility that I'm not seeing?
> > >> >
> > >> > Em sex., 13 de mar. de 2020 às 15:46, Daniel Takabayashi <
> > >> > daniel.takabaya...@gmail.com> escreveu:
> > >> >
> > >> > > Take a look at the Docker API
> > >> https://docs.docker.com/engine/api/v1.24/,
> > >> > I
> > >> > > believe this will simplify the solution. Cause if you add another
> > >> layer
> > >> > > (Flask application) you going to need to control/manage this layer
> > as
> > >> > well,
> > >> > > creating a chicken and egg problem.
> > >> > >
> > >> > > I believe the Docker API is a simple solution that solves the
> > problem
> > >> > > (communication between toolbox and containers).
> > >> > >
> > >> > > Take a look here
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602=461=1423=1=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > >> > >
> > >> > >
> > >> > > Em sex., 13 de mar. de 2020 às 11:19, Lucas Cardoso Silva <
> > >> > > cardosolucas61@gmail.com> escreveu:
> > >> > >
> > >> > > > I thought about using a flask application, which can become very
> > >> fast
> > >> > and
> > >> > > > scalable with a Python WSGI server for production like Gunicorn.
> > We
> > >> can
> > >> > > > make the communication establish by HTTPS protocol and use a
> > public
> > >> and
> > >&

Re: Communication between CLI and Docker container in the new architecture.

[Lucas Cardoso Silva]

I thought about using a flask application, which can become very fast and
scalable with a Python WSGI server for production like Gunicorn. We can
make the communication establish by HTTPS protocol and use a public and
private key generation system for each engine generated. Another, althougt
complex, alternative would be use OAuth 2.0.

Regards,
Lucas Cardoso

Em sex., 13 de mar. de 2020 às 13:58, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> @Daniel The biggest challenge is to run commands in a remote container
> without having access to the whole O.S. Basically the idea is to create an
> interface to make possible, in a secure way the communication
> between Toolbox and remote engines.
>
> Creating this we could start to run our engines in clouds and services like
> Google Run, Kubernetes, Lambda Functions and etc in the same way Marvin
> runs locally.
>
> @Lucas Lets talk about the details of these APIs (interfaces, technology)
>
> Thanks,
> Taka
>
> Em sex., 13 de mar. de 2020 às 07:21, Daniel Lucredio <
> daniel.lucre...@ufscar.br> escreveu:
>
> > Hi Lucas and everyone,
> >
> > Couldn't the developer just run the CLI from inside the container,
> opening
> > a shell inside it?
> >
> > []s
> >
> > Daniel
> >
> > Em qui., 5 de mar. de 2020 às 13:31, Lucas Cardoso Silva <
> > cardosolucas61@gmail.com> escreveu:
> >
> > > I think we could define better how we will make the communication CLI
> > with
> > > the Docker development instance in the new architecture. The container
> > > needs a running process to stay active. We could make a communication
> via
> > > API endpoints to keep a web service running on the container receiving
> > > information from the CLI for the execution of tasks in Marvin. I
> created
> > a
> > > use-case scenario to facilitate understanding and further discussion.
> > >
> > > Scenario: A developer created an engine on marvin through the CLI, he
> > > configures his engine in order to describe all the dependencies of the
> > > operating system that will be used during development. After the engine
> > > configuration process, the developer uses the CLI to upload a
> development
> > > environment on a Docker container, that development environment will
> > > contain the refined Marvin toolbox. The CLI then changes its interface
> in
> > > order to contain the standard commands for using engines. He uses the
> CLI
> > > to communicate with the toolbox endpoints and inform that he wants to
> run
> > > an instance of the notebook. After building the model, the developer
> can,
> > > through the CLI, perform a dryrun and upload the http-server for
> testing.
> > > All of these procedures will be done using the communication through
> API
> > > endpoints.
> > >
> > > Does that sound like a good strategy? Do you have any suggestions, or
> > > something that was already foreseen in the original project?
> > >
> > > Best regards,
> > > Lucas Cardoso
> > >
> >
>

Re: Communication between CLI and Docker container in the new architecture.

[Lucas Cardoso Silva]

I did some testing with the Docker API. The only problem is that a
container needs a process running to stay active, if it does not exist the
instance stop before receiving the command through the API. There is the
possibility to maintain an infinite and inexpensive process like: tail -f /
dev / null (not a good idea). Another possibility would be refactor the
part of the marvin that would remain in the container to become a daemon or
leave an instance of Jupyter running (the easy way). Is there any other
possibility that I'm not seeing?

Em sex., 13 de mar. de 2020 às 15:46, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> Take a look at the Docker API https://docs.docker.com/engine/api/v1.24/, I
> believe this will simplify the solution. Cause if you add another layer
> (Flask application) you going to need to control/manage this layer as well,
> creating a chicken and egg problem.
>
> I believe the Docker API is a simple solution that solves the problem
> (communication between toolbox and containers).
>
> Take a look here
>
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602=461=1423=1=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
>
>
> Em sex., 13 de mar. de 2020 às 11:19, Lucas Cardoso Silva <
> cardosolucas61@gmail.com> escreveu:
>
> > I thought about using a flask application, which can become very fast and
> > scalable with a Python WSGI server for production like Gunicorn. We can
> > make the communication establish by HTTPS protocol and use a public and
> > private key generation system for each engine generated. Another,
> althougt
> > complex, alternative would be use OAuth 2.0.
> >
> > Regards,
> > Lucas Cardoso
> >
> > Em sex., 13 de mar. de 2020 às 13:58, Daniel Takabayashi <
> > daniel.takabaya...@gmail.com> escreveu:
> >
> > > @Daniel The biggest challenge is to run commands in a remote container
> > > without having access to the whole O.S. Basically the idea is to create
> > an
> > > interface to make possible, in a secure way the communication
> > > between Toolbox and remote engines.
> > >
> > > Creating this we could start to run our engines in clouds and services
> > like
> > > Google Run, Kubernetes, Lambda Functions and etc in the same way Marvin
> > > runs locally.
> > >
> > > @Lucas Lets talk about the details of these APIs (interfaces,
> technology)
> > >
> > > Thanks,
> > > Taka
> > >
> > > Em sex., 13 de mar. de 2020 às 07:21, Daniel Lucredio <
> > > daniel.lucre...@ufscar.br> escreveu:
> > >
> > > > Hi Lucas and everyone,
> > > >
> > > > Couldn't the developer just run the CLI from inside the container,
> > > opening
> > > > a shell inside it?
> > > >
> > > > []s
> > > >
> > > > Daniel
> > > >
> > > > Em qui., 5 de mar. de 2020 às 13:31, Lucas Cardoso Silva <
> > > > cardosolucas61@gmail.com> escreveu:
> > > >
> > > > > I think we could define better how we will make the communication
> CLI
> > > > with
> > > > > the Docker development instance in the new architecture. The
> > container
> > > > > needs a running process to stay active. We could make a
> communication
> > > via
> > > > > API endpoints to keep a web service running on the container
> > receiving
> > > > > information from the CLI for the execution of tasks in Marvin. I
> > > created
> > > > a
> > > > > use-case scenario to facilitate understanding and further
> discussion.
> > > > >
> > > > > Scenario: A developer created an engine on marvin through the CLI,
> he
> > > > > configures his engine in order to describe all the dependencies of
> > the
> > > > > operating system that will be used during development. After the
> > engine
> > > > > configuration process, the developer uses the CLI to upload a
> > > development
> > > > > environment on a Docker container, that development environment
> will
> > > > > contain the refined Marvin toolbox. The CLI then changes its
> > interface
> > > in
> > > > > order to contain the standard commands for using engines. He uses
> the
> > > CLI
> > > > > to communicate with the toolbox endpoints and inform that he wants
> to
> > > run
> > > > > an instance of the notebook. After building the model, the
> developer
> > > can,
> > > > > through the CLI, perform a dryrun and upload the http-server for
> > > testing.
> > > > > All of these procedures will be done using the communication
> through
> > > API
> > > > > endpoints.
> > > > >
> > > > > Does that sound like a good strategy? Do you have any suggestions,
> or
> > > > > something that was already foreseen in the original project?
> > > > >
> > > > > Best regards,
> > > > > Lucas Cardoso
> > > > >
> > > >
> > >
> >
>

Re: Communication between CLI and Docker container in the new architecture.

[Lucas Cardoso Silva]

What do you think about using the Pyro4 library to maintain the daemon? The
license is compatible and it deals with security issues. We can use the
docker API to provision containers and interact at a high level while
remote method calls are made through Pyro lib.

Pyro4 github: https://github.com/irmen/Pyro4

Best regards,
Lucas Cardoso

Em qui., 19 de mar. de 2020 às 16:24, Lucas Cardoso Silva <
cardosolucas61@gmail.com> escreveu:

> Great! I will do some tests with the "daemon" approach and report any
> updates. Thanks!
>
> Em ter., 17 de mar. de 2020 às 17:40, Daniel Takabayashi <
> daniel.takabaya...@gmail.com> escreveu:
>
>> No, I think everything is here. The "daemon" idea is the best way to
>> handle
>> this problem, basically, we need to have a Marvin agent running inside the
>> container responsible to receive and run the commands from outside. But I
>> didn't get the part related to Jupyter.
>>
>>
>> Em sex., 13 de mar. de 2020 às 15:15, Lucas Cardoso Silva <
>> cardosolucas61@gmail.com> escreveu:
>>
>> > I did some testing with the Docker API. The only problem is that a
>> > container needs a process running to stay active, if it does not exist
>> the
>> > instance stop before receiving the command through the API. There is the
>> > possibility to maintain an infinite and inexpensive process like: tail
>> -f /
>> > dev / null (not a good idea). Another possibility would be refactor the
>> > part of the marvin that would remain in the container to become a
>> daemon or
>> > leave an instance of Jupyter running (the easy way). Is there any other
>> > possibility that I'm not seeing?
>> >
>> > Em sex., 13 de mar. de 2020 às 15:46, Daniel Takabayashi <
>> > daniel.takabaya...@gmail.com> escreveu:
>> >
>> > > Take a look at the Docker API
>> https://docs.docker.com/engine/api/v1.24/,
>> > I
>> > > believe this will simplify the solution. Cause if you add another
>> layer
>> > > (Flask application) you going to need to control/manage this layer as
>> > well,
>> > > creating a chicken and egg problem.
>> > >
>> > > I believe the Docker API is a simple solution that solves the problem
>> > > (communication between toolbox and containers).
>> > >
>> > > Take a look here
>> > >
>> > >
>> >
>> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602=461=1423=1=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
>> > >
>> > >
>> > > Em sex., 13 de mar. de 2020 às 11:19, Lucas Cardoso Silva <
>> > > cardosolucas61@gmail.com> escreveu:
>> > >
>> > > > I thought about using a flask application, which can become very
>> fast
>> > and
>> > > > scalable with a Python WSGI server for production like Gunicorn. We
>> can
>> > > > make the communication establish by HTTPS protocol and use a public
>> and
>> > > > private key generation system for each engine generated. Another,
>> > > althougt
>> > > > complex, alternative would be use OAuth 2.0.
>> > > >
>> > > > Regards,
>> > > > Lucas Cardoso
>> > > >
>> > > > Em sex., 13 de mar. de 2020 às 13:58, Daniel Takabayashi <
>> > > > daniel.takabaya...@gmail.com> escreveu:
>> > > >
>> > > > > @Daniel The biggest challenge is to run commands in a remote
>> > container
>> > > > > without having access to the whole O.S. Basically the idea is to
>> > create
>> > > > an
>> > > > > interface to make possible, in a secure way the communication
>> > > > > between Toolbox and remote engines.
>> > > > >
>> > > > > Creating this we could start to run our engines in clouds and
>> > services
>> > > > like
>> > > > > Google Run, Kubernetes, Lambda Functions and etc in the same way
>> > Marvin
>> > > > > runs locally.
>> > > > >
>> > > > > @Lucas Lets talk about the details of these APIs (interfaces,
>> > > technology)
>> > > > >
>> > > > > Thanks,
>> > > > > Taka
>> > > > >
>> > > > > Em sex., 13 de mar. de 2020 às 07:21, Daniel Lucredio <
>> > > > > daniel.lucre...@ufscar.br> escreveu:
>>

Re: Daemon gRPC

[Lucas Cardoso Silva]

I believe that we must keep both options. For develop in a controlled
environment, the insecure server option is easier. But gRPC has several
authentication methods. The best and easiest to implement, in my opinion,
is SSL / TLS. The user can generate his own keys and pass it to the modules
to establish a connection.

https://grpc.io/docs/guides/auth/#with-server-authentication-ssltls-4

Em sex., 8 de mai. de 2020 às 02:24, Wei Chen  escreveu:

> One question, should we include authentication information into the proto
> spec?
>
> On Fri, May 8, 2020 at 10:35 AM Lucas Cardoso Silva <
> cardosolucas61@gmail.com> wrote:
>
> > I did a grpc implementation for the daemon, as discussed earlier. I would
> > like to know your opinion about this code. To complete the changes, only
> a
> > user interface module would be missing, which I can do until the
> beginning
> > of next week. All unit tests have also been adapted and the daemon has a
> > code coverage of 70%, but I will try to improve this coverage before
> > commit. Thanks in advance.
> >
> > Proto:
> > https://gist.github.com/cardosolucas/49b8afa0767dd98b39f25365a0371bb4
> > Server:
> > https://gist.github.com/cardosolucas/e20855ce82e27526d976781cb46a6d1d
> >
> > Best regards,
> > Lucas Cardoso
> >
>

Re: Daemon gRPC

[Lucas Cardoso Silva]

Thanks. I made a topic to discuss how I can proceed with the MR. I agree
with the error type too, will include in proto code.

Best regards,

Em qui., 14 de mai. de 2020 às 15:14, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> That's a good initiative.
>
> I agree that we must make auth optional, and IMO it should be something
> that comes in a later stage. Maybe you just make sure your interface would
> be able to receive an authentication implementation in the future, but I
> don't think that is something that needs to be implemented right now. At
> the end of the day there are several ways one can workaround the lack of
> authentication at this level of an architecture.
>
> One suggestion I have is to include an Error type to the proto, so that we
> have a pre-defined way of returning exceptions.
>
> Are you planning to create an issue and MR for it?
>
> Best,
>
> On Fri, May 8, 2020 at 9:27 AM Lucas Cardoso Silva <
> cardosolucas61@gmail.com> wrote:
>
> > I believe that we must keep both options. For develop in a controlled
> > environment, the insecure server option is easier. But gRPC has several
> > authentication methods. The best and easiest to implement, in my opinion,
> > is SSL / TLS. The user can generate his own keys and pass it to the
> modules
> > to establish a connection.
> >
> > https://grpc.io/docs/guides/auth/#with-server-authentication-ssltls-4
> >
> > Em sex., 8 de mai. de 2020 às 02:24, Wei Chen 
> > escreveu:
> >
> > > One question, should we include authentication information into the
> proto
> > > spec?
> > >
> > > On Fri, May 8, 2020 at 10:35 AM Lucas Cardoso Silva <
> > > cardosolucas61@gmail.com> wrote:
> > >
> > > > I did a grpc implementation for the daemon, as discussed earlier. I
> > would
> > > > like to know your opinion about this code. To complete the changes,
> > only
> > > a
> > > > user interface module would be missing, which I can do until the
> > > beginning
> > > > of next week. All unit tests have also been adapted and the daemon
> has
> > a
> > > > code coverage of 70%, but I will try to improve this coverage before
> > > > commit. Thanks in advance.
> > > >
> > > > Proto:
> > > >
> https://gist.github.com/cardosolucas/49b8afa0767dd98b39f25365a0371bb4
> > > > Server:
> > > >
> https://gist.github.com/cardosolucas/e20855ce82e27526d976781cb46a6d1d
> > > >
> > > > Best regards,
> > > > Lucas Cardoso
> > > >
> > >
> >
>

MR for daemon gRPC code

[Lucas Cardoso Silva]

Hello everyone. I created this topic to discuss how we will MR the gRPC
daemon code and the new architecture approach. I thought about creating an
issue and a new branch for this, since there are many changes in the code
and we are not sure if they will be put in the standard development
repository. What do you think?

Best regards,

Re: Communication between CLI and Docker container in the new architecture.

[Lucas Cardoso Silva]

Great! I will do some tests with the "daemon" approach and report any
updates. Thanks!

Em ter., 17 de mar. de 2020 às 17:40, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> No, I think everything is here. The "daemon" idea is the best way to handle
> this problem, basically, we need to have a Marvin agent running inside the
> container responsible to receive and run the commands from outside. But I
> didn't get the part related to Jupyter.
>
>
> Em sex., 13 de mar. de 2020 às 15:15, Lucas Cardoso Silva <
> cardosolucas61@gmail.com> escreveu:
>
> > I did some testing with the Docker API. The only problem is that a
> > container needs a process running to stay active, if it does not exist
> the
> > instance stop before receiving the command through the API. There is the
> > possibility to maintain an infinite and inexpensive process like: tail
> -f /
> > dev / null (not a good idea). Another possibility would be refactor the
> > part of the marvin that would remain in the container to become a daemon
> or
> > leave an instance of Jupyter running (the easy way). Is there any other
> > possibility that I'm not seeing?
> >
> > Em sex., 13 de mar. de 2020 às 15:46, Daniel Takabayashi <
> > daniel.takabaya...@gmail.com> escreveu:
> >
> > > Take a look at the Docker API
> https://docs.docker.com/engine/api/v1.24/,
> > I
> > > believe this will simplify the solution. Cause if you add another layer
> > > (Flask application) you going to need to control/manage this layer as
> > well,
> > > creating a chicken and egg problem.
> > >
> > > I believe the Docker API is a simple solution that solves the problem
> > > (communication between toolbox and containers).
> > >
> > > Take a look here
> > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602=461=1423=1=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > >
> > >
> > > Em sex., 13 de mar. de 2020 às 11:19, Lucas Cardoso Silva <
> > > cardosolucas61@gmail.com> escreveu:
> > >
> > > > I thought about using a flask application, which can become very fast
> > and
> > > > scalable with a Python WSGI server for production like Gunicorn. We
> can
> > > > make the communication establish by HTTPS protocol and use a public
> and
> > > > private key generation system for each engine generated. Another,
> > > althougt
> > > > complex, alternative would be use OAuth 2.0.
> > > >
> > > > Regards,
> > > > Lucas Cardoso
> > > >
> > > > Em sex., 13 de mar. de 2020 às 13:58, Daniel Takabayashi <
> > > > daniel.takabaya...@gmail.com> escreveu:
> > > >
> > > > > @Daniel The biggest challenge is to run commands in a remote
> > container
> > > > > without having access to the whole O.S. Basically the idea is to
> > create
> > > > an
> > > > > interface to make possible, in a secure way the communication
> > > > > between Toolbox and remote engines.
> > > > >
> > > > > Creating this we could start to run our engines in clouds and
> > services
> > > > like
> > > > > Google Run, Kubernetes, Lambda Functions and etc in the same way
> > Marvin
> > > > > runs locally.
> > > > >
> > > > > @Lucas Lets talk about the details of these APIs (interfaces,
> > > technology)
> > > > >
> > > > > Thanks,
> > > > > Taka
> > > > >
> > > > > Em sex., 13 de mar. de 2020 às 07:21, Daniel Lucredio <
> > > > > daniel.lucre...@ufscar.br> escreveu:
> > > > >
> > > > > > Hi Lucas and everyone,
> > > > > >
> > > > > > Couldn't the developer just run the CLI from inside the
> container,
> > > > > opening
> > > > > > a shell inside it?
> > > > > >
> > > > > > []s
> > > > > >
> > > > > > Daniel
> > > > > >
> > > > > > Em qui., 5 de mar. de 2020 às 13:31, Lucas Cardoso Silva <
> > > > > > cardosolucas61@gmail.com> escreveu:
> > > > > >
> > > > > > > I think we could define better how we will make the
> communication
> > > CLI
> > > > > > with
> > > > > > > the 

Daemon gRPC

[Lucas Cardoso Silva]

I did a grpc implementation for the daemon, as discussed earlier. I would
like to know your opinion about this code. To complete the changes, only a
user interface module would be missing, which I can do until the beginning
of next week. All unit tests have also been adapted and the daemon has a
code coverage of 70%, but I will try to improve this coverage before
commit. Thanks in advance.

Proto: https://gist.github.com/cardosolucas/49b8afa0767dd98b39f25365a0371bb4
Server:
https://gist.github.com/cardosolucas/e20855ce82e27526d976781cb46a6d1d

Best regards,
Lucas Cardoso

Marvin's Community Meeting

[Lucas Cardoso Silva]

Hey guys,

In another thread we discussed the possibility of a meeting to address some
issues that are too complex to be discussed through the mailing list. So I
took the initiative to try to organize it. Everyone is invited to speak and
watch, but nothing is required.

Here are the meeting notes (
https://docs.google.com/document/d/1TxejuReHN8Q62oPx9ZPJb5nwEfBDYWrLVQI-aKSVCFg/edit?usp=sharing),
please fill in the attendance area if you are going to attend. To
participate actively, put the subject you want to discuss in the Agenda
area.

My suggestion is to limit 10 min + questions for each member.
Here is the doodle to facilitate scheduling (
https://doodle.com/poll/75dutabeiqfzy2nu), but the dates are only
suggestions and we can work on that.

If all goes well, we will be able to set a date until next Monday.

@Wei, I tried to adapt the times for your timezone, let us know if it works
for you.

Best Regards,
Lucas Cardoso

Re: Septembr Report

[Lucas Cardoso Silva]

+1

Em seg., 31 de ago. de 2020 às 12:03, Zhang Yifei 
escreveu:

> Hello guys,
>
> Here is our report
>
> https://cwiki.apache.org/confluence/display/INCUBATOR/September2020#marvin-ai
>
> Please  feel free to make any updates.
>
>
> Best regards
>

Documentation improvement

[Lucas Cardoso Silva]

Hi guys! As discussed in the other threads, we will improve our
documentation.

I created an issue on ReadTheDocs setup:
https://issues.apache.org/jira/projects/MARVIN/issues/MARVIN-76
We need someone with access to the infrastructure to do some of this work.

I also made a sketch of a new documentation structure. We can discuss these
points and then, it will be open to contributions. We will split the topics
between contributors who are comfortable to write about it, this is a good
first issue as well. A lot of this points are already documented, so we
only need to improve a bit.

Installation:
   - Linux;
   - Mac.
Basic Usage:
   - Toolbox Commands;
   - Iris Example;
   - DASFE;
   - Unit testing.
Serializers:
   - KerasSerializer;
   - How to make a custom serializer.
Engine:
   - engine.metadata;
   - engine.messages;
   - engine.params;
   - feedback.messages;
Engine Executor:
   - EE Features;
   - Remote artifact persistence:
   - File System;
   - HDFS;
   - Azure;
   - AmazonS3;
Spark:
   - Spark Example
Developer guide

Any suggestions?

Best regards,
Lucas Cardoso

Re: Presentation of the architectural proposal

[Lucas Cardoso Silva]

A call would be great if we were able to organize. Some things can be
confusing by email, but I will try to clarify some points of what I meant.

The email was to carry out a control step within the architectural analysis
I am proposing, but as I believe that everyone is familiar with an
architectural proposal, even more than I am, I tried to generate discussion
suggesting a small change.

Answering the question from @lucas. I believe it would be easier to work
with bind mounts, as it would allow the toolbox to create an engine outside
the container, and it would be easily accessible inside the container. I
couldn't find any way to put files inside a docker volume without using a
container.

I just did a PR with a new CLI, if it is accepted, the architectural change
will be well advanced: https://github.com/apache/incubator-marvin/pull/52

Best regards,
Lucas Cardoso

Em qua., 26 de ago. de 2020 às 15:35, Wei Chen 
escreveu:

> That will be nice, I am in Taipei (GMT+8)
>
> Best Regards,
> Wei
>
> On Thu, Aug 27, 2020 at 12:20 AM Daniel Takabayashi <
> daniel.takabaya...@gmail.com> wrote:
>
> > Guys if you prefer we could organize a call to discuss these more complex
> > points. Maybe could be more productive.
> >
> > Em dom., 23 de ago. de 2020 às 22:39, Daniel Takabayashi <
> > daniel.takabaya...@gmail.com> escreveu:
> >
> > > Hi Lucas,
> > >
> > > I am a little confused about your e-mail, I couldn't get your point.
> > > Could you please be a little more specific?
> > >
> > > Thanks,
> > > Taka
> > >
> > > Em sáb., 15 de ago. de 2020 às 14:32, Lucas Cardoso Silva <
> > > cardosolucas61@gmail.com> escreveu:
> > >
> > >> Hi guys,
> > >>
> > >> To continue our goal to evaluate the architecture, we have to present
> > the
> > >> proposal for all involved. I will take the opportunity to discuss some
> > >> points about the implementation of the daemon and new CLI.
> > >>
> > >> Each engine has a Dockerfile for development where all dependencies
> > >> related
> > >> to the operating system will be declared. When the CLI is invoked for
> > the
> > >> engine, the image is created and the container starts, with the daemon
> > as
> > >> the main execution. All modifications to the engine are saved directly
> > to
> > >> the files (via bind mount). All commands, except the creational and
> > >> versioning commands, run on the daemon (called by the CLI). Do you
> agree
> > >> with this approach? It’s a little different from that represented in
> the
> > >> image, which represents the use of a volume.
> > >>
> > >> Architectural proposal image:
> > >>
> > >>
> >
> https://github.com/lucasbm88/incubator-apache-marvin/wiki/Development-Resources#v005---refactoring-architecture
> > >>
> > >> Image of the current architecture:
> > >>
> > >>
> >
> https://docs.google.com/drawings/d/1UOR8Bk0fpLAOnotdeAn4Ww1GBbRxhdtM5mfoTJKQNVo/edit?usp=sharing
> > >>
> > >> Thanks a lot,
> > >> Lucas
> > >>
> > >
> >
>

Re: Moving to python 3

[Lucas Cardoso Silva]

As we are in the middle of the change where we separate the toolbox into a
daemon and CLI, there are still no python 2.7 compatibility modules. The PR
was to include the daemon in CI, already with python 3, before the addition
of the compatibility modules and to allow other developers to include
support for new tools, such as autokeras.

PR link: https://github.com/apache/incubator-marvin/pull/51

Best regards,
Lucas Cardoso

Em seg., 24 de ago. de 2020 às 02:45, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> The migration from python2 to python3 is vital. And the toolbox already had
> supported it a long time ago, but not as the default version. So, if I
> understood well, the only change here should be making py3 as the default
> of the toolbox and for templates. Could you send us your PR link?
>
> Thanks!
>
> Em qui., 6 de ago. de 2020 às 14:43, Bruno Sette <
> brunosilvase...@gmail.com>
> escreveu:
>
> > +1
> >
> > Em qui, 6 de ago de 2020 16:02, Lucas Cardoso Silva <
> > cardosolucas61@gmail.com> escreveu:
> >
> > > I made a pull request to change our components to python 3. In my
> opinion
> > > it's easier to make the architectural change this way, also python 2
> > > reached it's end of support. I would like to know the opinion of you
> guys
> > > about it, so we can make it in the best way for everyone.
> > >
> > > Best regards,
> > > Lucas Cardoso
> > >
> >
>

Re: Marvin’s mission discussion

[Lucas Cardoso Silva]

Good! I agree.

The Apache Marvin-AI platform aims to offer a practical and standardized
solution to help its users to perform data exploration, model development
and application lifecycle management for artificial intelligence tasks,
aiming to offer: scalability, language agnosticism and a standardized
pipeline.

Something like this?


Em sáb., 15 de ago. de 2020 às 16:03, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> +1
>
> Em sáb., 15 de ago. de 2020 às 08:57, Lucas Bonatto Miguel <
> lucasb...@apache.org> escreveu:
>
> > It's good, the only thing I would change would be to mention what sort of
> > applications. Although we have AI in the name, one may mistakenly think
> > Marvin is intended to serve any type of application.
> >
> > Best
> >
> > On Fri, Aug 14, 2020 at 11:37 AM Lucas Cardoso Silva <
> > cardosolucas61@gmail.com> wrote:
> >
> > > Hi guys,
> > >
> > > Here comes the summarized Marvin mission:
> > >
> > > The Apache Marvin-AI platform aims to offer a practical and
> standardized
> > > solution to help its users to perform data exploration, model
> development
> > > and application lifecycle management, aiming to offer: scalability,
> > > language agnosticism and a standardized pipeline.
> > >
> > > Thanks for the help,
> > > Lucas Cardoso
> > >
> > > Em qua., 29 de jul. de 2020 às 17:05, Lucas Cardoso Silva <
> > > cardosolucas61@gmail.com> escreveu:
> > >
> > > > Hi guys!
> > > > Great Lucas, I will wait a couple of days to see if anyone has other
> > > > things to add, and then we can close this phase!
> > > >
> > > > Wei, we can discuss how to make the data pipelines easier to the
> users
> > in
> > > > another step of the evaluation. With the experience of the users and
> > > > developers with this topic we can track their needs better and make
> > > > use-case scenarios. I agree with you that data preparation is messy
> and
> > > can
> > > > take a lot of time and will be great if Marvin could help in that.
> > > >
> > > > Best regards,
> > > > Lucas
> > > >
> > > >
> > > > Em qua., 29 de jul. de 2020 às 11:59, Wei Chen 
> > > > escreveu:
> > > >
> > > >> Hello Lucas,
> > > >>
> > > >> I am thinking of processing JSON or XML files with a hierarchy
> dynamic
> > > >> structure.
> > > >> Or building a pipeline to crop image with object detection metadata.
> > > >> Data preparation can be very messy,
> > > >> I wonder if we can have a stage to handle both batch and streaming
> > > >> processing well.
> > > >>
> > > >> I simply think we don't need to focus on this part since we can
> > utilize
> > > a
> > > >> wide variety of tools for our specific needs.
> > > >>
> > > >> Best Regards,
> > > >> Wei
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Jul 29, 2020 at 8:48 PM Lucas Bonatto Miguel <
> > > >> lucasb...@apache.org>
> > > >> wrote:
> > > >>
> > > >> > Hi folks,
> > > >> >
> > > >> > In regards to the mission, you're correct. If I could summarize
> it,
> > it
> > > >> > would be like: *to help its users to perform data exploration,
> model
> > > >> > development and application lifecycle management*.
> > > >> >
> > > >> > I'm all in for having a better integration with Kubernetes. I
> think
> > > that
> > > >> > the first step is to create a new thread in order to design
> > something
> > > >> > following their operator pattern:
> > > >> > https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
> > > >> >
> > > >> > Wei, currently one already can perform merges and joins in the
> > > >> > transformation step. Could you comment a bit more on what you
> think
> > we
> > > >> > could improve there? Maybe something for a new thread as well?
> > > >> >
> > > >> > Best!
> > > >> > Lucas
> > > >> >
> > > >> > On Wed, Jul 29, 2020 at 1:24 AM Wei Chen 
> > wrote:
> > > >> >
> &

Re: Architecture meeting

[Lucas Cardoso Silva]

Hello guys, the meeting was interrupted but I believe we were almost
finishing anyway. It was a good discussion!

My notes:

1. We need to work continuously on the documentation, as it is currently a
weak spot. We discussed and more or less agreed on the following:
- Let's use ReadTheDocs for the user guides (we will need to configure that
in the Github repository and configure redirect and DNS in Apache
infrastructure).
- Let's include "deeper" use cases, in the form of guides. We can divide
this work and invite more contributors to write. We need to decide on what
topics could appear in the documentation, so that we can start writing.
- For development, we can use RFC format. Maybe we can start by writing a
RFC with the proposal for Marvin's new direction (see item 3 below)

2. "We should focus on what we are good at, and reuse what others are good
at" (Taka's words). It is very difficult to be always updated with the
industry and new technologies.

3. A "new" direction for Marvin started to appear, with two main "new
features":
- We could provide alternatives to Marvin's serving. Like TFX, as it seems
close enough to DASFE (it might even have been inspired by it!). The idea
is to make Marvin extensible, more like a SDK, and focus on making the
learning curve for these tools smoother (as DASFE is a really easy to learn
design pattern).
- We could use Apache Beam to improve data acquisition in Marvin.
- I could try to write a simple RFC for this, so that we can work out the
details!

Did I forget something?

Best regards!

Em qui., 24 de set. de 2020 às 22:05, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> Folks, our meeting was interrupted due to Google Meets issues:
> [image: image.png]
>
> It was a good discussions. Let's share our notes here.
>
> Best,
> Lucas
>

Re: Marvin's Community Meeting

[Lucas Cardoso Silva]

Hey guys, hope you're all fine!
Thank you for participating. Unfortunately we were unable to find a day and
time when everyone will be available, so I selected one of the most voted:
Sep 24, 9 PM GMT -3 (https://doodle.com/poll/75dutabeiqfzy2nu).
The meeting will be on: https://meet.google.com/oyz-ijqw-pmh (I have
unlimited access on this platform).

If you have any questions, please let me know. Thank you :)

Best regards,
Lucas Cardoso

Em seg., 14 de set. de 2020 às 16:57, Lucas Cardoso Silva <
cardosolucas61@gmail.com> escreveu:

> Hey guys,
>
> In another thread we discussed the possibility of a meeting to address
> some issues that are too complex to be discussed through the mailing list.
> So I took the initiative to try to organize it. Everyone is invited to
> speak and watch, but nothing is required.
>
> Here are the meeting notes (
> https://docs.google.com/document/d/1TxejuReHN8Q62oPx9ZPJb5nwEfBDYWrLVQI-aKSVCFg/edit?usp=sharing),
> please fill in the attendance area if you are going to attend. To
> participate actively, put the subject you want to discuss in the Agenda
> area.
>
> My suggestion is to limit 10 min + questions for each member.
> Here is the doodle to facilitate scheduling (
> https://doodle.com/poll/75dutabeiqfzy2nu), but the dates are only
> suggestions and we can work on that.
>
> If all goes well, we will be able to set a date until next Monday.
>
> @Wei, I tried to adapt the times for your timezone, let us know if
> it works for you.
>
> Best Regards,
> Lucas Cardoso
>

Re: Marvin's Community Meeting

[Lucas Cardoso Silva]

Hello guys!
We will share all the resources after the meeting and work on you joining
us next time, Harshit!
Thanks for the ical, @lucas!

Best regards,
Lucas Cardoso

Em ter., 22 de set. de 2020 às 09:04, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> Here's an ical file for those who want to add the meeting to your calendar.
>
>
> On Tue, Sep 22, 2020 at 8:48 AM Lucas Bonatto Miguel 
> wrote:
>
>> Hey Harshit, yup I can do it.
>>
>> I will also propose that we do another session soon, which time zone
>> would work best for you in the future?
>>
>> Best,
>> Lucas
>>
>> On Tue, Sep 22, 2020 at 4:49 AM harshit rohatgi 
>> wrote:
>>
>>> Hello all,
>>> Been following the list. Contributed to Marvin once and looking to get
>>> back
>>> to it. The meeting notes caught my attention. Unfortunately, due to the
>>> time zone, I wouldn't be able to join. Could someone collate the meeting
>>> notes from the meeting and send it to the group. Would help in
>>> contribution.
>>>
>>>
>>> On Tue, 15 Sep, 2020, 1:28 AM Lucas Cardoso Silva, <
>>> cardosolucas61@gmail.com> wrote:
>>>
>>> > Hey guys,
>>> >
>>> > In another thread we discussed the possibility of a meeting to address
>>> some
>>> > issues that are too complex to be discussed through the mailing list.
>>> So I
>>> > took the initiative to try to organize it. Everyone is invited to
>>> speak and
>>> > watch, but nothing is required.
>>> >
>>> > Here are the meeting notes (
>>> >
>>> >
>>> https://docs.google.com/document/d/1TxejuReHN8Q62oPx9ZPJb5nwEfBDYWrLVQI-aKSVCFg/edit?usp=sharing
>>> > ),
>>> > please fill in the attendance area if you are going to attend. To
>>> > participate actively, put the subject you want to discuss in the Agenda
>>> > area.
>>> >
>>> > My suggestion is to limit 10 min + questions for each member.
>>> > Here is the doodle to facilitate scheduling (
>>> > https://doodle.com/poll/75dutabeiqfzy2nu), but the dates are only
>>> > suggestions and we can work on that.
>>> >
>>> > If all goes well, we will be able to set a date until next Monday.
>>> >
>>> > @Wei, I tried to adapt the times for your timezone, let us know if it
>>> works
>>> > for you.
>>> >
>>> > Best Regards,
>>> > Lucas Cardoso
>>> >
>>>
>>

Re: Cookiecutter for templates and plugins

[Lucas Cardoso Silva]

Sure, Iet's make this work.

Best regards,
Lucas Cardoso

Em ter., 26 de mai. de 2020 às 19:43, Zhang Yifei 
escreveu:

> Hello Lucas,
>
> It looks very interesting, do you have any availability this week?
> Let's talk about it.
>
> Em sex., 22 de mai. de 2020 às 18:44, Lucas Cardoso Silva <
> cardosolucas61@gmail.com> escreveu:
>
> > Marvin uses his own code to generate the engines. I would like to propose
> > the use of the Cookiecutter library. It is an interesting approach with
> > several features included that could easily compose a strategy to allow
> > third parties to develop plugins. This would contribute to, for example,
> > contributions like those in AutoML being made separately from the main
> > development and not making things bloated. Some plugins could include
> > custom serializers for different platforms, like Keras, and make it
> easier
> > to use.
> >
> > The Cookiecutter library is under BSD license, which is very permissive.
> >
> > Project documentation link: https://cookiecutter.readthedocs.io/
> >
>
>
> --
> --
> Zhang Yifei
>

Re: June report

[Lucas Cardoso Silva]

Hi Zhang,

LGTM.

Best regards,
Lucas Cardoso

Em qui, 28 de mai de 2020 14:15, Zhang Yifei 
escreveu:

> Hello guys,
>
> Please take a look at our report here
> 
> We need to finish it as soon as possible.
>
> If anyone has anything to complete,
> post here or edit from the report page.
>
> Thanks
>
> --
> --
> Zhang Yifei
>

Cookiecutter for templates and plugins

[Lucas Cardoso Silva]

Marvin uses his own code to generate the engines. I would like to propose
the use of the Cookiecutter library. It is an interesting approach with
several features included that could easily compose a strategy to allow
third parties to develop plugins. This would contribute to, for example,
contributions like those in AutoML being made separately from the main
development and not making things bloated. Some plugins could include
custom serializers for different platforms, like Keras, and make it easier
to use.

The Cookiecutter library is under BSD license, which is very permissive.

Project documentation link: https://cookiecutter.readthedocs.io/

Re: Marvin’s mission discussion

[Lucas Cardoso Silva]

Hi guys!
Great Lucas, I will wait a couple of days to see if anyone has other things
to add, and then we can close this phase!

Wei, we can discuss how to make the data pipelines easier to the users in
another step of the evaluation. With the experience of the users and
developers with this topic we can track their needs better and make
use-case scenarios. I agree with you that data preparation is messy and can
take a lot of time and will be great if Marvin could help in that.

Best regards,
Lucas


Em qua., 29 de jul. de 2020 às 11:59, Wei Chen 
escreveu:

> Hello Lucas,
>
> I am thinking of processing JSON or XML files with a hierarchy dynamic
> structure.
> Or building a pipeline to crop image with object detection metadata.
> Data preparation can be very messy,
> I wonder if we can have a stage to handle both batch and streaming
> processing well.
>
> I simply think we don't need to focus on this part since we can utilize a
> wide variety of tools for our specific needs.
>
> Best Regards,
> Wei
>
>
>
> On Wed, Jul 29, 2020 at 8:48 PM Lucas Bonatto Miguel  >
> wrote:
>
> > Hi folks,
> >
> > In regards to the mission, you're correct. If I could summarize it, it
> > would be like: *to help its users to perform data exploration, model
> > development and application lifecycle management*.
> >
> > I'm all in for having a better integration with Kubernetes. I think that
> > the first step is to create a new thread in order to design something
> > following their operator pattern:
> > https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
> >
> > Wei, currently one already can perform merges and joins in the
> > transformation step. Could you comment a bit more on what you think we
> > could improve there? Maybe something for a new thread as well?
> >
> > Best!
> > Lucas
> >
> > On Wed, Jul 29, 2020 at 1:24 AM Wei Chen  wrote:
> >
> > > I think deploying to K8S does expend our capabilities for inference
> > scaling
> > > and managing.
> > > I am not familiar with Luigi, but it makes sense since we are going to
> > > setup data pipelines.
> > >
> > > Best Regards,
> > > Wei
> > >
> > > On Wed, Jul 29, 2020 at 5:32 AM Lucas Cardoso Silva <
> > > cardosolucas61@gmail.com> wrote:
> > >
> > > > Great Wei! I find the suggestions really interesting. I think we can
> > work
> > > > with the deployment on K8s. The idea of it in Marvin would be, after
> > > > development, the user would give some parameters and a script would
> > > > facilitate a deployment in a kubernetes cluster, right? Regarding
> data
> > > > acquisition, I think it would be great if we were able to integrate
> > some
> > > > third party library like Luigi. Thanks!
> > > >
> > > >
> > > >
> > > > Em qua., 22 de jul. de 2020 às 14:27, Wei Chen 
> > > > escreveu:
> > > >
> > > > > Hello Lucas,
> > > > >
> > > > > I have some ideas:
> > > > >
> > > > > 1. Should we consider to use K8S or similar tools for inference
> > > container
> > > > > scaling and management?
> > > > > Marvin's current container management is not as powerful as some
> > > > container
> > > > > focus projects.
> > > > > K8S can also be deployed into most environments now.
> > > > >
> > > > > 2. Is our current data cleaning stage flexible enough for multiple
> > data
> > > > > sources with table join?
> > > > > Or if we should cut the data preparation stage out for the user to
> > make
> > > > > their own data pipeline on their data storage.
> > > > > I figured that preprocessing might be too complex to be generalized
> > for
> > > > > different ML projects.
> > > > >
> > > > > Best Regards
> > > > > Wei
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jul 23, 2020 at 12:26 AM Lucas Cardoso Silva <
> > > > > cardosolucas61@gmail.com> wrote:
> > > > >
> > > > > > Hi guys.
> > > > > > I would like to know if anyone else has any ideas about this
> > > evaluation
> > > > > > phase. Both the opinion of those who have been in the community
> > for a
> > > > > long
> > > > > > time and those who are still ge

Engine Unit Tests

[Lucas Cardoso Silva]

I have a question about unit testing on the engine. The test are designed
to serve as a framework for the user to write their own tests, or they
should work as a standard test, which would work for any user code? I
particularly believe in the first option, but I would like a better
understanding of this design decision.

Best regards,
Lucas Cardoso

Marvin’s mission discussion

[Lucas Cardoso Silva]

Hello guys. The time has come for us to take the first step in
architectural assessment: the definition of the mission. Basically we have
to decide here what is important in Marvin and what is outside the scope of
the project. This is important because, during this analysis and the
development process as a whole, we will be able to segment what is really
important and make things more simple and functional. Also, if it looks
cool, we can include that on the Marvin-AI homepage.

As stated earlier, I will post an initial draft and would like to receive
your feedback to complete a few points:

The Apache Marvin-AI platform aims to offer:

   -

   a practical and standardized solution,
   -

   for the development and deployment of machine learning applications.


Aiming to offer the user:

   -

   scalability,
   -

   language agnosticism,
   -

   standardized pipeline (DASFE),
   -

   possibility of remote versioning of artifacts.


Does anyone have any suggestions for more important features, resources or
design decisions in Marvin?

Thank you very much,

Lucas Cardoso

Architecture Evaluation process on Marvin-AI

[Lucas Cardoso Silva]

Hello everyone. I would like to propose a more in-depth analysis of
Marvin's architectural proposal and for that I need your help. The idea
isn’t to change the proposal, but to identify architectural approaches and
describe scenarios around different contexts of use. This is part of my
M.Sc. research, which is a case study of how to carry out such analysis in
an open source environment. The activity is also interesting for the
Marvin-AI community, as it has the potential to generate important insight
regarding the proposed architecture, allowing us to identify new features
or modify some approaches as needed.

The evaluation will be carried out using a traditional evaluation method
called ATAM, developed at the Software Engineering Institute at CMU. It was
originally developed to be carried out in a room with the presence of all
stakeholders, in short sessions. Obviously, we can't do that, therefore I
will make some adaptations, so that we can do it via email, GitHub, and the
other tools we have available.

Q:

What do you need from me?

R: Just check your e-mail once in a while and give us your opinion and
ideas. That's it! Each interaction in the process should only take the
response time of a normal email (about 15 minutes). I expect to send no
more than one or two e-mails every week.

Do you really need me?

Literature recommends that at least 5 stakeholders are involved in the
process. We have 5 PMC/committers, and it would be great if everyone is on
board. I will take the liberty to encourage member participation, so that
the interactions become more valuable.

How long will it take?

R: There is no time limit, but I expect around 6-8 weeks of effort. That
depends on the discussions, though.

How does this work?

R: Long answer ->
https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=5177.

R: Short answer:

   -

   First we define Marvin-AI's mission, what Marvin proposes to do and what
   is out of bounds. This formal definition is important for future steps,
   ensuring that only what is in scope is evaluated.
   -

   Next we try to identify the architectural approaches and quality
   attributes. The architectural approach is how Marvin-AI is structured.
   Quality attributes are a form of non-functional requirements that are
   supposed to be implemented at the architectural level. Marvin-AI is
   supposed to satisfy these attributes.
   -

   Scenario brainstorming and prioritization: here we try to describe some
   usage scenarios (real or imaginary) that cover Marvin's main architectural
   approaches. We then vote to prioritize these scenario as critical / common
   / not important.


   -

   Analysis of the scenarios within the architectural proposal: we will
   make an exercise to verify how the architectural proposal performs in each
   scenario considered as critical or common.
   -

   Compilation of results: Presentation and compilation of results and
   lessons learned.


That seems complicated. Can you explain better?

R: Yes! I will try to be very careful and explain in details what everyone
should do, before each step in the process.

Thank you very much,

Lucas

Re: Cookiecutter for templates and plugins

[Lucas Cardoso Silva]

Thanks guys. I create an issue.

https://issues.apache.org/jira/projects/MARVIN/issues/MARVIN-73


Em qua., 3 de jun. de 2020 às 04:38, Mario de Sá Vera 
escreveu:

> Good stuff !
>
> On Wed, 3 Jun 2020, 03:21 Lucas Bonatto Miguel, 
> wrote:
>
> > It's a good idea. Have you guys ended up making progress in it? Is there
> an
> > issue for us to follow and participate?
> >
> > - Lucas
> >
> > On Wed, May 27, 2020 at 2:07 PM Lucas Cardoso Silva <
> > cardosolucas61@gmail.com> wrote:
> >
> > > Sure, Iet's make this work.
> > >
> > > Best regards,
> > > Lucas Cardoso
> > >
> > > Em ter., 26 de mai. de 2020 às 19:43, Zhang Yifei <
> > yifei.z.l...@gmail.com>
> > > escreveu:
> > >
> > > > Hello Lucas,
> > > >
> > > > It looks very interesting, do you have any availability this week?
> > > > Let's talk about it.
> > > >
> > > > Em sex., 22 de mai. de 2020 às 18:44, Lucas Cardoso Silva <
> > > > cardosolucas61@gmail.com> escreveu:
> > > >
> > > > > Marvin uses his own code to generate the engines. I would like to
> > > propose
> > > > > the use of the Cookiecutter library. It is an interesting approach
> > with
> > > > > several features included that could easily compose a strategy to
> > allow
> > > > > third parties to develop plugins. This would contribute to, for
> > > example,
> > > > > contributions like those in AutoML being made separately from the
> > main
> > > > > development and not making things bloated. Some plugins could
> include
> > > > > custom serializers for different platforms, like Keras, and make it
> > > > easier
> > > > > to use.
> > > > >
> > > > > The Cookiecutter library is under BSD license, which is very
> > > permissive.
> > > > >
> > > > > Project documentation link: https://cookiecutter.readthedocs.io/
> > > > >
> > > >
> > > >
> > > > --
> > > > --
> > > > Zhang Yifei
> > > >
> > >
> >
>

Re: Marvin’s mission discussion

[Lucas Cardoso Silva]

Great Wei! I find the suggestions really interesting. I think we can work
with the deployment on K8s. The idea of it in Marvin would be, after
development, the user would give some parameters and a script would
facilitate a deployment in a kubernetes cluster, right? Regarding data
acquisition, I think it would be great if we were able to integrate some
third party library like Luigi. Thanks!



Em qua., 22 de jul. de 2020 às 14:27, Wei Chen 
escreveu:

> Hello Lucas,
>
> I have some ideas:
>
> 1. Should we consider to use K8S or similar tools for inference container
> scaling and management?
> Marvin's current container management is not as powerful as some container
> focus projects.
> K8S can also be deployed into most environments now.
>
> 2. Is our current data cleaning stage flexible enough for multiple data
> sources with table join?
> Or if we should cut the data preparation stage out for the user to make
> their own data pipeline on their data storage.
> I figured that preprocessing might be too complex to be generalized for
> different ML projects.
>
> Best Regards
> Wei
>
>
>
> On Thu, Jul 23, 2020 at 12:26 AM Lucas Cardoso Silva <
> cardosolucas61@gmail.com> wrote:
>
> > Hi guys.
> > I would like to know if anyone else has any ideas about this evaluation
> > phase. Both the opinion of those who have been in the community for a
> long
> > time and those who are still getting to know Marvin is now important for
> > this step, so your suggestion or validation of the initial text is always
> > welcome!
> >
> > Best regards,
> > Lucas Cardoso
> >
> > Em sex., 10 de jul. de 2020 às 13:48, Lucas Cardoso Silva <
> > cardosolucas61@gmail.com> escreveu:
> >
> > > Hello guys. The time has come for us to take the first step in
> > > architectural assessment: the definition of the mission. Basically we
> > have
> > > to decide here what is important in Marvin and what is outside the
> scope
> > of
> > > the project. This is important because, during this analysis and the
> > > development process as a whole, we will be able to segment what is
> really
> > > important and make things more simple and functional. Also, if it looks
> > > cool, we can include that on the Marvin-AI homepage.
> > >
> > > As stated earlier, I will post an initial draft and would like to
> receive
> > > your feedback to complete a few points:
> > >
> > > The Apache Marvin-AI platform aims to offer:
> > >
> > >-
> > >
> > >a practical and standardized solution,
> > >-
> > >
> > >for the development and deployment of machine learning applications.
> > >
> > >
> > > Aiming to offer the user:
> > >
> > >-
> > >
> > >scalability,
> > >-
> > >
> > >language agnosticism,
> > >-
> > >
> > >standardized pipeline (DASFE),
> > >-
> > >
> > >possibility of remote versioning of artifacts.
> > >
> > >
> > > Does anyone have any suggestions for more important features, resources
> > or
> > > design decisions in Marvin?
> > >
> > > Thank you very much,
> > >
> > > Lucas Cardoso
> > >
> >
>

Re: Marvin’s mission discussion

[Lucas Cardoso Silva]

Hi guys.
I would like to know if anyone else has any ideas about this evaluation
phase. Both the opinion of those who have been in the community for a long
time and those who are still getting to know Marvin is now important for
this step, so your suggestion or validation of the initial text is always
welcome!

Best regards,
Lucas Cardoso

Em sex., 10 de jul. de 2020 às 13:48, Lucas Cardoso Silva <
cardosolucas61@gmail.com> escreveu:

> Hello guys. The time has come for us to take the first step in
> architectural assessment: the definition of the mission. Basically we have
> to decide here what is important in Marvin and what is outside the scope of
> the project. This is important because, during this analysis and the
> development process as a whole, we will be able to segment what is really
> important and make things more simple and functional. Also, if it looks
> cool, we can include that on the Marvin-AI homepage.
>
> As stated earlier, I will post an initial draft and would like to receive
> your feedback to complete a few points:
>
> The Apache Marvin-AI platform aims to offer:
>
>-
>
>a practical and standardized solution,
>-
>
>for the development and deployment of machine learning applications.
>
>
> Aiming to offer the user:
>
>-
>
>scalability,
>-
>
>language agnosticism,
>-
>
>standardized pipeline (DASFE),
>-
>
>possibility of remote versioning of artifacts.
>
>
> Does anyone have any suggestions for more important features, resources or
> design decisions in Marvin?
>
> Thank you very much,
>
> Lucas Cardoso
>

Re: Marvin’s mission discussion

[Lucas Cardoso Silva]

Hi guys,

Here comes the summarized Marvin mission:

The Apache Marvin-AI platform aims to offer a practical and standardized
solution to help its users to perform data exploration, model development
and application lifecycle management, aiming to offer: scalability,
language agnosticism and a standardized pipeline.

Thanks for the help,
Lucas Cardoso

Em qua., 29 de jul. de 2020 às 17:05, Lucas Cardoso Silva <
cardosolucas61@gmail.com> escreveu:

> Hi guys!
> Great Lucas, I will wait a couple of days to see if anyone has other
> things to add, and then we can close this phase!
>
> Wei, we can discuss how to make the data pipelines easier to the users in
> another step of the evaluation. With the experience of the users and
> developers with this topic we can track their needs better and make
> use-case scenarios. I agree with you that data preparation is messy and can
> take a lot of time and will be great if Marvin could help in that.
>
> Best regards,
> Lucas
>
>
> Em qua., 29 de jul. de 2020 às 11:59, Wei Chen 
> escreveu:
>
>> Hello Lucas,
>>
>> I am thinking of processing JSON or XML files with a hierarchy dynamic
>> structure.
>> Or building a pipeline to crop image with object detection metadata.
>> Data preparation can be very messy,
>> I wonder if we can have a stage to handle both batch and streaming
>> processing well.
>>
>> I simply think we don't need to focus on this part since we can utilize a
>> wide variety of tools for our specific needs.
>>
>> Best Regards,
>> Wei
>>
>>
>>
>> On Wed, Jul 29, 2020 at 8:48 PM Lucas Bonatto Miguel <
>> lucasb...@apache.org>
>> wrote:
>>
>> > Hi folks,
>> >
>> > In regards to the mission, you're correct. If I could summarize it, it
>> > would be like: *to help its users to perform data exploration, model
>> > development and application lifecycle management*.
>> >
>> > I'm all in for having a better integration with Kubernetes. I think that
>> > the first step is to create a new thread in order to design something
>> > following their operator pattern:
>> > https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
>> >
>> > Wei, currently one already can perform merges and joins in the
>> > transformation step. Could you comment a bit more on what you think we
>> > could improve there? Maybe something for a new thread as well?
>> >
>> > Best!
>> > Lucas
>> >
>> > On Wed, Jul 29, 2020 at 1:24 AM Wei Chen  wrote:
>> >
>> > > I think deploying to K8S does expend our capabilities for inference
>> > scaling
>> > > and managing.
>> > > I am not familiar with Luigi, but it makes sense since we are going to
>> > > setup data pipelines.
>> > >
>> > > Best Regards,
>> > > Wei
>> > >
>> > > On Wed, Jul 29, 2020 at 5:32 AM Lucas Cardoso Silva <
>> > > cardosolucas61@gmail.com> wrote:
>> > >
>> > > > Great Wei! I find the suggestions really interesting. I think we can
>> > work
>> > > > with the deployment on K8s. The idea of it in Marvin would be, after
>> > > > development, the user would give some parameters and a script would
>> > > > facilitate a deployment in a kubernetes cluster, right? Regarding
>> data
>> > > > acquisition, I think it would be great if we were able to integrate
>> > some
>> > > > third party library like Luigi. Thanks!
>> > > >
>> > > >
>> > > >
>> > > > Em qua., 22 de jul. de 2020 às 14:27, Wei Chen 
>> > > > escreveu:
>> > > >
>> > > > > Hello Lucas,
>> > > > >
>> > > > > I have some ideas:
>> > > > >
>> > > > > 1. Should we consider to use K8S or similar tools for inference
>> > > container
>> > > > > scaling and management?
>> > > > > Marvin's current container management is not as powerful as some
>> > > > container
>> > > > > focus projects.
>> > > > > K8S can also be deployed into most environments now.
>> > > > >
>> > > > > 2. Is our current data cleaning stage flexible enough for multiple
>> > data
>> > > > > sources with table join?
>> > > > > Or if we should cut the data preparation stage out for the user to
>> > make
>> &g

Re: December report

[Lucas Cardoso Silva]

+1

Em ter., 1 de dez. de 2020 às 16:34, Zhang Yifei 
escreveu:

> Hello guys,
>
> Here is our report
>
> https://cwiki.apache.org/confluence/display/INCUBATOR/December2020#marvin-ai
>
> please feel free to make any changes.
>
>
> Regards
>

Re: [DISCUSS] About ReadTheDocs integration

[Lucas Cardoso Silva]

In any case, if we are not allowed to do it, we can compile the source
files that we already made using Sphinx. This will generate html files in a
ReadTheDocs-like format.
A example of a Apache software documentation made this way:
https://usergrid.apache.org/docs/

Best regards,
Lucas Cardoso

Em qui., 29 de out. de 2020 às 14:54, Zhang Yifei 
escreveu:

> Hello guys,
>
> We were working on issue MARVIN-76
> 
>  to
> replace the documentation page,
> since we have better docs in Readthedocs
> (Associated with Bruno
> Sette's forked repo currently) than our site.
>
> When I logged in Readthedocs with my github account, it appears some extra
> permissions are needed at the organization(Apache) level.
> [image: Screenshot from 2020-10-29 14-25-28.png]
>
> Any ideas about this? Actually I don't even know if we are allowed to do
> this kind of integration...
>
> Tks
>
>
> Yifei Zhang
>

Re: Did you saw that?

[Lucas Cardoso Silva]

This is very interesting! In our architecture meeting we discussed a lot of
possibilities to keep the architecture change on Marvin. Integrate with TFX
and data acquisition with beam were some of them. Studying TFX
orchestration on Kubeflow I saw that we could use KFP and other Kubeflow
features as well! We will discuss these issues in RFC format, feel free to
share your thoughts on that!

@luciano Elyra is a very interesting project as well. I saw your
presentation on B2WSummit! It would be great if you could share any ideas
in our RFCs!

Maybe it's time to discuss a pattern on our RFC submissions, so it's
gonna be easier for everyone to participate and understand the new features.

Best regards,
Lucas Cardoso

Em ter., 27 de out. de 2020 às 18:59, Rafael Novello <
rafa.reis.nove...@gmail.com> escreveu:

> Hi guys!
>
> Did you have saw that?
>
> https://github.com/kubeflow-kale/kale
>
> It`s like Marvin Jupyter integration but it integrate with Kubeflow
> Atenciosamente,
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>

TFX RFC and how to proceed with it

[Lucas Cardoso Silva]

Hi guys!

I wrote an RFC containing an implementation of an integration between the
Tensorflow Extended framework and Marvin, as foreseen in our architectural
meeting. I would like everyone to participate by giving suggestions of
modifications and even better solutions that the proposal offers. Here is
the link: https://hackmd.io/@cardosolucas/B1E6aysKw/edit

I would also like to ask what would be the community's preference for
registering RFCs, since this is the first time that we have done something
like this. I suggest creating a folder in the repository to store a
history, modify and discuss it.

Looking forward to your participation!

Best Regards
Lucas Cardoso

Re: June Report

[Lucas Cardoso Silva]

+1

On Thu, Jun 3, 2021 at 2:43 PM Zhang Yifei  wrote:

> Hello guys,
>
> Here is our initial version of the report:
> https://cwiki.apache.org/confluence/display/INCUBATOR/June2021#marvin-ai
>
> Please feel free to add anything.
>
>
> Best regards
>
> Yifei Zhang
>

Re: Repeating report

[Lucas Cardoso Silva]

+1

On Mon, Mar 29, 2021 at 4:35 PM Zhang Yifei  wrote:

> Hello guys,
>
> Since our March report did not have signoff by mentors,
> we had to send the same report again for April.
>
> Here is our report :
> https://cwiki.apache.org/confluence/display/INCUBATOR/April2021#marvin-ai
>
> Thanks.
>
> --
> --
> Zhang Yifei
>

Feedback on Architecture Evaluation

[Lucas Cardoso Silva]

Hello guys, how are you doing? I'm halfway to finish the process to get my
master degree. My work was an architecture evaluation on Marvin's
architectural proposal, as some of you may remember from discussions on the
mailing list. I'm sending this email to ask a favor for you guys. I need a
feedback on this evaluation from the community. The procedure is simple:

1) Watch this video: https://youtu.be/6Pbai1Wa_18, here I explain some
features on Apache Marvin-AI, some of them will be in a PR I'm preparing
and will come out soon.
2) Fill this form: https://forms.gle/WCiYXhBFL8pBpfms7. It has some
questions about some use case scenarios on Apache Marvin-AI.

Thanks in advance for your help!
Lucas Cardoso Silva

Architecture Meeting

[Lucas Cardoso Silva]

Hey guys,

In another thread we discussed the possibility of a meeting to address some
architecture issues, like the last time. So let's organize it. Everyone is
invited to speak and watch, but nothing is required.

Here are the meeting notes (
https://docs.google.com/document/d/1TxejuReHN8Q62oPx9ZPJb5nwEfBDYWrLVQI-aKSVCFg/edit?usp=sharing),
please fill in the attendance area if you are going to attend. To
participate actively, put the subject you want to discuss in the Agenda
area.

Here is the doodle to facilitate scheduling (
https://doodle.com/poll/8ncqbdmhcxr9ctf3?utm_source=poll_medium=link),
but the dates are only suggestions and we can work on that.

If all goes well, we will be able to set a date until next Monday.

Best Regards,
Lucas Cardoso

Re: Apache Marvin-AI Meeting

[Lucas Cardoso Silva]

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//ical.marudot.com//iCal Event Maker
CALSCALE:GREGORIAN
BEGIN:VTIMEZONE
TZID:America/Sao_Paulo
TZURL:http://tzurl.org/zoneinfo-outlook/America/Sao_Paulo
X-LIC-LOCATION:America/Sao_Paulo
BEGIN:DAYLIGHT
TZOFFSETFROM:-0300
TZOFFSETTO:-0200
TZNAME:-02
DTSTART:19701101T00
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0300
TZOFFSETTO:-0300
TZNAME:-03
DTSTART:19700215T00
RRULE:FREQ=YEARLY;BYMONTH=2;BYDAY=3SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20210316T182959Z
UID:20210316t182959z-2057372...@marudot.com
DTSTART;TZID=America/Sao_Paulo:20210318T22
DTEND;TZID=America/Sao_Paulo:20210318T23
SUMMARY:Apache Marvin-AI Meeting
URL:https://meet.google.com/hcf-snia-geg
END:VEVENT
END:VCALENDAR
BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//ical.marudot.com//iCal Event Maker
CALSCALE:GREGORIAN
BEGIN:VTIMEZONE
TZID:America/Sao_Paulo
TZURL:http://tzurl.org/zoneinfo-outlook/America/Sao_Paulo
X-LIC-LOCATION:America/Sao_Paulo
BEGIN:DAYLIGHT
TZOFFSETFROM:-0300
TZOFFSETTO:-0200
TZNAME:-02
DTSTART:19701101T00
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0300
TZOFFSETTO:-0300
TZNAME:-03
DTSTART:19700215T00
RRULE:FREQ=YEARLY;BYMONTH=2;BYDAY=3SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20210316T182959Z
UID:20210316t182959z-2057372...@marudot.com
DTSTART;TZID=America/Sao_Paulo:20210318T22
DTEND;TZID=America/Sao_Paulo:20210318T23
SUMMARY:Apache Marvin-AI Meeting
URL:https://meet.google.com/hcf-snia-geg
END:VEVENT
END:VCALENDAR

Architecture meeting

[Lucas Cardoso Silva]

Hey guys, hope you're doing well! In the next few days I'll do some PRs
regarding the implementation of some features discussed in the other
meeting. I would like to propose another meeting next month to discuss
these features. I think it is a quick way to discuss the design decisions.
Do you guys agree? Any suggestions are welcome!

Thanks,
Lucas Cardoso

Commiter status

[Lucas Cardoso Silva]

Hey guys, how are you?

I know that this is not the standard procedure, but I would like to ask the
community to give me the commiter status. We will have 3 new undergrad
students at our lab mainly working on Marvin, so it would be great to
review their code as I will be their supervisor.

We are going to discuss with all the community some new features and ideas
to make the project up and running again, and new guys and girls are coming.

Thanks in advance,
Lucas Cardoso Silva

December report

[Lucas Cardoso Silva]

Hey folks! Please take a look at this month's report:
https://cwiki.apache.org/confluence/display/INCUBATOR/December2021#marvin-ai
If you guys have any questions, additions or corrections please feel free
to point or edit!

Best regards,
Lucas Cardoso

Re: Request for vision review

[Lucas Cardoso Silva]

+1 I think it's a good plan to discuss these changes.

On Thu, Nov 4, 2021 at 6:03 PM Lucas Bonatto Miguel 
wrote:

> Hi,
>
> As the current version of Marvin is not getting enough traction in both
> usage and development, some action needs to be taken. After discussing with
> a few stakeholders, we believe that the scope of the problem Marvin is
> trying to solve should be simplified. One of the ideas that I would like to
> propose is for us to remove some parts of the code, related to training and
> inference execution in production, and instead generate packages that can
> be executed by other de-facto platforms out there, argo, kfp, tecton, tfx,
> etc.
>
> I have no doubts that this would simplify the codebase immensely, and in
> addition, it would make Marvin useful for more people, by taking advantage
> of this new integration layer.
>
> If enough people are interested in this change, I suggest we schedule a
> meeting to kick off the ideas for this RFC.
>
> Please respond with +1 or -1
>
> Best,
> Lucas
>

Re: New Marvin Release (0.0.6)

[Lucas Cardoso Silva]

Yes, it's just planning for now. Sorry for the confusion, I was not
familiar with the process to make an Apache compliant release. We will work
towards making the release following the Apache release policy guidelines.
Thanks for the information!

On Wed, Feb 23, 2022 at 11:20 PM Luciano Resende 
wrote:

> On Wed, Feb 23, 2022 at 12:08 PM Lucas Cardoso Silva <
> cardosolucas61@gmail.com> wrote:
>
> > Hi guys, how are you? Since we are preparing a new release for Marvin,
> I'm
> > writing this to let you know that I'll start to work to publish this new
> > version. Does anyone have anything to suggest or an impediment of any
> kind?
> > I'd also like to know if Apache has a guide on releasing versions on Pypi
> > and official docker images.
> >
> > The changelog for this version is available here:
> >
> >
> https://github.com/apache/incubator-marvin/blob/develop/python-toolbox/CHANGES.md
> >
> > Best regards,
> > Lucas Cardoso
> >
>
> What do you mean by "New Marvin Release (0.0.6)" ? I am assuming you are
> planning to start working on a new release? You should NOT publish or call
> something a release without proper vote and IPMC approval.
>
> Here is the release management docs
> https://incubator.apache.org/guides/releasemanagement.html
>
> And release distribution docs
> https://incubator.apache.org/guides/distribution.html
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

New Marvin Release (0.0.6)

[Lucas Cardoso Silva]

Hi guys, how are you? Since we are preparing a new release for Marvin, I'm
writing this to let you know that I'll start to work to publish this new
version. Does anyone have anything to suggest or an impediment of any kind?
I'd also like to know if Apache has a guide on releasing versions on Pypi
and official docker images.

The changelog for this version is available here:
https://github.com/apache/incubator-marvin/blob/develop/python-toolbox/CHANGES.md

Best regards,
Lucas Cardoso

Re: [ISSUE] MARVIN issue posted on jira

[Lucas Cardoso Silva]

Welcome to the community Daniel!
I'll take a look at this issue and get back here soon.

On Mon, Jul 4, 2022 at 7:15 PM Daniel Lombardi 
wrote:

> Hello Marvin Team,
>
> Last week, while testing marvin development branch, I found a bug regarding
> the generation and updating of the scripts inside data_handler directory
> (from within the container). I am just sending this e-mail to let everyone
> know. Thanks!
>
> Issue link: https://issues.apache.org/jira/browse/MARVIN-81
>

Marvin Report

[Lucas Cardoso Silva]

I made a draft of our may/22 report:
https://cwiki.apache.org/confluence/display/INCUBATOR/May2022#marvin-ai

Please feel free to add anything that may seem necessary.

Best regards,
Lucas Cardoso Silva

Design proposal draft finished

[Lucas Cardoso Silva]

Hi guys! A while ago Lucas Bonatto reached out to the dev list with a RFC
template to make a design review. Now we have a full draft on this design
proposal and is the time to get your final feedback before we vote to
decide on this new architecture.

 This is a proposal to modernize Marvin's architecture given the latest
developments in the MLOps area. Discussions and changes, including in core
concepts of the proposal, are very welcome.

Link to the draft:
https://www.notion.so/WIP-RFC-Scope-Revamp-9cff5b9633094e9f9cc32e92edce145f

Best regards,
Lucas Cardoso Silva

Re: Missing PPMC and lack of activity on this project

[Lucas Cardoso Silva]

Hi everyone,
It is indeed hard to keep up with development and community tasks with such
low community engagement. I'll follow the community decision on this issue,
especially PPMC members.
Best regards,
Lucas Cardoso

On Wed, Oct 26, 2022 at 9:05 PM Justin Mclean  wrote:

> Hi,
> The  Incubator recently asked for a roll call of the PPMC members for this
> project; no one responded. A project needs at least 3 active PPMC members
> to continue in the Incubator. I can see that traffic on the list is low. Is
> there any interest in continuing to have this project at the ASF or should
> the IPMC consider retiring it?
> Kind Regards,
> Justin
>