Did you saw that?

[Rafael Novello]

Hi guys!

Did you have saw that?

https://github.com/kubeflow-kale/kale

It`s like Marvin Jupyter integration but it integrate with Kubeflow
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/

Re: Marvin Architecture Validation

[Rafael Novello]

Hi Lucas!

I believe all diagrams are correct and precise. It'll be so good for Marvin
Docs!

Thanks and Congrats!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em ter., 3 de dez. de 2019 às 14:32, Lucas Cardoso Silva <
cardosolucas61@gmail.com> escreveu:

> Hi guys,
> I am considering using Marvin as an artifact for my master's dissertation.
> The idea is to conduct a formal architectural design and validation process
> in an Open Source environment. While conducting the process we should
> produce sufficient documentation to guide developers through the
> refactoring of architecture (architectural views, scenarios, requirements
> ...) and validate the architecture by adapting to the scenarios listed by
> contributors.
>
> To make this happen, I ask your collaboration to validate Marvin's current
> architectural views as we will use them as the starting point of the
> process. Marvin UML diagrams have also been requested at:
> https://issues.apache.org/jira/projects/MARVIN/issues/MARVIN-67.
>
> Figure 1 is a class diagram of the base engine:
>
> https://docs.google.com/drawings/d/183GbZHci4vIAqZDLP0jcUL231wdQ5cNAO2rhyTKCUCY/edit?usp=sharing
> Figure 2 is the component diagram:
>
> https://docs.google.com/drawings/d/1qSd1eAksJ_mqn4Tu7ZCnGnEWuJaL7hfPQGyCPhW2fdk/edit?usp=sharing
> Figure 3 is the deploy diagram:
>
> https://docs.google.com/drawings/d/1CdtjHa-QR9SwZbBqkalUHN9cL2sHiP75NAFN7bRt8-0/edit?usp=sharing
> Figure 4 is a generic architectural view inspired by the new architecture
> figure:
>
> https://docs.google.com/drawings/d/1UOR8Bk0fpLAOnotdeAn4Ww1GBbRxhdtM5mfoTJKQNVo/edit?usp=sharing
>
> Thanks in advance!
>

Re: dynamic_import error in docker

[Rafael Novello]

Hi Zhang,

It's look like your engine was not installed successfully on you docker
image. Try to rebuild your docker image and check the installation process
or access your engine container by bash and try to install the engine
manually (with something like python setup install)

Best!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em seg., 4 de nov. de 2019 às 11:17, Zhang Yifei 
escreveu:

> Hello guys,,
>
> Any idea about this error here
>
> https://issues.apache.org/jira/projects/MARVIN/issues/MARVIN-69?filter=allopenissues
> ?
>
> The engine works perfectly while running directly in local machine with
> command: marvin engine-httpserver,
> When encapsulated in docker container, the same command to start httpserver
> raise dynamic_import error - ImportError: No module named
> marvin_cba3_engine
>
> Looks like some dependency mapping error
>
>
> Many thanks
>
> --
> --
> Zhang Yifei
>

Re: HttpServer Concurrency

[Rafael Novello]

Hi Taka!

To be effective you need to merge the max-workers and max-rpc-workers
options. I was able to achieve much better performance by adjusting these
parameters simultaneously!

Anyway, thanks for the help!

Best!

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qui, 31 de out de 2019 às 19:02, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> Hi Rafael,
>
> So basically the problem here is resource vs. configuration in the gRPC
> side, take a look:
>
>
> https://stackoverflow.com/questions/54316767/how-to-handle-resource-exhausted-error-in-api-calls
>
> The actual configuration is just a generic tune we found to offer
> "acceptable" performance, given some tests we did in the past.
> You can find this configuration inside Marvin right here:
>
>
> https://github.com/apache/incubator-marvin/blob/develop/engine-executor/src/main/resources/application.conf
>
> https://github.com/apache/incubator-marvin/blob/06c2138e28021ea181466c6661512757502d2c19/python-toolbox/marvin_python_toolbox/engine_base/engine_base_action.py#L185
>
> https://github.com/apache/incubator-marvin/blob/06c2138e28021ea181466c6661512757502d2c19/python-toolbox/marvin_python_toolbox/management/engine.py#L289
>
> Each model should have a different requirement, which leads us to a
> different configuration of workers, parallelism, and more.
>
> I hope I've helped you.
>
> Taka
>
>
>
>
>
>
>
>
> Em qui, 31 de out de 2019 às 13:38, Rafael Novello <
> rafa.reis.nove...@gmail.com> escreveu:
>
> > Hi guys!
> >
> > Someone knows how to deal with the following error:
> >
> > [INFO] [10/31/2019 17:29:35.737]
> > [lendico_score-akka.actor.default-dispatcher-3]
> > [GenericAPI(akka://lendico_score)] RECEIVE FAILURE!!! RESOURCE_EXHAUSTED:
> > Concurrent RPC limit exceeded!class io.grpc.StatusRuntimeException
> >
> > I had tested the max-workers and max-rpc-workers at engine-httpserver
> but I
> > saw no difference.
> >
> > Thanks!!
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
>

HttpServer Concurrency

[Rafael Novello]

Hi guys!

Someone knows how to deal with the following error:

[INFO] [10/31/2019 17:29:35.737]
[lendico_score-akka.actor.default-dispatcher-3]
[GenericAPI(akka://lendico_score)] RECEIVE FAILURE!!! RESOURCE_EXHAUSTED:
Concurrent RPC limit exceeded!class io.grpc.StatusRuntimeException

I had tested the max-workers and max-rpc-workers at engine-httpserver but I
saw no difference.

Thanks!!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/

Re: make docker-build fail

[Rafael Novello]

@Daniel

I did a PR for this task! Could you review and merge?

Thanks!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em ter, 18 de jun de 2019 às 15:07, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> Yes, I agree and I am quite sure this change will not break anything. I
> signed this task for me, and I am working on this right now.
>

Re: Dockerfile Updates

[Rafael Novello]

Sure! I'll also create a branch for each one to help on merge!

Thanks Wei!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em ter, 11 de jun de 2019 às 16:28, Wei Chen  escreveu:

> I think we should have a Jira ticket for each task.
> And we work on the PR one by one, so it is more clear.
>
> On Tue, Jun 11, 2019 at 2:16 PM Rafael Novello <
> rafa.reis.nove...@gmail.com>
> wrote:
>
> > Hi guys!
> >
> > I would like to do some improvements on Dockerfile like:
> >
> > 1. Change to openjdk instead oracle jdk [1]
> > 2. Update to Python 3
> > 3. Make the docker image small
> >
> > The first item is a bug fix (we can't build new images without it) but
> the
> > last two are improvements. What is the right process to follow? Could I
> > just do this updates and push or we need to vote?
> >
> > Thanks!
> > [1] - https://issues.apache.org/jira/browse/MARVIN-58
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
>

Dockerfile Updates

[Rafael Novello]

Hi guys!

I would like to do some improvements on Dockerfile like:

1. Change to openjdk instead oracle jdk [1]
2. Update to Python 3
3. Make the docker image small

The first item is a bug fix (we can't build new images without it) but the
last two are improvements. What is the right process to follow? Could I
just do this updates and push or we need to vote?

Thanks!
[1] - https://issues.apache.org/jira/browse/MARVIN-58
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/

Re: Incubator Report Draft

[Rafael Novello]

Thank you Luciano!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em seg, 10 de jun de 2019 às 18:15, Luciano Resende 
escreveu:

> I have added the report to the wiki.
>
> On Wed, Jun 5, 2019 at 11:17 AM Rafael Novello
>  wrote:
> >
> > Hi Lucas, thanks for your help!
> >
> > Could someone update the wiki for me pls? I still do not have write
> access.
> >
> > Thanks!
> > 
> >
> > Marvin-AI
> >
> > Marvin-AI is an open-source artificial intelligence (AI) platform that
> helps
> > data scientists, prototype and productionalize complex solutions with a
> > scalable, low-latency, language-agnostic, and standardized architecture
> > while
> > simplifies the process of exploration and modeling.
> >
> > Marvin-AI has been incubating since 2018-08-21.
> >
> > Three most important issues to address in the move towards graduation:
> >
> >   1.Move development infrastructure to Apache
> >   2.Build Community
> >   3.Build Releases
> >
> > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > aware of?
> >
> >   * No. *(I'm right here?)*
> >
> > How has the community developed since the last report?
> >
> >   * Open voting for a new PMC member, Wei Chen.
> >   * Participation in the local PyData event with a presentation of the
> > project by Zhang Yifei.
> >
> > How has the project developed since the last report?
> >
> >   * Started the development for R support.
> >
> > Have your mentors been helpful and responsive or are things falling
> > through the cracks? In the latter case, please list any open issues
> > that need to be addressed.
> >
> >   * The mentors are responsive when the community needs it.
> >
> > How would you assess the podling's maturity?
> > Please feel free to add your own commentary.
> >
> >   [ ] Initial setup
> >   [x] Working towards first release
> >   [ ] Community building
> >   [ ] Nearing graduation
> >   [ ] Other:
> >
> > Date of last release:
> >
> >   -XX-XX
> >
> > When were the last committers or PPMC members elected?
> >
> >   * N/A
> >
> > Signed-off-by:
> >
> >   [ ](marvin-ai) Jim Jagielski
> >  Comments:
> >   [ ](marvin-ai) Luciano Resende
> >  Comments:
> >   [ ](marvin-ai) William Colen
> >  Comments:
> >
> >
> >
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em qua, 5 de jun de 2019 às 10:30, Daniel Takabayashi <
> > daniel.takabaya...@gmail.com> escreveu:
> >
> > > +1
> > >
> > > Sent from my iPhone
> > >
> > > > On Jun 5, 2019, at 5:35 AM, Luciano Resende 
> > > wrote:
> > > >
> > > > +1
> > > >
> > > > On Mon, Jun 3, 2019 at 18:29 Rafael Novello <
> rafa.reis.nove...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi guys!
> > > >>
> > > >> So sorry about the delay but here is a draft for the report. What
> are
> > > you
> > > >> think?
> > > >>
> > > >> Some one could help to put it on the Wiki?
> > > >>
> > > >> Best Regards!
> > > >>
> > > >> 
> > > >>
> > > >> Marvin-AI
> > > >>
> > > >> Marvin-AI is an open-source artificial intelligence (AI) platform
> that
> > > >> helps
> > > >> data scientists, prototype and productionalize complex solutions
> with a
> > > >> scalable, low-latency, language-agnostic, and standardized
> architecture
> > > >> while
> > > >> simplifies the process of exploration and modeling.
> > > >>
> > > >> Marvin-AI has been incubating since 2018-08-21.
> > > >>
> > > >> Three most important issues to address in the move towards
> graduation:
> > > >>
> > > >>  1.Move development infrastructure to Apache
> > > >>  2.Build Community
> > > >>  3.Build Releases
> > > >>
> > > >> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to
> be
> > > >> aware of?
> > > >>
&g

Re: Incubator Report Draft

[Rafael Novello]

Hi Lucas, thanks for your help!

Could someone update the wiki for me pls? I still do not have write access.

Thanks!


Marvin-AI

Marvin-AI is an open-source artificial intelligence (AI) platform that helps
data scientists, prototype and productionalize complex solutions with a
scalable, low-latency, language-agnostic, and standardized architecture
while
simplifies the process of exploration and modeling.

Marvin-AI has been incubating since 2018-08-21.

Three most important issues to address in the move towards graduation:

  1.Move development infrastructure to Apache
  2.Build Community
  3.Build Releases

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

  * No. *(I'm right here?)*

How has the community developed since the last report?

  * Open voting for a new PMC member, Wei Chen.
  * Participation in the local PyData event with a presentation of the
project by Zhang Yifei.

How has the project developed since the last report?

  * Started the development for R support.

Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.

  * The mentors are responsive when the community needs it.

How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [x] Working towards first release
  [ ] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  -XX-XX

When were the last committers or PPMC members elected?

  * N/A

Signed-off-by:

  [ ](marvin-ai) Jim Jagielski
 Comments:
  [ ](marvin-ai) Luciano Resende
 Comments:
  [ ](marvin-ai) William Colen
 Comments:



Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qua, 5 de jun de 2019 às 10:30, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> +1
>
> Sent from my iPhone
>
> > On Jun 5, 2019, at 5:35 AM, Luciano Resende 
> wrote:
> >
> > +1
> >
> > On Mon, Jun 3, 2019 at 18:29 Rafael Novello  >
> > wrote:
> >
> >> Hi guys!
> >>
> >> So sorry about the delay but here is a draft for the report. What are
> you
> >> think?
> >>
> >> Some one could help to put it on the Wiki?
> >>
> >> Best Regards!
> >>
> >> 
> >>
> >> Marvin-AI
> >>
> >> Marvin-AI is an open-source artificial intelligence (AI) platform that
> >> helps
> >> data scientists, prototype and productionalize complex solutions with a
> >> scalable, low-latency, language-agnostic, and standardized architecture
> >> while
> >> simplifies the process of exploration and modeling.
> >>
> >> Marvin-AI has been incubating since 2018-08-21.
> >>
> >> Three most important issues to address in the move towards graduation:
> >>
> >>  1.Move development infrastructure to Apache
> >>  2.Build Community
> >>  3.Build Releases
> >>
> >> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> >> aware of?
> >>
> >>  * No. *(I'm right here?)*
> >>
> >> How has the community developed since the last report?
> >>
> >>  * Open voting for a new PMC member, Wei Chen.
> >>  * Participation in the local PyData event with a presentation of the
> >> project by Zhang Yifei.
> >>
> >> How has the project developed since the last report?
> >>
> >>  * Started the development for R support.
> >>
> >> Have your mentors been helpful and responsive or are things falling
> >> through the cracks? In the latter case, please list any open issues
> >> that need to be addressed.
> >>
> >>  * At least two of the mentors are sometimes active.
> >>
> >> How would you assess the podling's maturity?
> >> Please feel free to add your own commentary.
> >>
> >>  [ ] Initial setup
> >>  [x] Working towards first release
> >>  [ ] Community building
> >>  [ ] Nearing graduation
> >>  [ ] Other:
> >>
> >> Date of last release:
> >>
> >>  -XX-XX
> >>
> >> When were the last committers or PPMC members elected?
> >>
> >>  * N/A
> >>
> >> Signed-off-by:
> >>
> >>  [ ](marvin-ai) Jim Jagielski
> >> Comments:
> >>  [ ](marvin-ai) Luciano Resende
> >> Comments:
> >>  [ ](marvin-ai) William Colen
> >> Comments:
> >>
> >> Atenciosamente,
> >> Rafael J. R. Novello
> >>
> >> Skype: rafael.novello
> >> Blog: http://rafanovello.blogspot.com.br/
> >>
> > --
> > Sent from my Mobile device
>

Re: make docker-build fail

[Rafael Novello]

Apparently, there are no more a way to install oracle jdk with repository
(apt-get): https://launchpad.net/~webupd8team/+archive/ubuntu/java

I'm a little bit confused, Do you think we should change?

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em ter, 4 de jun de 2019 às 19:30, Wei Chen  escreveu:

> I think we should change to openjdk.
> just change  oraclejdk8  in .travis.yml to openjdk8 I faced similar error
> in other projects,
> I think Travis-ci is not supporting oraclejdk8 for Scala 2.12?
> Cus my Scala 2.11 projects still work while all Scala 2.12 cannot.
>
> Best Regards
> Wei
>
>
> On Tue, Jun 4, 2019 at 5:03 PM Rafael Novello  >
> wrote:
>
> > Hi guys!
> >
> > I had some problems with docker-buid command as I described here [1]
> >
> > I'm doing some successful tests with openjdk instead Oracle JDK. Have
> > someone any concerns about this change?
> >
> > [1] - https://issues.apache.org/jira/browse/MARVIN-58
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
>

make docker-build fail

[Rafael Novello]

Hi guys!

I had some problems with docker-buid command as I described here [1]

I'm doing some successful tests with openjdk instead Oracle JDK. Have
someone any concerns about this change?

[1] - https://issues.apache.org/jira/browse/MARVIN-58
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/

Incubator Report Draft

[Rafael Novello]

Hi guys!

So sorry about the delay but here is a draft for the report. What are you
think?

Some one could help to put it on the Wiki?

Best Regards!



Marvin-AI

Marvin-AI is an open-source artificial intelligence (AI) platform that helps
data scientists, prototype and productionalize complex solutions with a
scalable, low-latency, language-agnostic, and standardized architecture
while
simplifies the process of exploration and modeling.

Marvin-AI has been incubating since 2018-08-21.

Three most important issues to address in the move towards graduation:

  1.Move development infrastructure to Apache
  2.Build Community
  3.Build Releases

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

  * No. *(I'm right here?)*

How has the community developed since the last report?

  * Open voting for a new PMC member, Wei Chen.
  * Participation in the local PyData event with a presentation of the
project by Zhang Yifei.

How has the project developed since the last report?

  * Started the development for R support.

Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.

  * At least two of the mentors are sometimes active.

How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [x] Working towards first release
  [ ] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  -XX-XX

When were the last committers or PPMC members elected?

  * N/A

Signed-off-by:

  [ ](marvin-ai) Jim Jagielski
 Comments:
  [ ](marvin-ai) Luciano Resende
 Comments:
  [ ](marvin-ai) William Colen
 Comments:

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/

Re: [VOTE] New PMC candidate: Wei Chen

[Rafael Novello]

+1 for sure!

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qui, 30 de mai de 2019 às 10:42, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> As we are growing our community we have been fortunate to have some very
> dedicated people gaining interest in the project. @weichen aka Wei Chen has
> been very active in both the mail list and code contribution. As a
> committer Wei has contributed to important parts of our documentation, our
> website and more recently reviewing code. Please, vote in order to have Wei
> at the Project Management Committee for Marvin.
>
> Voting ends one week from today, i.e. midnight UTC on
>  2019-06-06 00:00 UTC
>
> http://www.timeanddate.com/counters/customcounter.html?year=2019&month=06&day=06
> See voting guidelines athttps://community.apache.org/newcommitter.html
>

Re: Not able to merge through GitHub

[Rafael Novello]

Are there any way that I can help with this? How?

Best
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em ter, 21 de mai de 2019 às 17:45, Wei Chen  escreveu:

> I have the same issue.
> I thought we won't be able to merge it with GitHub directly?
>
> Best Regards
> Wei
>
> On Tue, May 21, 2019 at 3:18 PM Lucas Bonatto Miguel <
> lucasbona...@gmail.com>
> wrote:
>
> > Hey guys, I recently need to change my GitHub user. My new user is
> > lucasbmiguel
> >
> > After updating ApacheID everything is working fine with GitBox however, I
> > can't merge MRs via the GitHub interface. It says I don't have write
> > permissions.
> >
> > Regards,
> > Lucas
> >
>

Incubator Report Draft

[Rafael Novello]

Hi guys!

So sorry about the delay but here is a draft for the report. What are you
think?

Some one could help to put it on the Wiki?

Best Regards!



Marvin-AI

Marvin-AI is an open-source artificial intelligence (AI) platform that helps
data scientists, prototype and productionalize complex solutions with a
scalable, low-latency, language-agnostic, and standardized architecture
while
simplifies the process of exploration and modeling.

Marvin-AI has been incubating since 2018-08-21.

Three most important issues to address in the move towards graduation:

  1.Move development infrastructure to Apache
  2.Build Community
  3.Build Releases

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

  * No.

How has the community developed since the last report?

  * Started a new project with Federal University of São Carlos to help
with project development

How has the project developed since the last report?

  * A new version (0.0.5) was released to PyPI.

Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.

  * At least two of the mentors are sometimes active.

How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [x] Working towards first release
  [ ] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  -XX-XX

When were the last committers or PPMC members elected?

  * N/A

Signed-off-by:

  [ ](marvin-ai) Jim Jagielski
 Comments:
  [ ](marvin-ai) Luciano Resende
 Comments:
  [ ](marvin-ai) William Colen
 Comments:

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/

Re: Exhibit A from Software Grant

[Rafael Novello]

Nice Lucas!!

Very good news!

+1 for your text

Best!
Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em ter, 2 de abr de 2019 às 16:02, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> Hi list,
>
> I was working with B2W, the company who started Marvin, to get their
> signature on the Apache Software Grant (
> https://www.apache.org/licenses/software-grant.txt), so that they formally
> donate the code to our community.
>
> As part of the software grant we need to write an exhibit, that explains
> what's the software.
>
> Here is my proposal, please do your edits and send it back here in the
> list.
>
> ---
> Marvin is an open source project that focuses on empowering data science
> teams to deliver industrial-grade applications supported by a high-scale,
> low-latency, language agnostic and standardized architecture platform,
> while simplifying the process of exploration and modeling. Marvin is a set
> of tools that working together form a platform for creating and operating
> model-based applications in production. The building blocks of Marvin are:
>  1. Engine executor is an application server that exposes HTTP endpoints to
> interact with the deployed model.
>  2. Python toolbox lib is a software library that contain utility methods
> and classes that are needed in order to create a Marvin model.
>  3. Python toolbox cli is a command line utility that is currently the main
> user interface with the platform.
>  4. DASFE is a conceptual design pattern that helps all models to follow
> the same common code structure.
>

Incubator report

[Rafael Novello]

Hi guys!

So sorry about the delay but here is a draft for the report. What are you
think?

Some one could help to put it on the Wiki?

Best Regards!



Marvin-AI

Marvin-AI is an open-source artificial intelligence (AI) platform that helps
data scientists, prototype and productionalize complex solutions with a
scalable, low-latency, language-agnostic, and standardized architecture
while
simplifies the process of exploration and modeling.

Marvin-AI has been incubating since 2018-08-21.

Three most important issues to address in the move towards graduation:

  1.Move development infrastructure to Apache
  2.Build Community
  3.Build Releases

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

  * No.

How has the community developed since the last report?

  * The project began to receive help from community members.
  * The community started discussing new releases.

How has the project developed since the last report?

  * New improvements in documentation and official website
  * Some improvements for a new release

Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.

  * At least two of the mentors are sometimes active.

How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [x] Working towards first release
  [ ] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  -XX-XX

When were the last committers or PPMC members elected?

  * N/A

Signed-off-by:

  [ ](marvin-ai) Jim Jagielski
 Comments:
  [ ](marvin-ai) Luciano Resende
 Comments:
  [ ](marvin-ai) William Colen
 Comments:

Re: Website status

[Rafael Novello]

Very good news Chen!!

We have already started to spread the new site in the communities we
participate! Please, do it too!

Thanks for all you help!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em seg, 7 de jan de 2019 às 10:18, Wei Chen 
escreveu:

> Hello All,
>
> I am glad to say that after several months without a website.
> We finally have our website up and running!
> https://marvin.apache.org/
> @Taka, please also help to redirect http://www.marvin-ai.org to the new
> URL!
>
> Best Regards
> Wei
>
> On Sat, Jan 5, 2019 at 9:54 PM Wei Chen  wrote:
>
> > Hello Taka and Yifei,
> >
> > I noticed that the fix was incomplete and just submitted a new PR.
> > https://github.com/apache/incubator-marvin-website/pull/4
> > Please help to verify it and merge it.
> > Also, please help to assign me the committer authentication for Marvin
> and
> > Marvin-website repositories.
> > I believe it will make the initial setup of the project faster.
> > My apache ID is: weichen
> >
> > Best Regards
> > Wei
> >
> > On Fri, Jan 4, 2019 at 6:50 AM Daniel Takabayashi <
> > daniel.takabaya...@gmail.com> wrote:
> >
> >> ok thanks!
> >>
> >> Em qui, 3 de jan de 2019 às 10:01, Zhang Yifei 
> >> escreveu:
> >>
> >> > Hi Taka,
> >> >
> >> > The Apache infra team asked to rename the directory site to content to
> >> > continue the process.
> >> > I just reviewed and merged the changes hours ago, details here:
> >> > https://github.com/apache/incubator-marvin-website/pull/3
> >> >
> >> > Em qui, 3 de jan de 2019 às 15:54, Daniel Takabayashi <
> >> > daniel.takabaya...@gmail.com> escreveu:
> >> >
> >> > > Hello guys, I check our website link and is broken, any one knows
> >> what is
> >> > > the problem?
> >> > >
> >> > > Thanks,
> >> > > Taka
> >> > >
> >> > > Em sex, 21 de dez de 2018 às 12:34, Lucas Bonatto Miguel <
> >> > > lucasb...@apache.org> escreveu:
> >> > >
> >> > > > Here it is: https://issues.apache.org/jira/browse/INFRA-17490
> >> > > >
> >> > > > Let me know if the description is not good.
> >> > > >
> >> > > > - Lucas
> >> > > >
> >> > > > On Fri, Dec 21, 2018 at 11:44 AM Luciano Resende <
> >> luckbr1...@gmail.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > Anyone want to volunteer to create the infra ticket ? I can help
> >> if
> >> > we
> >> > > > get
> >> > > > > stuck.
> >> > > > >
> >> > > > > On Fri, Dec 21, 2018 at 14:10 Daniel Takabayashi <
> >> > > > > daniel.takabaya...@gmail.com> wrote:
> >> > > > >
> >> > > > > > Good morning guys,
> >> > > > > >
> >> > > > > > Once ready let me know then I can redirect the old domain
> >> > > > > > http://www.marvin-ai.org to the new one.
> >> > > > > >
> >> > > > > > Em sex, 21 de dez de 2018 às 05:54, Wei Chen <
> >> > > weichentai...@gmail.com>
> >> > > > > > escreveu:
> >> > > > > >
> >> > > > > > > Hello Luciano,
> >> > > > > > >
> >> > > > > > > I think our website is ready to be launched.
> >> > > > > > > We should be able to raise the ticket to infra now.
> >> > > > > > >
> >> > > > > > > Best Regards
> >> > > > > > > Wei
> >> > > > > > >
> >> > > > > > > On Fri, Dec 21, 2018 at 9:52 PM Luciano Resende <
> >> > > > luckbr1...@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > What’s the status of the website ? Just asking as i have
> >> seen a
> >> > > lot
> >> > > > > of
> >> > > > > > > > content but the marvin.apache.org is still down.
> >> > > > > > > >
> >> > > > > > > > If we are ready, we need to create a infra ticket to
> enable
> >> > > > gitpubsub
> >> > > > > > to
> >> > > > > > > > the website repo.
> >> > > > > > > >
> >> > > > > > > > Thanks
> >> > > > > > > > --
> >> > > > > > > > Sent from my Mobile device
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > --
> >> > > > > Sent from my Mobile device
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> > --
> >> > --
> >> > Zhang Yifei
> >> >
> >>
> >
>

Re: [MARVIN-IA] Incubator report

[Rafael Novello]

Guys,

Could someone update de wiki incubator report [1] for me please? I can't
login for now.

Thanks a lot!
[1] - https://wiki.apache.org/incubator/January2019
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qua, 2 de jan de 2019 às 12:07, William Colen 
escreveu:

> +1
>
> Em qua, 2 de jan de 2019 às 12:06, Alan Silva  escreveu:
>
> > +1
> >
> > On Wed, Jan 2, 2019 at 11:48 AM Jim Jagielski  wrote:
> >
> > > +1
> > >
> > > > On Jan 2, 2019, at 8:44 AM, Rafael Novello <
> > rafa.reis.nove...@gmail.com>
> > > wrote:
> > > >
> > > > Hi guys!
> > > >
> > > > So sorry about the delay but here is a draft for the report. What are
> > you
> > > > think?
> > > >
> > > > Best Regards!
> > > >
> > > > 
> > > > Marvin-AI
> > > >
> > > > Marvin-AI is an open-source artificial intelligence (AI) platform
> that
> > > helps
> > > > data scientists, prototype and productionalize complex solutions
> with a
> > > > scalable, low-latency, language-agnostic, and standardized
> architecture
> > > > while
> > > > simplifies the process of exploration and modeling.
> > > >
> > > > Marvin-AI has been incubating since 2018-08-21.
> > > >
> > > > Three most important issues to address in the move towards
> graduation:
> > > >
> > > >  1.Move development infrastructure to Apache
> > > >  2.Build Community
> > > >  3.Build Releases
> > > >
> > > > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > > > aware of?
> > > >
> > > >  * No.
> > > >
> > > > How has the community developed since the last report?
> > > >
> > > >  * The project began to receive help from community members (not PPMC
> > > > members) more actively with the development of the project site.
> > > >  * The community started discussing contribution patterns in the
> > > > repository and came up with some good practices that should be
> followed
> > > >  * A new member is being included as a contributor to the project.
> > Only a
> > > > few details are missing for inclusion.
> > > >
> > > > How has the project developed since the last report?
> > > >
> > > >  * A new code repository was created for the development of the
> project
> > > > site following the Apache template.
> > > >  * Bug fixes were done in the code and in the project documentation.
> > > >  * Some PoC's to a new Toolbox version was included to a new branch
> on
> > > the
> > > > repo.
> > > >
> > > > Have your mentors been helpful and responsive or are things falling
> > > > through the cracks? In the latter case, please list any open issues
> > > > that need to be addressed.
> > > >
> > > >  * At least two of the mentors are sometimes active.
> > > >
> > > > How would you assess the podling's maturity?
> > > > Please feel free to add your own commentary.
> > > >
> > > >  [ ] Initial setup
> > > >  [x] Working towards first release
> > > >  [ ] Community building
> > > >  [ ] Nearing graduation
> > > >  [ ] Other:
> > > >
> > > > Date of last release:
> > > >
> > > >  -XX-XX
> > > >
> > > > When were the last committers or PPMC members elected?
> > > >
> > > >  * N/A
> > > >
> > > > Signed-off-by:
> > > >
> > > >  [ ](marvin-ai) Jim Jagielski
> > > > Comments:
> > > >  [ ](marvin-ai) Luciano Resende
> > > > Comments:
> > > >  [ ](marvin-ai) William Colen
> > > > Comments:
> > > >
> > > >
> > > > Atenciosamente,
> > > > Rafael J. R. Novello
> > > >
> > > > Skype: rafael.novello
> > > > Blog: http://rafanovello.blogspot.com.br/
> > >
> > >
> >
> --
> William Colen
>

[MARVIN-IA] Incubator report

[Rafael Novello]

Hi guys!

So sorry about the delay but here is a draft for the report. What are you
think?

Best Regards!


Marvin-AI

Marvin-AI is an open-source artificial intelligence (AI) platform that helps
data scientists, prototype and productionalize complex solutions with a
scalable, low-latency, language-agnostic, and standardized architecture
while
simplifies the process of exploration and modeling.

Marvin-AI has been incubating since 2018-08-21.

Three most important issues to address in the move towards graduation:

  1.Move development infrastructure to Apache
  2.Build Community
  3.Build Releases

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

  * No.

How has the community developed since the last report?

  * The project began to receive help from community members (not PPMC
members) more actively with the development of the project site.
  * The community started discussing contribution patterns in the
repository and came up with some good practices that should be followed
  * A new member is being included as a contributor to the project. Only a
few details are missing for inclusion.

How has the project developed since the last report?

  * A new code repository was created for the development of the project
site following the Apache template.
  * Bug fixes were done in the code and in the project documentation.
  * Some PoC's to a new Toolbox version was included to a new branch on the
repo.

Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.

  * At least two of the mentors are sometimes active.

How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [x] Working towards first release
  [ ] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  -XX-XX

When were the last committers or PPMC members elected?

  * N/A

Signed-off-by:

  [ ](marvin-ai) Jim Jagielski
 Comments:
  [ ](marvin-ai) Luciano Resende
 Comments:
  [ ](marvin-ai) William Colen
 Comments:


Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/

Incubator report is due

[Rafael Novello]

Hi guys!

So sorry about the delay but here is a draft for the report. What are you
think?

Best Regards!


Marvin-AI

Marvin-AI is an open-source artificial intelligence (AI) platform that helps
data scientists, prototype and productionalize complex solutions with a
scalable, low-latency, language-agnostic, and standardized architecture
while
simplifies the process of exploration and modeling.

Marvin-AI has been incubating since 2018-08-21.

Three most important issues to address in the move towards graduation:

  1.Move development infrastructure to Apache
  2.Build Community
  3.Build Releases

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

  * No.

How has the community developed since the last report?

  * The project began to receive help from community members (not PPMC
members) more actively with the development of the project site.
  * The community started discussing contribution patterns in the
repository and came up with some good practices that should be followed

How has the project developed since the last report?

  * A new code repository was created for the development of the project
site following the Apache template.
  * Bug fixes were done in the code and in the project documentation.

Have your mentors been helpful and responsive or are things falling
through the cracks? In the latter case, please list any open issues
that need to be addressed.

  * At least two of the mentors are sometimes active.

How would you assess the podling's maturity?
Please feel free to add your own commentary.

  [ ] Initial setup
  [x] Working towards first release
  [ ] Community building
  [ ] Nearing graduation
  [ ] Other:

Date of last release:

  -XX-XX

When were the last committers or PPMC members elected?

  * N/A

Signed-off-by:

  [ ](marvin-ai) Jim Jagielski
 Comments:
  [ ](marvin-ai) Luciano Resende
 Comments:
  [ ](marvin-ai) William Colen
 Comments:

Re: Pattern for commit comment

[Rafael Novello]

+1 for both suggestions!

I'll use this patterns and we could add it on "how to contribute" doc!

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em sex, 23 de nov de 2018 às 16:05, William Colen 
escreveu:

> What do you think of defining a pattern for commit comment? There are
> guidelines for that, but basically you could:
> - start with the issue so it looks better in reports
> - keep the first line message small so it takes just one line in reports
>
> Something like this:
> `MARVIN-00: This is a commit message`
>
> What do you think?
>
> William
>

Model reload not working

[Rafael Novello]

Hi guys!

After some tests, we notice the reload action is not working. To
be clearer, all the reload process is did, download the artifact and
replace the reference attribute on EngineBaseAction class but the reload
did not work.

We had tested with model reload. We did 2 models, one always return true
and the other always return false and we tried to reload between them
without success.

The only way to change the model in memory is restarting the Engine
Executor process.

How should we proceed in this case? Just open a issue on Jira? What kind of
information should I provide?

Thanks a lot for help!
Rafael Novello

Re: POCs, Prototypes, experiments ...

[Rafael Novello]

Guys,

How can we organize the tasks to start working at this new version?

I believe we had the core devs approval.

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em ter, 30 de out de 2018 às 14:30, Zhang Yifei 
escreveu:

> +1
>
> Em seg, 29 de out de 2018 às 18:23, Rafael Novello <
> rafa.reis.nove...@gmail.com> escreveu:
>
> > Lucas, yes! It make sense!
> >
> > In the end, CLI and REPL will share a big part of code and they will be
> > just different interfaces to the same features.
> >
> > +1 for this new concept!
> >
> > Thanks for make it clear to me!
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em seg, 29 de out de 2018 às 17:43, Lucas Bonatto Miguel <
> > lucasb...@apache.org> escreveu:
> >
> > > That's a good point Rafael. I believe the main motivation for adding a
> > REPL
> > > to Marvin's toolbox is to enhance user experience.
> > >
> > > Imo we can't just switch CLI per REPL. I don't think one excludes the
> > > other, both have different use cases.
> > >
> > > We may be mixing too much stuff together, but in general, I think we
> are:
> > >  - Refactoring toolbox architecture to avoid unnecessarily duplicated
> > logic
> > > when adding support to new languages.
> > >  - Adding a new interface to enable users to open an interactive
> session
> > > with the engine (i.e. REPL)
> > >  - Making it easier to add support for new languages.
> > >
> > > Does it make sense for you?
> > >
> > > On Mon, Oct 29, 2018 at 11:22 AM Rafael Novello <
> > > rafa.reis.nove...@gmail.com>
> > > wrote:
> > >
> > > > I do not want to be the annoying conversation but...
> > > >
> > > > If we will have an unique CLI that will be able to handle any kind of
> > > > engines (in any language) by using docker, Why we need a REPL?
> > > >
> > > > I think we can focus on creating this new architecture using docker
> and
> > > > docker SDK and use only a CLI for that.
> > > >
> > > > Atenciosamente,
> > > > Rafael J. R. Novello
> > > >
> > > > Skype: rafael.novello
> > > > Blog: http://rafanovello.blogspot.com.br/
> > > >
> > > >
> > > > Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
> > > > lucasb...@apache.org> escreveu:
> > > >
> > > > > No, I think the idea is to do not create different versions of the
> > CLI.
> > > > > Let's use one single tech stack for the CLI, ideally, it will be
> > shared
> > > > > with the REPL. The way I see the CLI would be very similar to the
> > REPL,
> > > > but
> > > > > instead of opening an interactive session the user could send
> > > > > unattended commands. As an analogy, you can think on spark-shell
> (our
> > > > REPL)
> > > > > and spark-submit (our CLI).
> > > > >
> > > > > - Lucas
> > > > >
> > > > > On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
> > > > > rafa.reis.nove...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Guys!
> > > > > >
> > > > > > Let me ask about this new version that include CLI's and REPL.
> Will
> > > we
> > > > > > create different versions of CLI?
> > > > > >
> > > > > > One of the reasons to create the REPL is to avoid re-implementing
> > the
> > > > > same
> > > > > > features many times,
> > > > > > each time for a different language.
> > > > > >
> > > > > > Regards!
> > > > > > Atenciosamente,
> > > > > > Rafael J. R. Novello
> > > > > >
> > > > > > Skype: rafael.novello
> > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > >
> > > > > >
> > > > > > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> > > > > > daniel.takabaya...@gmail.com> escreveu:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > &g

Re: POCs, Prototypes, experiments ...

[Rafael Novello]

Lucas, yes! It make sense!

In the end, CLI and REPL will share a big part of code and they will be
just different interfaces to the same features.

+1 for this new concept!

Thanks for make it clear to me!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em seg, 29 de out de 2018 às 17:43, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> That's a good point Rafael. I believe the main motivation for adding a REPL
> to Marvin's toolbox is to enhance user experience.
>
> Imo we can't just switch CLI per REPL. I don't think one excludes the
> other, both have different use cases.
>
> We may be mixing too much stuff together, but in general, I think we are:
>  - Refactoring toolbox architecture to avoid unnecessarily duplicated logic
> when adding support to new languages.
>  - Adding a new interface to enable users to open an interactive session
> with the engine (i.e. REPL)
>  - Making it easier to add support for new languages.
>
> Does it make sense for you?
>
> On Mon, Oct 29, 2018 at 11:22 AM Rafael Novello <
> rafa.reis.nove...@gmail.com>
> wrote:
>
> > I do not want to be the annoying conversation but...
> >
> > If we will have an unique CLI that will be able to handle any kind of
> > engines (in any language) by using docker, Why we need a REPL?
> >
> > I think we can focus on creating this new architecture using docker and
> > docker SDK and use only a CLI for that.
> >
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
> > lucasb...@apache.org> escreveu:
> >
> > > No, I think the idea is to do not create different versions of the CLI.
> > > Let's use one single tech stack for the CLI, ideally, it will be shared
> > > with the REPL. The way I see the CLI would be very similar to the REPL,
> > but
> > > instead of opening an interactive session the user could send
> > > unattended commands. As an analogy, you can think on spark-shell (our
> > REPL)
> > > and spark-submit (our CLI).
> > >
> > > - Lucas
> > >
> > > On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
> > > rafa.reis.nove...@gmail.com>
> > > wrote:
> > >
> > > > Hi Guys!
> > > >
> > > > Let me ask about this new version that include CLI's and REPL. Will
> we
> > > > create different versions of CLI?
> > > >
> > > > One of the reasons to create the REPL is to avoid re-implementing the
> > > same
> > > > features many times,
> > > > each time for a different language.
> > > >
> > > > Regards!
> > > > Atenciosamente,
> > > > Rafael J. R. Novello
> > > >
> > > > Skype: rafael.novello
> > > > Blog: http://rafanovello.blogspot.com.br/
> > > >
> > > >
> > > > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> > > > daniel.takabaya...@gmail.com> escreveu:
> > > >
> > > > > +1
> > > > >
> > > > > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> > > > > lucasb...@apache.org> escreveu:
> > > > >
> > > > > > Makes perfect sense to me. I believe that if the majority of
> people
> > > > > agrees
> > > > > > with this architecture we could create issues and start
> > implementing
> > > > it.
> > > > > >
> > > > > > +1
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > > > > > daniel.takabaya...@gmail.com> wrote:
> > > > > >
> > > > > > > I see... what about this new version?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > >
> > > > > > > I changed the toolbox ...from 1 component to 3... docker
> client,
> > > cli
> > > > > and
> > > > > > > repl. Both (cli and repl

Re: POCs, Prototypes, experiments ...

[Rafael Novello]

I do not want to be the annoying conversation but...

If we will have an unique CLI that will be able to handle any kind of
engines (in any language) by using docker, Why we need a REPL?

I think we can focus on creating this new architecture using docker and
docker SDK and use only a CLI for that.

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> No, I think the idea is to do not create different versions of the CLI.
> Let's use one single tech stack for the CLI, ideally, it will be shared
> with the REPL. The way I see the CLI would be very similar to the REPL, but
> instead of opening an interactive session the user could send
> unattended commands. As an analogy, you can think on spark-shell (our REPL)
> and spark-submit (our CLI).
>
> - Lucas
>
> On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
> rafa.reis.nove...@gmail.com>
> wrote:
>
> > Hi Guys!
> >
> > Let me ask about this new version that include CLI's and REPL. Will we
> > create different versions of CLI?
> >
> > One of the reasons to create the REPL is to avoid re-implementing the
> same
> > features many times,
> > each time for a different language.
> >
> > Regards!
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> > daniel.takabaya...@gmail.com> escreveu:
> >
> > > +1
> > >
> > > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> > > lucasb...@apache.org> escreveu:
> > >
> > > > Makes perfect sense to me. I believe that if the majority of people
> > > agrees
> > > > with this architecture we could create issues and start implementing
> > it.
> > > >
> > > > +1
> > > >
> > > >
> > > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > > > daniel.takabaya...@gmail.com> wrote:
> > > >
> > > > > I see... what about this new version?
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > >
> > > > > I changed the toolbox ...from 1 component to 3... docker client,
> cli
> > > and
> > > > > repl. Both (cli and repl) now are using the docker client to
> control
> > > the
> > > > > engines.
> > > > >
> > > > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > > > > lucasb...@apache.org> escreveu:
> > > > >
> > > > > > For sure, both suggestions are to make sure we're not killing the
> > CLI
> > > > and
> > > > > > that we have a component (the SDK client) that is shared by CLI
> and
> > > > REPL.
> > > > > >
> > > > > > Thanks for clarifying where the system calls will live.
> > > > > >
> > > > > > - Lucas
> > > > > >
> > > > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > > > daniel.takabaya...@gmail.com> wrote:
> > > > > >
> > > > > > > Hi Lucas, could you please justified your two suggestions?
> > > > > > >
> > > > > > > The general ideia is to use the Docker DSK to execute local
> > > commands,
> > > > > to
> > > > > > > consume outputs and also to execute Marvin "specific language"
> > > > > commands.
> > > > > > > Today almost all CLIs commands in Marvin is a mixed of systems
> > > > commands
> > > > > > and
> > > > > > > library method calls. The plan is to put all "specific language
> > > > calls"
> > > > > > > inside the marvin-common-libs and group the systems call on the
> > > > toolbox
> > > > > > > (using the docker SDK in this task).
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Em qui, 25 de o

Re: POCs, Prototypes, experiments ...

[Rafael Novello]

Hi Guys!

Let me ask about this new version that include CLI's and REPL. Will we
create different versions of CLI?

One of the reasons to create the REPL is to avoid re-implementing the same
features many times,
each time for a different language.

Regards!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> +1
>
> Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> lucasb...@apache.org> escreveu:
>
> > Makes perfect sense to me. I believe that if the majority of people
> agrees
> > with this architecture we could create issues and start implementing it.
> >
> > +1
> >
> >
> > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > daniel.takabaya...@gmail.com> wrote:
> >
> > > I see... what about this new version?
> > >
> > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > >
> > > I changed the toolbox ...from 1 component to 3... docker client, cli
> and
> > > repl. Both (cli and repl) now are using the docker client to control
> the
> > > engines.
> > >
> > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > > lucasb...@apache.org> escreveu:
> > >
> > > > For sure, both suggestions are to make sure we're not killing the CLI
> > and
> > > > that we have a component (the SDK client) that is shared by CLI and
> > REPL.
> > > >
> > > > Thanks for clarifying where the system calls will live.
> > > >
> > > > - Lucas
> > > >
> > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > daniel.takabaya...@gmail.com> wrote:
> > > >
> > > > > Hi Lucas, could you please justified your two suggestions?
> > > > >
> > > > > The general ideia is to use the Docker DSK to execute local
> commands,
> > > to
> > > > > consume outputs and also to execute Marvin "specific language"
> > > commands.
> > > > > Today almost all CLIs commands in Marvin is a mixed of systems
> > commands
> > > > and
> > > > > library method calls. The plan is to put all "specific language
> > calls"
> > > > > inside the marvin-common-libs and group the systems call on the
> > toolbox
> > > > > (using the docker SDK in this task).
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> > > > > lucasb...@apache.org> escreveu:
> > > > >
> > > > > > Great, thanks.
> > > > > >
> > > > > > So here some suggestions:
> > > > > >  1. We could create one new layer between Toolbox and the
> Engines,
> > > this
> > > > > > would be Docker client.
> > > > > >  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL
> > > > > >
> > > > > > It's not clear to me if the commands listed on General Commands
> > will
> > > > need
> > > > > > to be written for every language, from the image I believe not,
> but
> > > how
> > > > > it
> > > > > > would work, since starting a notebook in Python is different than
> > > > > starting
> > > > > > in Scala.
> > > > > >
> > > > > > - Lucas
> > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > > > daniel.takabaya...@gmail.com> wrote:
> > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > >
> > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > > > > rafa.reis.nove...@gmail.com> escreveu:
> > > > > > >
> > > > > > > > Hi Taka,
> > > > > > > >
> > > > > >

Re: POCs, Prototypes, experiments ...

[Rafael Novello]

Hi Taka,

I can't see the image! Maybe the Apache email list don't support
attachments.

Best!
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> Hi all,
>
> To try to give a little more details about this "new concept" that we want
> to bring to the marvin toolbox, I did this simple architecture draw.
>
> [image: marvin-architecture-views (5).png]
>
> The general idea here is try to transform the toolbox something
> disconnected with the "language", something more agnostic. Also in this new
> architecture we could use remote resource to process engines and make easy
> the support for new languages.
>
> This "new toolbox" will be the only thing that a Marvin user must to
> install and also we could start to support multiples O.S (once the REPL is
> a dummy application that only interprets and by pass commands).
>
> Regards,
> Taka
>
> Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> rafa.reis.nove...@gmail.com> escreveu:
>
>> Alan,
>>
>> Yes! We are using the Docker SDK and it's possible to use the API to
>> automate, but it's a little bit harder than automate CLI calls.
>>
>> Atenciosamente,
>> Rafael J. R. Novello
>>
>> Skype: rafael.novello
>> Blog: http://rafanovello.blogspot.com.br/
>>
>>
>> Em qua, 24 de out de 2018 às 12:02, Alan Silva 
>> escreveu:
>>
>> > Hi,
>> >
>> > One question here, I understand that we start to use with this PoC the
>> > Docker SDK API, right?
>> >
>> > Why not use the API to expose some endpoints to permit this kind of
>> > automation by devops?
>> >
>> > I think it is possible and it solves the CLI problem, right?
>> >
>> >
>> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
>> > rafa.reis.nove...@gmail.com>
>> > wrote:
>> >
>> > > Lucas,
>> > >
>> > > Sorry, I didn't understood your question bellow.
>> > >
>> > > "Would it make sense to use the same solution that we will use for
>> > having a
>> > > single-language REPL to have a single-language CLI?"
>> > >
>> > > For DevOps purposes, maybe this new toolbox concept is not ideal. I
>> think
>> > > we can keep the CLI inside the docker container but it will not easy
>> to
>> > > automate
>> > > jobs by this way.
>> > >
>> > > How to deal with this issue? Voting?
>> > > Atenciosamente,
>> > > Rafael J. R. Novello
>> > >
>> > > Skype: rafael.novello
>> > > Blog: http://rafanovello.blogspot.com.br/
>> > >
>> > >
>> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
>> > > lucasb...@apache.org> escreveu:
>> > >
>> > > > Got it! Thanks for clarifying.
>> > > >
>> > > > Would it make sense to use the same solution that we will use for
>> > having
>> > > a
>> > > > single-language REPL to have a single-language CLI? My only concern
>> > with
>> > > > killing the CLI is that you remove an essential feature for DevOps.
>> > > >
>> > > > - Lucas
>> > > >
>> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
>> > > > rafa.reis.nove...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Lucas,
>> > > > >
>> > > > > The idea is that REPL will substitute the actual CLI. It's because
>> > with
>> > > > the
>> > > > > actual concept (using language specific CLI) we will need to
>> > > re-implement
>> > > > > the same features many times and probably each CLI will have a
>> > > different
>> > > > > behavior because some language specific restrictions and/or
>> > > limitations.
>> > > > >
>> > > > > With this new concept, all users will have the same experience
>> > > > interacting
>> > > > > with Marvin REPL and they will use the bests tools to develop
>> their
>> > > > engines
>> > > > > (Jupyter Notebook and/or Lab with different languages support or
>> even
>> > > > > Apache Zeppelin). All the interact will

Re: Future support to R

[Rafael Novello]

Alan,

To be honest, I don't know. The new toolbox concept help us with some
problems
but it create others. I believe we need to vote if the new concept will be
the future
or not. Am I right?

Best
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qua, 24 de out de 2018 às 12:05, Alan Silva  escreveu:

> Hi,
>
> One question here, if I will start with development about this R support at
> our engine, I will make it based on that we have today or in this new
> concept?
>
> Because I understood the concepts change a lot, right?
>
>
>
> On Mon, Oct 15, 2018 at 1:56 PM Rafael Novello <
> rafa.reis.nove...@gmail.com>
> wrote:
>
> > Hi guys!
> >
> > Despite issues related to performance and scalability I believe we need
> > to support R engines to cover the most part of data science/machine
> > learning
> > projects. Sometimes the code performance isn't the main question and the
> > team performance (how fast the team can delivery some value) is more
> > important.
> >
> > We are working (doing some POCs and trials) on a new toolbox concept to
> > facilitate the inclusion of new languages support. We could discuss this
> in
> > more details but I believe this kind of detail is not the focus for this
> > thread.
> >
> > +1 for R support
> >
> > Best!
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em qui, 11 de out de 2018 às 23:01, Lucas Bonatto Miguel <
> > lucasbona...@gmail.com> escreveu:
> >
> > > The idea is to create the R toolbox, providing a similar user
> experience
> > > that we provide in Python today.
> > >
> > > Regards,
> > > Lucas
> > >
> > > On Thu, Oct 11, 2018 at 6:30 PM Alan Silva  > > .invalid>
> > > wrote:
> > >
> > > > Hi to all,
> > > >
> > > > Yes, R would be a language like Python that I believe we should have
> in
> > > our
> > > > toolset and even as I had already raised it, I'm also very interested
> > in
> > > > working on this part of making a frontend with R.
> > > >
> > > > However, am I understanding after your last mail that the idea here
> is
> > to
> > > > do something different from what happens with Python today? Was not
> > that
> > > > clear to me? Or would it make a frontend similar to what we have in
> > > Python
> > > > and in the future for Scala too, correct?
> > > >
> > > >
> > > > On Thu, Oct 11, 2018 at 8:21 PM Lucas Bonatto Miguel <
> > > lucasb...@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > Hi community, I already participated in several discussions with
> some
> > > of
> > > > > you regarding adding support to R on Marvin. Allowing users to
> write
> > > > > engines in R would be a great step for Marvin since that's the
> > primary
> > > > > language for many statisticians, physicists and math people.
> > > > >
> > > > > One of the points that always arise during those conversations is
> the
> > > > fact
> > > > > that R is not fast nor scalable, and it would be hard to either
> make
> > it
> > > > > process large chunks of data for training or make it support high
> > > > > throughput and have low latency for predictions. I want to start a
> > > thread
> > > > > on whether we should invest more time trying to add R support to
> > Marvin
> > > > or
> > > > > not. I invite you all to read about this related work:
> > > > > https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb
> > > > >
> > > > > I am looking forward to hearing opinions and ideas.
> > > > >
> > > > > Thanks,
> > > > > Lucas
> > > > >
> > > >
> > > >
> > > > --
> > > > *Alan Silva* | Solutions Architect
> > > > t. +55 (11) 99951- <00>5242 e. alan.si...@cloudera.com
> > > > cloudera.com <http://www.cloudera.com>
> > > >
> > > > [image: Cloudera] <http://www.cloudera.com/>
> > > >
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera
> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

[Rafael Novello]

Alan,

Yes! We are using the Docker SDK and it's possible to use the API to
automate, but it's a little bit harder than automate CLI calls.

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qua, 24 de out de 2018 às 12:02, Alan Silva  escreveu:

> Hi,
>
> One question here, I understand that we start to use with this PoC the
> Docker SDK API, right?
>
> Why not use the API to expose some endpoints to permit this kind of
> automation by devops?
>
> I think it is possible and it solves the CLI problem, right?
>
>
> On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> rafa.reis.nove...@gmail.com>
> wrote:
>
> > Lucas,
> >
> > Sorry, I didn't understood your question bellow.
> >
> > "Would it make sense to use the same solution that we will use for
> having a
> > single-language REPL to have a single-language CLI?"
> >
> > For DevOps purposes, maybe this new toolbox concept is not ideal. I think
> > we can keep the CLI inside the docker container but it will not easy to
> > automate
> > jobs by this way.
> >
> > How to deal with this issue? Voting?
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> > lucasb...@apache.org> escreveu:
> >
> > > Got it! Thanks for clarifying.
> > >
> > > Would it make sense to use the same solution that we will use for
> having
> > a
> > > single-language REPL to have a single-language CLI? My only concern
> with
> > > killing the CLI is that you remove an essential feature for DevOps.
> > >
> > > - Lucas
> > >
> > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > rafa.reis.nove...@gmail.com>
> > > wrote:
> > >
> > > > Lucas,
> > > >
> > > > The idea is that REPL will substitute the actual CLI. It's because
> with
> > > the
> > > > actual concept (using language specific CLI) we will need to
> > re-implement
> > > > the same features many times and probably each CLI will have a
> > different
> > > > behavior because some language specific restrictions and/or
> > limitations.
> > > >
> > > > With this new concept, all users will have the same experience
> > > interacting
> > > > with Marvin REPL and they will use the bests tools to develop their
> > > engines
> > > > (Jupyter Notebook and/or Lab with different languages support or even
> > > > Apache Zeppelin). All the interact will occur through REPL and docker
> > > > protocol.
> > > >
> > > > Alan, Yes! As Lucas said, the concept is the same but we can use
> docker
> > > to
> > > > do the same job.
> > > >
> > > > Best!
> > > >
> > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> > > > lucasb...@apache.org> escreveu:
> > > >
> > > > > Thanks for the clarifications Rafael, so from what I understood,
> > > Marvin's
> > > > > developers would not use the REPL to explore data and test models,
> > i.e.
> > > > > develop the engine. Is the idea to build something more like an
> > > > interactive
> > > > > CLI? I think an interactive CLI would be useful for the developer
> > > > > experience, however, it's important to keep the unattended
> (current)
> > > CLI
> > > > in
> > > > > place for scenarios where users want to automate Marvin tasks.
> > > > >
> > > > > Alan, I believe the idea is still the same as you started, but
> using
> > > > > docker SDK now.
> > > > >
> > > > > - Lucas
> > > > >
> > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > > > rafa.reis.nove...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Lucas!
> > > > > >
> > > > > > First of all +1 for REPL POCs to Apache!
> > > > > >
> > > > > > Let me help with some comments:
> > > > > >
> > > > > > 1 - We have tested NodeJS, Scala and Python and the easiest one
> was
> > > > > Python.
> > > > > 

Re: POCs, Prototypes, experiments ...

[Rafael Novello]

Lucas,

Sorry, I didn't understood your question bellow.

"Would it make sense to use the same solution that we will use for having a
single-language REPL to have a single-language CLI?"

For DevOps purposes, maybe this new toolbox concept is not ideal. I think
we can keep the CLI inside the docker container but it will not easy to
automate
jobs by this way.

How to deal with this issue? Voting?
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> Got it! Thanks for clarifying.
>
> Would it make sense to use the same solution that we will use for having a
> single-language REPL to have a single-language CLI? My only concern with
> killing the CLI is that you remove an essential feature for DevOps.
>
> - Lucas
>
> On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> rafa.reis.nove...@gmail.com>
> wrote:
>
> > Lucas,
> >
> > The idea is that REPL will substitute the actual CLI. It's because with
> the
> > actual concept (using language specific CLI) we will need to re-implement
> > the same features many times and probably each CLI will have a different
> > behavior because some language specific restrictions and/or limitations.
> >
> > With this new concept, all users will have the same experience
> interacting
> > with Marvin REPL and they will use the bests tools to develop their
> engines
> > (Jupyter Notebook and/or Lab with different languages support or even
> > Apache Zeppelin). All the interact will occur through REPL and docker
> > protocol.
> >
> > Alan, Yes! As Lucas said, the concept is the same but we can use docker
> to
> > do the same job.
> >
> > Best!
> >
> > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> > lucasb...@apache.org> escreveu:
> >
> > > Thanks for the clarifications Rafael, so from what I understood,
> Marvin's
> > > developers would not use the REPL to explore data and test models, i.e.
> > > develop the engine. Is the idea to build something more like an
> > interactive
> > > CLI? I think an interactive CLI would be useful for the developer
> > > experience, however, it's important to keep the unattended (current)
> CLI
> > in
> > > place for scenarios where users want to automate Marvin tasks.
> > >
> > > Alan, I believe the idea is still the same as you started, but using
> > > docker SDK now.
> > >
> > > - Lucas
> > >
> > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > rafa.reis.nove...@gmail.com>
> > > wrote:
> > >
> > > > Hi Lucas!
> > > >
> > > > First of all +1 for REPL POCs to Apache!
> > > >
> > > > Let me help with some comments:
> > > >
> > > > 1 - We have tested NodeJS, Scala and Python and the easiest one was
> > > Python.
> > > > We have found a small project [1] that have all features we desired
> for
> > > > REPL:
> > > > - Autocomplete
> > > > - Python commands disabled (the user have only the commands we
> > provide).
> > > > The other languages REPL options don't have this feature.
> > > > - Easy to show output
> > > > - Etc
> > > > So, I think the language chosen will not be important here because
> the
> > > user
> > > > will only interact with the commands that we create.
> > > >
> > > > 2 - The "engine-generate" command will download a docker image that
> we
> > > > create for that language and start a container with the basic project
> > > > structure for that language.
> > > >
> > > > 3 - The REPL client will use the "docker protocol" to run command,
> > > > start/stop services and etc inside the container and it will receive
> > the
> > > > log stream to show. No, the REPL will no pass code snippets for
> docker
> > > > container (I think it will not be necessary)
> > > >
> > > > 4 - Yep! Like I said on the first item.
> > > >
> > > > [1] - https://github.com/italorossi/ishell
> > > >
> > > > Let me know if there is any other question!
> > > > Best!
> > > > Rafael J. R. Novello
> > > >
> > > > Skype: rafael.novello
> > > > Blog: http://rafanovello.blogspot.com.br/
> > > >
> > > >
> &g

Re: POCs, Prototypes, experiments ...

[Rafael Novello]

Lucas,

The idea is that REPL will substitute the actual CLI. It's because with the
actual concept (using language specific CLI) we will need to re-implement
the same features many times and probably each CLI will have a different
behavior because some language specific restrictions and/or limitations.

With this new concept, all users will have the same experience interacting
with Marvin REPL and they will use the bests tools to develop their engines
(Jupyter Notebook and/or Lab with different languages support or even
Apache Zeppelin). All the interact will occur through REPL and docker
protocol.

Alan, Yes! As Lucas said, the concept is the same but we can use docker to
do the same job.

Best!

Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> Thanks for the clarifications Rafael, so from what I understood, Marvin's
> developers would not use the REPL to explore data and test models, i.e.
> develop the engine. Is the idea to build something more like an interactive
> CLI? I think an interactive CLI would be useful for the developer
> experience, however, it's important to keep the unattended (current) CLI in
> place for scenarios where users want to automate Marvin tasks.
>
> Alan, I believe the idea is still the same as you started, but using
> docker SDK now.
>
> - Lucas
>
> On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> rafa.reis.nove...@gmail.com>
> wrote:
>
> > Hi Lucas!
> >
> > First of all +1 for REPL POCs to Apache!
> >
> > Let me help with some comments:
> >
> > 1 - We have tested NodeJS, Scala and Python and the easiest one was
> Python.
> > We have found a small project [1] that have all features we desired for
> > REPL:
> > - Autocomplete
> > - Python commands disabled (the user have only the commands we provide).
> > The other languages REPL options don't have this feature.
> > - Easy to show output
> > - Etc
> > So, I think the language chosen will not be important here because the
> user
> > will only interact with the commands that we create.
> >
> > 2 - The "engine-generate" command will download a docker image that we
> > create for that language and start a container with the basic project
> > structure for that language.
> >
> > 3 - The REPL client will use the "docker protocol" to run command,
> > start/stop services and etc inside the container and it will receive the
> > log stream to show. No, the REPL will no pass code snippets for docker
> > container (I think it will not be necessary)
> >
> > 4 - Yep! Like I said on the first item.
> >
> > [1] - https://github.com/italorossi/ishell
> >
> > Let me know if there is any other question!
> > Best!
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> > lucasb...@apache.org> escreveu:
> >
> > > + 1 for migrating the REPL repo to Apache
> > >
> > > I have a few questions about the previous explanation:
> > >  1) The REPL itself, it would be an application in which language?
> > Remember
> > > that the main idea is to allow the user to program on his preferred
> > > language in the REPL.
> > >  2) Should the engine-generate command also generate a docker image
> with
> > > the user's application?
> > >  3) What type of communication would happen between the REPL and the
> > engine
> > > via Docker SDK? Would the REPL pass snippets of code to be executed by
> > the
> > > docker container?
> > >  4) Have you considered code completion in the REPL?
> > >
> > >
> > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei 
> > > wrote:
> > >
> > > > Ok guys,
> > > >
> > > > The basic idea is to provide only one Toolbox for multiple languages.
> > > > We are looking for possibility to build a single Marvin Repl,
> > > > instead of severals toolboxes with differentes interfaces or
> commands.
> > > >
> > > > In this case, the engine-generate command will download and start a
> > > Docker
> > > > container with basic engine structure corresponding the choosed
> > language.
> > > > this means we don't need to build Toolboxes of differents languages,
> we
> > > > just
> > > > build the engine template of all languages that we want to support
> and
> > > > provide it
> > > &g

Re: POCs, Prototypes, experiments ...

[Rafael Novello]

Hi Lucas!

First of all +1 for REPL POCs to Apache!

Let me help with some comments:

1 - We have tested NodeJS, Scala and Python and the easiest one was Python.
We have found a small project [1] that have all features we desired for
REPL:
- Autocomplete
- Python commands disabled (the user have only the commands we provide).
The other languages REPL options don't have this feature.
- Easy to show output
- Etc
So, I think the language chosen will not be important here because the user
will only interact with the commands that we create.

2 - The "engine-generate" command will download a docker image that we
create for that language and start a container with the basic project
structure for that language.

3 - The REPL client will use the "docker protocol" to run command,
start/stop services and etc inside the container and it will receive the
log stream to show. No, the REPL will no pass code snippets for docker
container (I think it will not be necessary)

4 - Yep! Like I said on the first item.

[1] - https://github.com/italorossi/ishell

Let me know if there is any other question!
Best!
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
lucasb...@apache.org> escreveu:

> + 1 for migrating the REPL repo to Apache
>
> I have a few questions about the previous explanation:
>  1) The REPL itself, it would be an application in which language? Remember
> that the main idea is to allow the user to program on his preferred
> language in the REPL.
>  2) Should the engine-generate command also generate a docker image with
> the user's application?
>  3) What type of communication would happen between the REPL and the engine
> via Docker SDK? Would the REPL pass snippets of code to be executed by the
> docker container?
>  4) Have you considered code completion in the REPL?
>
>
> On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei 
> wrote:
>
> > Ok guys,
> >
> > The basic idea is to provide only one Toolbox for multiple languages.
> > We are looking for possibility to build a single Marvin Repl,
> > instead of severals toolboxes with differentes interfaces or commands.
> >
> > In this case, the engine-generate command will download and start a
> Docker
> > container with basic engine structure corresponding the choosed language.
> > this means we don't need to build Toolboxes of differents languages, we
> > just
> > build the engine template of all languages that we want to support and
> > provide it
> > as Docker containers
> >
> > We have started researches around our basic requirements like:
> > - Repl interface
> > - System communication
> > - Connection security
> > - Tool popularity
> > - Update complexity
> > - Languages support
> > - ..
> >
> > And we did some POC with code here:
> > https://github.com/marvin-ai/marvin-repl
> >
> > There is POC testing gRPC using Scala and Python,
> > Repl inteface and Docker SDK with NodeJS,
> > Repl interface and Docker SDK with Python.
> >
> > At this moment we prefer the Repl interface + Docker SDK way, because
> good
> > part of the requirements
> > will be guaranteed by Docker.
> >
> > With this informations, what do you think? Should we submit all this POCs
> > to Apache Repo?
> > Please feel free to opine.
> >
> >
> > Thats all, thanks!!!
> >
> >
> > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > daniel.takabaya...@gmail.com> escreveu:
> >
> > > Zhang,
> > >
> > > I think the best approach is give us a better explanation about this
> new
> > > feature and how this can help us to archive the desired feature
> (support
> > > multiple languages). Than if the majority agree than I can do the
> merge,
> > > just like I did before, but in another branch.
> > >
> > > Let me know if makes sense, ok?
> > >
> > > Taka
> > >
> > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > luckbr1...@gmail.com>
> > > escreveu:
> > >
> > > > So, what is the POC, is it a refactoring of the existing repo? Or is
> > > > it a new rewrite of the repo?
> > > >
> > > > Just asking as it might make sense to make it a branch then actually
> a
> > > > thing in parallel, as this will have an effect for releases, etc. but
> > > > you guys know more here than I do.
> > > >
> > > > Also, it's probably good to have a write up of the main direction of
> > > > the design that can help people get familiar with the new approach.
> > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei  >
> > > > wrote:
> > > > >
> > > > > Hey guys,
> > > > >
> > > > > We have reorganized the Poc repo, and want to merge it to our
> Apache
> > > > repo.
> > > > > Just making sure here before we do the merge, because im not sure
> if
> > > > doing
> > > > > this will perserve the Git commit history.
> > > > > What we are planning to do is:
> > > > >
> > > > > git filter-branch --subdirectory-filter  > > directory>
> > > > > -- --all
> > > > > mkdir POCs/
> > > > > git mv * POCs/
> > > > > git commit -m "colleted the

Move users to apache mail list

[Rafael Novello]

Hi Guys!

We have some users on a google group e-mail list that is used to ask
questions and share tips, generally in portuguese. My question is should we
"move" Marvin users to an apache mail list? Which list?

Thanks a lot!
Best!
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/

Re: Future support to R

[Rafael Novello]

Hi guys!

Despite issues related to performance and scalability I believe we need
to support R engines to cover the most part of data science/machine learning
projects. Sometimes the code performance isn't the main question and the
team performance (how fast the team can delivery some value) is more
important.

We are working (doing some POCs and trials) on a new toolbox concept to
facilitate the inclusion of new languages support. We could discuss this in
more details but I believe this kind of detail is not the focus for this
thread.

+1 for R support

Best!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qui, 11 de out de 2018 às 23:01, Lucas Bonatto Miguel <
lucasbona...@gmail.com> escreveu:

> The idea is to create the R toolbox, providing a similar user experience
> that we provide in Python today.
>
> Regards,
> Lucas
>
> On Thu, Oct 11, 2018 at 6:30 PM Alan Silva  .invalid>
> wrote:
>
> > Hi to all,
> >
> > Yes, R would be a language like Python that I believe we should have in
> our
> > toolset and even as I had already raised it, I'm also very interested in
> > working on this part of making a frontend with R.
> >
> > However, am I understanding after your last mail that the idea here is to
> > do something different from what happens with Python today? Was not that
> > clear to me? Or would it make a frontend similar to what we have in
> Python
> > and in the future for Scala too, correct?
> >
> >
> > On Thu, Oct 11, 2018 at 8:21 PM Lucas Bonatto Miguel <
> lucasb...@apache.org
> > >
> > wrote:
> >
> > > Hi community, I already participated in several discussions with some
> of
> > > you regarding adding support to R on Marvin. Allowing users to write
> > > engines in R would be a great step for Marvin since that's the primary
> > > language for many statisticians, physicists and math people.
> > >
> > > One of the points that always arise during those conversations is the
> > fact
> > > that R is not fast nor scalable, and it would be hard to either make it
> > > process large chunks of data for training or make it support high
> > > throughput and have low latency for predictions. I want to start a
> thread
> > > on whether we should invest more time trying to add R support to Marvin
> > or
> > > not. I invite you all to read about this related work:
> > > https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb
> > >
> > > I am looking forward to hearing opinions and ideas.
> > >
> > > Thanks,
> > > Lucas
> > >
> >
> >
> > --
> > *Alan Silva* | Solutions Architect
> > t. +55 (11) 99951- <00>5242 e. alan.si...@cloudera.com
> > cloudera.com 
> >
> > [image: Cloudera] 
> >
> > [image: Cloudera on Twitter]  [image:
> > Cloudera on Facebook]  [image:
> Cloudera
> > on LinkedIn] 
> >
>

Re: Importing Source Code

[Rafael Novello]

To me it look like perfect. What is the next step? Could I help?

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em ter, 9 de out de 2018 às 14:08, Daniel Takabayashi <
daniel.takabaya...@gmail.com> escreveu:

> Hello Guys, anyone has something to say?
>
> Sent from my iPhone
>
> > On Oct 4, 2018, at 7:40 AM, Rafael Novello 
> wrote:
> >
> > +1
> >
> > All the history was preserved! I think we are fine!
> >
> > Best!
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em qui, 4 de out de 2018 às 06:50, Lucas Bonatto Miguel <
> > lucasbona...@gmail.com> escreveu:
> >
> >> +1
> >>
> >> Looks good to me. We should prioritize better READMEs in the near
> future.
> >> For the scope of this story I think you did a good job.
> >>
> >> On Wed, Oct 3, 2018 at 12:05 PM Daniel Takabayashi <
> >> daniel.takabaya...@gmail.com> wrote:
> >>
> >>> Ok guys, I finished the merge of our 3 main repos:
> >>>
> >>>
> >>> *https://github.com/marvin-ai/marvin-python-toolbox
> >>> <https://github.com/marvin-ai/marvin-python-toolbox>*
> >>>
> >>> *https://github.com/marvin-ai/marvin-public-engines
> >>> <https://github.com/marvin-ai/marvin-public-engines>*
> >>> *https://github.com/marvin-ai/marvin-engine-executor
> >>> <https://github.com/marvin-ai/marvin-engine-executor>*
> >>>
> >>> Into this one ...
> >>>
> >>> *https://github.com/marvin-ai/incubator-apache-marvin
> >>> <https://github.com/marvin-ai/incubator-apache-marvin>*
> >>>
> >>> All commits history remained and ... please check and feel free to
> >>> collaborate before we replicate into our apache account. And any
> problem
> >>> ...let me know!
> >>>
> >>> Thanks,
> >>> Taka
> >>>
> >>> Em ter, 2 de out de 2018 às 11:48, Daniel Takabayashi <
> >>> daniel.takabaya...@gmail.com> escreveu:
> >>>
> >>>> Sounds good.
> >>>>
> >>>> Em ter, 2 de out de 2018 às 11:44, Luciano Resende <
> >> luckbr1...@gmail.com
> >>>>
> >>>> escreveu:
> >>>>
> >>>>> So, before pushing to the apache repo, make a draft repo outside
> >> apache
> >>>>> and let couple people to review
> >>>>>
> >>>>> The  you can push to apache... as a mirror.
> >>>>>
> >>>>> On Tue, Oct 2, 2018 at 12:07 Daniel Takabayashi <
> >>>>> daniel.takabaya...@gmail.com> wrote:
> >>>>>
> >>>>>> Ok, now is clear
> >>>>>>
> >>>>>> This is the repo
> >>>>>> <https://github.com/marvin-ai/incubator-apache-marvin.git> I am
> >> going
> >>>>>> to push the new merged code. Another question is about the license,
> >>> right
> >>>>>> now all files has this license here:
> >>>>>>
> >>>>>> # Copyright [2017] [B2W Digital]
> >>>>>> #
> >>>>>> # Licensed under the Apache License, Version 2.0 (the "License");
> >>>>>> # you may not use this file except in compliance with the License.
> >>>>>> # You may obtain a copy of the License at
> >>>>>> #
> >>>>>> #http://www.apache.org/licenses/LICENSE-2.0
> >>>>>> #
> >>>>>> # Unless required by applicable law or agreed to in writing,
> software
> >>>>>> # distributed under the License is distributed on an "AS IS" BASIS,
> >>>>>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> >>>>>> implied.
> >>>>>> # See the License for the specific language governing permissions
> and
> >>>>>> # limitations under the License.
> >>>>>>
> >>>>>> should we change this before the import commit or later is ok?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Taka
> >>>>>>
> >>>>>>
> >>>>>> Em se

Re: Importing Source Code

[Rafael Novello]

+1

All the history was preserved! I think we are fine!

Best!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qui, 4 de out de 2018 às 06:50, Lucas Bonatto Miguel <
lucasbona...@gmail.com> escreveu:

> +1
>
> Looks good to me. We should prioritize better READMEs in the near future.
> For the scope of this story I think you did a good job.
>
> On Wed, Oct 3, 2018 at 12:05 PM Daniel Takabayashi <
> daniel.takabaya...@gmail.com> wrote:
>
> > Ok guys, I finished the merge of our 3 main repos:
> >
> >
> > *https://github.com/marvin-ai/marvin-python-toolbox
> > *
> >
> > *https://github.com/marvin-ai/marvin-public-engines
> > *
> > *https://github.com/marvin-ai/marvin-engine-executor
> > *
> >
> > Into this one ...
> >
> > *https://github.com/marvin-ai/incubator-apache-marvin
> > *
> >
> > All commits history remained and ... please check and feel free to
> > collaborate before we replicate into our apache account. And any problem
> > ...let me know!
> >
> > Thanks,
> > Taka
> >
> > Em ter, 2 de out de 2018 às 11:48, Daniel Takabayashi <
> > daniel.takabaya...@gmail.com> escreveu:
> >
> > > Sounds good.
> > >
> > > Em ter, 2 de out de 2018 às 11:44, Luciano Resende <
> luckbr1...@gmail.com
> > >
> > > escreveu:
> > >
> > >> So, before pushing to the apache repo, make a draft repo outside
> apache
> > >> and let couple people to review
> > >>
> > >> The  you can push to apache... as a mirror.
> > >>
> > >> On Tue, Oct 2, 2018 at 12:07 Daniel Takabayashi <
> > >> daniel.takabaya...@gmail.com> wrote:
> > >>
> > >>> Ok, now is clear
> > >>>
> > >>> This is the repo
> > >>>  I am
> going
> > >>> to push the new merged code. Another question is about the license,
> > right
> > >>> now all files has this license here:
> > >>>
> > >>> # Copyright [2017] [B2W Digital]
> > >>> #
> > >>> # Licensed under the Apache License, Version 2.0 (the "License");
> > >>> # you may not use this file except in compliance with the License.
> > >>> # You may obtain a copy of the License at
> > >>> #
> > >>> #http://www.apache.org/licenses/LICENSE-2.0
> > >>> #
> > >>> # Unless required by applicable law or agreed to in writing, software
> > >>> # distributed under the License is distributed on an "AS IS" BASIS,
> > >>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > >>> implied.
> > >>> # See the License for the specific language governing permissions and
> > >>> # limitations under the License.
> > >>>
> > >>> should we change this before the import commit or later is ok?
> > >>>
> > >>> Thanks,
> > >>> Taka
> > >>>
> > >>>
> > >>> Em seg, 1 de out de 2018 às 19:24, Luciano Resende <
> > luckbr1...@gmail.com>
> > >>> escreveu:
> > >>>
> >  This is a little bit different because it's the first code import.
> > 
> >  As a suggestion, how about you create a repository somewhere in
> GitHub
> >  when you are ready to import, let the community do a quick review,
> and
> >  we can then import it as a replica.
> >  On Mon, Oct 1, 2018 at 7:06 PM Daniel Takabayashi
> >   wrote:
> >  >
> >  > Hey guys, I will try to finish this by today. But then I need to
> > open
> >  any PR or what??
> >  >
> >  > As is my first time as Apache contributor I’m a little lost about
> > the
> >  correct procedure. Could you clarify this Luciano?
> >  >
> >  > Sent from my iPhone
> >  >
> >  > > On Oct 1, 2018, at 6:50 PM, Lucas Bonatto Miguel <
> >  lucasbona...@gmail.com> wrote:
> >  > >
> >  > > Hey Luciano, Daniel and I started working on it this weekend, it
> > is
> >  being a
> >  > > bit more tricky then we thought. We should have the code pushed
> by
> >  the end
> >  > > of this week.
> >  > >
> >  > > - Lucas
> >  > >
> >  > >> On Thu, 27 Sep 2018 at 2:04 PM Luciano Resende <
> >  luckbr1...@gmail.com> wrote:
> >  > >>
> >  > >> Any progress here? Any help needed?
> >  > >>
> >  > >> Note that the software grant process is not a blocker on having
> > the
> >  > >> code imported and these can occur in parallel.
> >  > >> On Thu, Sep 20, 2018 at 9:47 PM Daniel Takabayashi
> >  > >>  wrote:
> >  > >>>
> >  > >>> Yes...the ideia is to merge all repos but keeping the history
> of
> >  all
> >  > >>> commits.
> >  > >>> I did this once in the past and is pretty easy and reliable.
> >  > >>>
> >  > >>> About the grant any information or request to B2W, I can
> > arrange.
> >  Just
> >  > >> let
> >  > >>> me know @Lucas Bonatto Miguel  .
> >  > >>>
> >  > >>> Em qui, 20 de set de 2018 às 18:43, Lucas Bonatto Miguel <

Re: Welcome to Marvin podling

[Rafael Novello]

Hi Luciano,

This is really cool!

Do we need invitations to access the slack?

Thanks for all the help!

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/

2018-09-20 15:40 GMT-03:00 Luciano Resende :

> Hi All,
>
> The basic infrastructure for the podling has been created:
>
> Git (empty repository):
>https://github.com/apache/incubator-marvin
>
> Mailing Lists:
>dev, commits, reviews (@marvin.incubator.apache.org)
>
> Note that for mailing lists subscriptions, there is a helper tool at
>https://whimsy.apache.org/committers/subscribe
>
> I have also created a slack channel: marvin-ai at the-asf.slack.com
>
> Now, let's start migrating the existing code base, and other artifacts
> to the Apache infrastructure.
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>