Re: Graal instead of docker?

Portability, at its core, is providing a spec for any runner to talk to any SDK. I personally think it's done a great job in cleaning up the model by forcing us to define a clean boundary (as specified at https://github.com/apache/beam/tree/master/model ) between these two components (even if the i

Re: Graal instead of docker?

Not sure what you mean? Can you point to a piece of code in Beam that you're currently characterizing as "hacking" and suggest how it could be refactored? On Sat, May 5, 2018 at 2:06 PM Romain Manni-Bucau wrote: > All are good points. > > The only "?" I keep is: why beam doesnt uses its visitor

Re: Graal instead of docker?

All are good points. The only "?" I keep is: why beam doesnt uses its visitor api to make the portability transversal to all runners "mutating" the user model before translation? Technically it sounds easy and avoid hacking all impl. Was it tested and failed? Le 5 mai 2018 22:50, "Thomas Weise"

Re: Graal instead of docker?

Docker isn't a silver bullet and may not be the best choice for all environments (I'm also looking at potentially launching SDK workers in a different way), but AFAIK there has not been any alternative proposal for default SDK execution that can handle all of Python, Go and Java. Regardless of the

Re: Graal instead of docker?

To add on that: Romain, if you are really excited about Graal as a project, here are some constructive suggestions as to what you can do on a reasonably short timeframe: - Propose/prototype a design for writing UDFs in Beam SQL using Graal - Go through the portability-related design documents, come

Re: Graal instead of docker?

Graal is a very young project, currently nowhere near the level of maturity or completeness as to be sufficient for Beam to fully bet its portability vision on it: - Graal currently only claims to support Java and Javascript, with Ruby and R in the status of "some applications may run", Python supp

Re: Graal instead of docker?

Agree The jvm is still mainstream for big data and it is trivial to have a remote facade to support natives but no point to have it in runners, it is some particular transforms or even dofn and sources only... Le 5 mai 2018 19:03, "Andrew Pilloud" a écrit : > Thanks for the examples earlier, I

Re: Graal instead of docker?

Thanks for the examples earlier, I think Hazelcast is a great example of something portability might make more difficult. I'm not working on portability, but my understanding is that the data sent to the runner is a blob of code and the name of the container to run it in. A runner with a native lan

Re: Graal instead of docker?

2018-05-05 9:27 GMT+02:00 Ismaël Mejía : > Graal would not be a viable solution for the reasons Henning and Andrew > mentioned, or put in other words, when users choose a programming language > they don’t choose only a ‘friendly’ syntax or programming model, they > choose also the ecosystem that c

Re: Graal instead of docker?

Graal would not be a viable solution for the reasons Henning and Andrew mentioned, or put in other words, when users choose a programming language they don’t choose only a ‘friendly’ syntax or programming model, they choose also the ecosystem that comes with it, and the libraries that make their li

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #29

See Changes: [mariagh] Update CoGroupByKey docstring [swegner] Remove Gradle worker cap. [swegner] Re-enable additional Jenkins worker parallelism for Dataflow [swegner] Adjust Jenkins Gradle

Re: Graal instead of docker?

A beam cluster with the spark runner would include a spark cluster, plus what's needed for portability, plus the beam sdk. On Fri, May 4, 2018, 11:55 PM Romain Manni-Bucau wrote: > > > Le 5 mai 2018 08:43, "Reuven Lax" a écrit : > > I don't believe we enforce docker anywhere. In fact if someone