Re: Hazelcast Jet Runner

I believe that is the case. Thanks Kenn. On 10.07.19 21:35, Ismaël Mejía wrote: > Yes please! > > On Wed, Jul 10, 2019 at 8:38 PM Kenneth Knowles wrote: > > > > Just to make sure we have closed on the Jet runner, my understanding is: I > > was the main person asking for

Re: Hazelcast Jet Runner

Yes please! On Wed, Jul 10, 2019 at 8:38 PM Kenneth Knowles wrote: > > Just to make sure we have closed on the Jet runner, my understanding is: I > was the main person asking for "runners-jet-experimental" but I am convinced > to go with plain "runners-jet". It seems everyone else is already

Re: Hazelcast Jet Runner

Just to make sure we have closed on the Jet runner, my understanding is: I was the main person asking for "runners-jet-experimental" but I am convinced to go with plain "runners-jet". It seems everyone else is already fine with this, so go ahead? On Tue, Jul 9, 2019 at 1:23 PM Maximilian Michels

Re: Hazelcast Jet Runner

We should fork the discussion around removing instances of @Experimental, but it was good to mention it here. As for the Jet runner, I can only second Ismael: The Jet runner is the first runner I can think of that came with ValidatesRunner and Nexmark out of the box. Of course that doesn't

Re: Hazelcast Jet Runner

On Wed, Jun 12, 2019 at 2:32 AM Ismaël Mejía wrote: > Seems the discussion moved a bit of my original intent that was to > make the Jet runner directory to be just called runners/jet in the > directory and mark the 'experimental' part of it in documentation as > we do for all other things in

Re: Hazelcast Jet Runner

Seems the discussion moved a bit of my original intent that was to make the Jet runner directory to be just called runners/jet in the directory and mark the 'experimental' part of it in documentation as we do for all other things in Beam. Can we do this or is there still any considerable argument

Re: Hazelcast Jet Runner

Hi, Over 800 usages under java, might be worth doing a few PR... Also suggest we use a very light review process: First round go for low hanging fruit, if anyone does a -1 against a change then we leave that for round two. Thoughts? Cheers Reza On Wed, 29 May 2019 at 12:05, Kenneth Knowles

Re: Hazelcast Jet Runner

On Mon, May 27, 2019 at 4:05 PM Reza Rokni wrote: > "Many APIs that have been in place for years and are used by most Beam > users are still marked Experimental." > > Should there be a formal process in place to start 'graduating' features > out of @Experimental? Perhaps even target an up coming

Re: Hazelcast Jet Runner

On Mon, May 27, 2019 at 3:44 PM Reuven Lax wrote: > We generally use Experimental for two different things, which leads to > confusion. > 1. Features that work stably, but where we think we might still make > some changes to the API. > 2. New features that we think might not yet be stable. >

Re: Hazelcast Jet Runner

"Many APIs that have been in place for years and are used by most Beam users are still marked Experimental." Should there be a formal process in place to start 'graduating' features out of @Experimental? Perhaps even target an up coming release with a PR to remove the annotation from well

Re: Hazelcast Jet Runner

We generally use Experimental for two different things, which leads to confusion. 1. Features that work stably, but where we think we might still make some changes to the API. 2. New features that we think might not yet be stable. This dual usage leads to a lot of confusion IMO. The fact that

Re: Hazelcast Jet Runner

> Personally, I think that it is good that moving from experimental to > non-experimental is a breaking change in the dependency - one has > backwards-incompatible changes and the other does not. If artifacts had > separate versioning we could use 0.x for this. In theory it seems so, but in

Re: Hazelcast Jet Runner

(minor related tangent for additional perspective) +1 from the perspective of SDKs on moving from experimental to production versions being a breaking change. I've long posited that the Go SDK, as it's currently experimental, is v0.X, and some breaking changes have been made accordingly. Once the

Re: Hazelcast Jet Runner

Personally, I think that it is good that moving from experimental to non-experimental is a breaking change in the dependency - one has backwards-incompatible changes and the other does not. If artifacts had separate versioning we could use 0.x for this. But biggest motivation for me are these:

Re: Hazelcast Jet Runner

Including the experimental suffix in artifact names is not a good idea either because once we decide that it is not experimantal anymore this will be a breaking change for users who will need then to update its dependencies code. Also it is error-prone to use different mappings for directories and

Re: Hazelcast Jet Runner

I missed Ken's input when writing my previous mail. Sorry. So, to recap: I should remove "experimental" from any directory names, but find an other way of configuring the artifact so that it still has "experimental" in it's name. Right? On Sat, May 25, 2019 at 9:32 AM Jozsef Bartok wrote: >

Re: Hazelcast Jet Runner

Yes, I'll gladly fix it, we aren't particularly keen to be labeled as experimental either.. Btw. initially the "experimental" word was only in the Gradle module name, but then there was some change ([BEAM-4046] decouple gradle project names and maven artifact ids - 4/2/19) which kind of ended up

Re: Hazelcast Jet Runner

My request was that the artifact be beam-runners-jet-experimental or beam-runners-experimental-jet so that a user was clearly opting in to experimental functionality, per the discussion. I try not to have a strong opinion about the mechanism. Probably the most natural thing to do is just configure

Re: Hazelcast Jet Runner

I see thanks Jozsef, marking things as Experimental was discussed but we never agreed on doing this at the directory level. We can cover the same ground by putting an annotation in the classes (in particular the JetRunner and JetPipelineOptions classes which are the real public interface, or in

Re: Hazelcast Jet Runner

Hi Ismaël! Quoting Kenn (from PR-8410 ): "We discussed on list that it would be better to have new things always start as experimental in a way that clearly distinguishes them from the core." Rgds On Thu, May 23, 2019 at 10:44 PM Ismaël Mejía wrote: >

Re: Hazelcast Jet Runner

I saw that the runner was merged but I don’t get why the foler is called ‘runners/jet experimental’ and not simply ‘runners/jet’. Is it because the runner does not pass ValidatesRunner? Or because the contributors are few? I don’t really see any reason behind this suffix. And even if the status is

Re: Hazelcast Jet Runner

Nice! That is *way* more than the PR I was looking for. I just meant that you could update the website/ directory. It is fine to keep the runner in your own repository if you want. But I think it is great if you want to contribute it to Apache Beam (hence donate it to the Apache Software

Re: Hazelcast Jet Runner

Hi Jozsef, If the Runner support the complete set of ValidatesRunner tests and the Nexmark suite, it is already in a very good state. Like Kenn already suggested, we can definitely add it to the capability matrix then. Thanks, Max On 19.04.19 22:52, Kenneth Knowles wrote: The

Re: Hazelcast Jet Runner

The ValidatesRunner tests are the best source we have for knowing the capabilities of a runner. Are there instructions for running the tests? Assuming we can check it out, then just open a PR to the website with the current capabilities and caveats. Since it is a big deal and could use lots of

Re: Hazelcast Jet Runner - validation tests

Robert - that appears to be a test of state & timers, not triggers. Should work for testing that the watermark at least advances. We do already have similar java-based ValidatesRunner tests in ParDoTest. The results of triggering, while nondeterministic, should generally fall into a testable

Re: Hazelcast Jet Runner - validation tests

On Thu, Apr 4, 2019 at 6:38 PM Lukasz Cwik wrote: > > The issue with unbounded tests that rely on triggers/late data/early > firings/processing time is that these are several sources of non-determinism. > The sources make non-deterministic decisions around when to produce data, > checkpoint,

Re: Hazelcast Jet Runner - validation tests

The issue with unbounded tests that rely on triggers/late data/early firings/processing time is that these are several sources of non-determinism. The sources make non-deterministic decisions around when to produce data, checkpoint, and resume and runners make non-deterministic decisions around

Re: Hazelcast Jet Runner

Hi Can, Like GreedyPipelineFuser, we have added many more components which makes building a Portable Runner easy. Here is a link [1] to slides which explains at a very high level what is needed to add a new portable runner. Still adding a portable runner will be more complex than adding a native

Re: Hazelcast Jet Runner

Documentation on portability is still a bit sparse although there are many design documents: https://beam.apache.org/contribute/design-documents/#portability The structure of portable Runners is not fundamentally different, but some of the operations are deferred to the SDK which runs code

Re: Hazelcast Jet Runner

I had a look at "GreedyPipelineFuser" and indeed this was what exactly I was talking about. Is https://beam.apache.org/roadmap/portability/ still the best information about the portable runners or is there a more in-depth guide available anywhere? On Wed, Mar 20, 2019 at 2:29 PM Can Gencer

Re: Hazelcast Jet Runner

Hi Max, Thanks. When you mean "old-style runner" is this meant that this style of runners will be obsolete and only the portable one will be supported? The documentation for portable runners wasn't quite complete and the barrier to entry for writing an old style runner seemed easier for us and

Re: Hazelcast Jet Runner

Hi Can, Thanks for the update. Interesting question. Flink has an optimization built in called chaining which works together nicely with Beam. Essentially, operators which share the same partitioning get executed one after another inside a master operator. This saves resources.

Re: Hazelcast Jet Runner

Elaborating on what Robert alluded to: when I wrote that runner author guide, portability was in its infancy. Now Beam Python can be run on Flink. So that guide is primarily focused on the "deserialize a Java DoFn and call its methods" approach. A decent amount of it is still really important to

Re: Hazelcast Jet Runner

On Fri, Feb 15, 2019 at 7:36 AM Can Gencer wrote: > > We at Hazelcast are looking into writing a Beam runner for Hazelcast Jet > (https://github.com/hazelcast/hazelcast-jet). I wanted to introduce myself as > we'll likely have questions as we start development. Welcome! Hazelcast looks