Just to make sure we have closed on the Jet runner, my understanding is: I was the main person asking for "runners-jet-experimental" but I am convinced to go with plain "runners-jet". It seems everyone else is already fine with this, so go ahead?
On Tue, Jul 9, 2019 at 1:23 PM Maximilian Michels <m...@apache.org> wrote: > We should fork the discussion around removing instances of @Experimental, > but it was good to mention it here. > > As for the Jet runner, I can only second Ismael: The Jet runner is the > first runner I can think of that came with ValidatesRunner and Nexmark out > of the box. Of course that doesn't mean the runner is "battled-tested", but > we do not have other means to test its maturity. > > For the future, we could come up with other criteria, e.g. a "probation > period", but enforcing this now seems arbitrary. > > If the authors of the Runners decide that it is experimental, so be it. > Otherwise I would leave it to the user to decide (it might be helpful to > list the inception date of each runner). That said, I value your concern > Kenn. I can see that we establish a consistent onboarding of new runners > which may involve marking them experimental for a while. > > -Max > > On 01.07.19 22:20, Kenneth Knowles wrote: > > > > > > On Wed, Jun 12, 2019 at 2:32 AM Ismaël Mejía <ieme...@gmail.com > > <mailto:ieme...@gmail.com>> wrote: > > > > Seems the discussion moved a bit of my original intent that was to > > make the Jet runner directory to be just called runners/jet in the > > directory and mark the 'experimental' part of it in documentation as > > we do for all other things in Beam. > > > > > > Thanks for returning to the one question at hand. We don't have to make > > an overall decision about all "experimental" things. > > > > > > Can we do this or is there still any considerable argument to not do > it? > > > > > > I think we actually have some competing goals: > > > > I agree 100% on the arguments, but let’s think in the reverse terms, > > highlighting lack of maturity can play against the intended goal of > > use and adoption even if for a noble reason. It is basic priming 101 > > [1]. > > > > > > _My_ goal is exactly to highlight lack of maturity so that users are not > > harmed by either (1) necessary breaking changes or (2) permanent low > > quality. Only users who are willing to follow along with the project and > > update their own code regularly should use experimental features. > > > > Evaluating the Jet runner I am convinced by your arguments, because > > looking at the two dangers: > > (1) necessary breaking changes -- runners don't really have their own > > APIs to break, except their own small set of APIs and pipeline options > > (2) permanent low quality -- because there is no API design possible, > > there's no risk of permanent low quality except by fundamental > > mismatches. Plus as you mention the testing is already quite good. > > > > So I am OK to not call it experimental. But I have a slight remaining > > concern that it did not really go through what other runners went > > through. I hope this just means it is more mature. I hope it does not > > indicate that we are reducing rigor. > > > > Kenn > > > > > > On Wed, May 29, 2019 at 3:02 PM Reza Rokni <r...@google.com > > <mailto:r...@google.com>> wrote: > > > > > > Hi, > > > > > > Over 800 usages under java, might be worth doing a few PR... > > > > > > Also suggest we use a very light review process: First round go > > for low hanging fruit, if anyone does a -1 against a change then we > > leave that for round two. > > > > > > Thoughts? > > > > > > Cheers > > > > > > Reza > > > > > > On Wed, 29 May 2019 at 12:05, Kenneth Knowles <k...@apache.org > > <mailto:k...@apache.org>> wrote: > > >> > > >> > > >> > > >> On Mon, May 27, 2019 at 4:05 PM Reza Rokni <r...@google.com > > <mailto:r...@google.com>> wrote: > > >>> > > >>> "Many APIs that have been in place for years and are used by > > most Beam users are still marked Experimental." > > >>> > > >>> Should there be a formal process in place to start 'graduating' > > features out of @Experimental? Perhaps even target an up coming > > release with a PR to remove the annotation from well established > API's? > > >> > > >> > > >> Good idea. I think a PR like this would be an opportunity to > > discuss whether the feature is non-experimental. Probably many of > > them are ready. It would help to address Ismael's very good point > > that this new practice could make users think the old Experimental > > stuff is not experimental. Maybe it is true that it is not really > > still Experimental. > > >> > > >> Kenn > > >> > > >> > > >>> > > >>> On Tue, 28 May 2019 at 06:44, Reuven Lax <re...@google.com > > <mailto:re...@google.com>> wrote: > > >>>> > > >>>> We generally use Experimental for two different things, which > > leads to confusion. > > >>>> 1. Features that work stably, but where we think we might > > still make some changes to the API. > > >>>> 2. New features that we think might not yet be stable. > > >>>> > > >>>> This dual usage leads to a lot of confusion IMO. The fact that > > we tend to forget to remove the @Experimental tag also makes it > > somewhat useless. Many APIs that have been in place for years and > > are used by most Beam users are still marked Experimental. > > >>>> > > >>>> Reuven > > >>>> > > >>>> On Mon, May 27, 2019 at 2:16 PM Ismaël Mejía <ieme...@gmail.com > > <mailto:ieme...@gmail.com>> wrote: > > >>>>> > > >>>>> > Personally, I think that it is good that moving from > > experimental to non-experimental is a breaking change in the > > dependency - one has backwards-incompatible changes and the other > > does not. If artifacts had separate versioning we could use 0.x for > > this. > > >>>>> > > >>>>> In theory it seems so, but in practice it is an annoyance to > > an end > > >>>>> user that already took the ‘risk’ of using an experimental > > feature. > > >>>>> Awareness is probably not the most important reason to break > > existing > > >>>>> code (even if it could be easily fixed). The alternative of > > doing this > > >>>>> with version numbers at least seems less impacting but can be > > >>>>> confusing. > > >>>>> > > >>>>> > But biggest motivation for me are these: > > >>>>> > > > >>>>> > - using experimental features should be opt-in > > >>>>> > - should be impossible to use an experimental feature > > without knowing it (so "opt-in" to a normal-looking feature is not > > enough) > > >>>>> > - developers of an experimental feature should be motivated > > to "graduate" it > > >>>>> > > >>>>> The fundamental problem of this approach is inconsistency with > our > > >>>>> present/past. So far we have ‘Experimental’ features > > everywhere. So > > >>>>> suddenly becoming opt-in let us in an inconsistent state. For > > example > > >>>>> all IOs are marked internally as Experimental but not at the > > level of > > >>>>> directories/artifacts. Adding this suffix in a new IO apart of > > adding > > >>>>> fear of use to the end users may also give the fake impression > > that > > >>>>> the older ones not explicitly marked are not experimental. > > >>>>> > > >>>>> What will be the state for example in the case of runner > > modules that > > >>>>> contain both mature and well tested runners like old Flink and > > Spark > > >>>>> runners vs the more experimental new translations for > Portability, > > >>>>> again more confusion. > > >>>>> > > >>>>> > FWIW I don't think "experimental" should be viewed as a bad > > thing. It just means you are able to make backwards-incompatible > > changes, and that users should be aware that they will need to > > adjust APIs (probably only a little) with new releases. Most > > software is not very good until it has been around for a long time, > > and in my experience the problem is missing the mark on > > abstractions, so backwards compatibility *must* be broken to achieve > > quality. Freezing it early dooms it to never achieving high quality. > > I know of projects where the users explicitly requested that the > > developers not freeze the API but instead prioritize speed and > quality. > > >>>>> > > >>>>> I agree 100% on the arguments, but let’s think in the reverse > > terms, > > >>>>> highlighting lack of maturity can play against the intended > > goal of > > >>>>> use and adoption even if for a noble reason. It is basic > > priming 101 > > >>>>> [1]. > > >>>>> > > >>>>> > Maybe the word is just too negative-sounding? Alternatives > > might be "unstable" or "incubating". > > >>>>> > > >>>>> Yes! “experimental” should not be viewed as a bad thing unless > > you are > > >>>>> a company that has less resources and is trying to protect its > > >>>>> investment so in that case they may doubt to use it. In this > case > > >>>>> probably incubating is a better term because it has less of the > > >>>>> ‘tentative’ dimension associated with Experimental. > > >>>>> > > >>>>> > Now, for the Jet runner, most runners sit on a branch for a > > while, not being released at all, and move to master as their > > "graduation". I think releasing under an "experimental" name is an > > improvement, making it available to users to try out. But we > > probably should have discussed before doing something different than > > all the other runners. > > >>>>> > > >>>>> There is something I don’t get in the case of Jet runner. From > the > > >>>>> discussion in this thread it seems it has everything required > > to not > > >>>>> be ‘experimental’. It passes ValidatesRunner and can even run > > Nexmark > > >>>>> that’s more that some runners already merged in master, so I > still > > >>>>> don’t get why we want to give it a different connotation. > > >>>>> > > >>>>> [1] https://en.wikipedia.org/wiki/Priming_(psychology) > > >>>>> > > >>>>> On Sun, May 26, 2019 at 4:43 AM Kenneth Knowles > > <k...@apache.org <mailto:k...@apache.org>> wrote: > > >>>>> > > > >>>>> > Personally, I think that it is good that moving from > > experimental to non-experimental is a breaking change in the > > dependency - one has backwards-incompatible changes and the other > > does not. If artifacts had separate versioning we could use 0.x for > > this. > > >>>>> > > > >>>>> > But biggest motivation for me are these: > > >>>>> > > > >>>>> > - using experimental features should be opt-in > > >>>>> > - should be impossible to use an experimental feature > > without knowing it (so "opt-in" to a normal-looking feature is not > > enough) > > >>>>> > - developers of an experimental feature should be motivated > > to "graduate" it > > >>>>> > > > >>>>> > So I think a user of an experimental feature should have to > > actually type the word "experimental" either on the command line or > > in their dependencies. That's just my opinion. In the thread [1] > > myself and Robert were the ones that went in this direction of > > opt-in. But it was mostly lazy consensus, plus the review on the > > pull request, that got us to this state. Definitely worth discussing > > more. > > >>>>> > > > >>>>> > FWIW I don't think "experimental" should be viewed as a bad > > thing. It just means you are able to make backwards-incompatible > > changes, and that users should be aware that they will need to > > adjust APIs (probably only a little) with new releases. Most > > software is not very good until it has been around for a long time, > > and in my experience the problem is missing the mark on > > abstractions, so backwards compatibility *must* be broken to achieve > > quality. Freezing it early dooms it to never achieving high quality. > > I know of projects where the users explicitly requested that the > > developers not freeze the API but instead prioritize speed and > quality. > > >>>>> > > > >>>>> > Maybe the word is just too negative-sounding? Alternatives > > might be "unstable" or "incubating". > > >>>>> > > > >>>>> > Now, for the Jet runner, most runners sit on a branch for a > > while, not being released at all, and move to master as their > > "graduation". I think releasing under an "experimental" name is an > > improvement, making it available to users to try out. But we > > probably should have discussed before doing something different than > > all the other runners. > > >>>>> > > > >>>>> > Kenn > > >>>>> > > > >>>>> > [1] > > > https://lists.apache.org/thread.html/302bd51c77feb5c9ce39882316d391535a0fc92e7608a623d9139160@%3Cdev.beam.apache.org%3E > > >>>>> > > > >>>>> > On Sat, May 25, 2019 at 1:03 AM Ismaël Mejía > > <ieme...@gmail.com <mailto:ieme...@gmail.com>> wrote: > > >>>>> >> > > >>>>> >> Including the experimental suffix in artifact names is not > > a good idea > > >>>>> >> either because once we decide that it is not experimantal > > anymore this > > >>>>> >> will be a breaking change for users who will need then to > > update its > > >>>>> >> dependencies code. Also it is error-prone to use different > > mappings > > >>>>> >> for directories and artifacts (even if possible). > > >>>>> >> > > >>>>> >> May we reconsider this Kenn? I understand the motivation > > but I hardly > > >>>>> >> see this making things better or more clear. Any runner > > user will end > > >>>>> >> up reading the runner documentation and capability matrix > > so he will > > >>>>> >> catch the current status that way. > > >>>>> >> > > >>>>> >> > > >>>>> >> > > >>>>> >> On Sat, May 25, 2019 at 8:35 AM Jozsef Bartok > > <jo...@hazelcast.com <mailto:jo...@hazelcast.com>> wrote: > > >>>>> >> > > > >>>>> >> > I missed Ken's input when writing my previous mail. Sorry. > > >>>>> >> > So, to recap: I should remove "experimental" from any > > directory names, but find an other way of configuring the artifact > > so that it still has "experimental" in it's name. > > >>>>> >> > Right? > > >>>>> >> > > > >>>>> >> > On Sat, May 25, 2019 at 9:32 AM Jozsef Bartok > > <jo...@hazelcast.com <mailto:jo...@hazelcast.com>> wrote: > > >>>>> >> >> > > >>>>> >> >> Yes, I'll gladly fix it, we aren't particularly keen to > > be labeled as experimental either.. > > >>>>> >> >> > > >>>>> >> >> Btw. initially the "experimental" word was only in the > > Gradle module name, but then there was some change > > >>>>> >> >> ([BEAM-4046] decouple gradle project names and maven > > artifact ids - 4/2/19) which kind of ended up > > >>>>> >> >> putting it in the directory name. Maybe I should have > > merged with that differently, but this is how > > >>>>> >> >> it seemed consistent. > > >>>>> >> >> > > >>>>> >> >> Anyways, will fix it in my next PR. > > >>>>> >> >> > > >>>>> >> >> On Fri, May 24, 2019 at 5:53 PM Ismaël Mejía > > <ieme...@gmail.com <mailto:ieme...@gmail.com>> wrote: > > >>>>> >> >>> > > >>>>> >> >>> I see thanks Jozsef, marking things as Experimental was > > discussed but > > >>>>> >> >>> we never agreed on doing this at the directory level. > > We can cover the > > >>>>> >> >>> same ground by putting an annotation in the classes (in > > particular the > > >>>>> >> >>> JetRunner and JetPipelineOptions classes which are the > > real public > > >>>>> >> >>> interface, or in the documentation (in particular > > website), I do not > > >>>>> >> >>> see how putting this in the directory name helps and if > > so we may need > > >>>>> >> >>> to put this in many other directories which is far from > > ideal. Any > > >>>>> >> >>> chance this can be fixed (jet-experimental -> jet) ? > > >>>>> >> >>> > > >>>>> >> >>> On Fri, May 24, 2019 at 9:08 AM Jozsef Bartok > > <jo...@hazelcast.com <mailto:jo...@hazelcast.com>> wrote: > > >>>>> >> >>> > > > >>>>> >> >>> > Hi Ismaël! > > >>>>> >> >>> > > > >>>>> >> >>> > Quoting Kenn (from PR-8410): "We discussed on list > > that it would be better to have new things always start as > > experimental in a way that clearly distinguishes them from the core." > > >>>>> >> >>> > > > >>>>> >> >>> > Rgds > > >>>>> >> >>> > > > >>>>> >> >>> > On Thu, May 23, 2019 at 10:44 PM Ismaël Mejía > > <ieme...@gmail.com <mailto:ieme...@gmail.com>> wrote: > > >>>>> >> >>> >> > > >>>>> >> >>> >> I saw that the runner was merged but I don’t get why > > the foler is > > >>>>> >> >>> >> called ‘runners/jet experimental’ and not simply > > ‘runners/jet’. Is it > > >>>>> >> >>> >> because the runner does not pass ValidatesRunner? Or > > because the > > >>>>> >> >>> >> contributors are few? I don’t really see any reason > > behind this > > >>>>> >> >>> >> suffix. And even if the status is not mature that’s > > not different from > > >>>>> >> >>> >> other already merged runners. > > >>>>> >> >>> >> > > >>>>> >> >>> >> On Fri, Apr 26, 2019 at 9:43 PM Kenneth Knowles > > <k...@apache.org <mailto:k...@apache.org>> wrote: > > >>>>> >> >>> >> > > > >>>>> >> >>> >> > Nice! That is *way* more than the PR I was looking > > for. I just meant that you could update the website/ directory. It > > is fine to keep the runner in your own repository if you want. > > >>>>> >> >>> >> > > > >>>>> >> >>> >> > But I think it is great if you want to contribute > > it to Apache Beam (hence donate it to the Apache Software > > Foundation). The benefits include: low-latency testing, free updates > > when someone does a refactor. Things to consider are: subject to ASF > > / Beam governance, PMC, commiters, subject to Beam's release cadence > > (and we might exclude from Beam releases for a little bit). > > Typically, we have kept runners on a branch until they are somewhat > > stable. I don't feel strongly about this for disjoint codebases that > > can easily be excluded from releases. We might want to suffix > > `-experimental` to the artifacts for some time. > > >>>>> >> >>> >> > > > >>>>> >> >>> >> > I commented on the PR about the necessary i.p. > > clearance steps. > > >>>>> >> >>> >> > > > >>>>> >> >>> >> > Kenn > > >>>>> >> >>> >> >the probl > > >>>>> >> >>> >> > On Fri, Apr 26, 2019 at 3:59 AM > > jo...@hazelcast.com <mailto:jo...@hazelcast.com> > > <jo...@hazelcast.com <mailto:jo...@hazelcast.com>> wrote: > > >>>>> >> >>> >> >> > > >>>>> >> >>> >> >> Hi Kenn. > > >>>>> >> >>> >> >> > > >>>>> >> >>> >> >> It took me a while to migrate our code to the > > Beam repo, but I finally have been able to create the Pull Request > > you asked for, this is it: https://github.com/apache/beam/pull/8410 > > >>>>> >> >>> >> >> > > >>>>> >> >>> >> >> Looking forward to your feedback! > > >>>>> >> >>> >> >> > > >>>>> >> >>> >> >> Best regards, > > >>>>> >> >>> >> >> Jozsef > > >>>>> >> >>> >> >> > > >>>>> >> >>> >> >> On 2019/04/19 20:52:42, Kenneth Knowles > > <k...@apache.org <mailto:k...@apache.org>> wrote: > > >>>>> >> >>> >> >> > The ValidatesRunner tests are the best source > > we have for knowing the > > >>>>> >> >>> >> >> > capabilities of a runner. Are there > > instructions for running the tests? > > >>>>> >> >>> >> >> > > > >>>>> >> >>> >> >> > Assuming we can check it out, then just open a > > PR to the website with the > > >>>>> >> >>> >> >> > current capabilities and caveats. Since it is a > > big deal and could use lots > > >>>>> >> >>> >> >> > of eyes, I would share the PR link on this > thread. > > >>>>> >> >>> >> >> > > > >>>>> >> >>> >> >> > Kenn > > >>>>> >> >>> >> >> > > > >>>>> >> >>> >> >> > On Thu, Apr 18, 2019 at 11:53 AM Jozsef Bartok > > <jo...@hazelcast.com <mailto:jo...@hazelcast.com>> wrote: > > >>>>> >> >>> >> >> > > > >>>>> >> >>> >> >> > > Hi. We at Hazelcast Jet have been working for > > a while now to implement a > > >>>>> >> >>> >> >> > > Java Beam Runner (non-portable) based on > > Hazelcast Jet ( > > >>>>> >> >>> >> >> > > https://jet.hazelcast.org/). The process is > > still ongoing ( > > >>>>> >> >>> >> >> > > > > https://github.com/hazelcast/hazelcast-jet-beam-runner), but we are > > >>>>> >> >>> >> >> > > aiming for a fully functional, reliable > > Runner which can proudly join the > > >>>>> >> >>> >> >> > > Capability Matrix. For that purpose I would > > like to ask what’s your process > > >>>>> >> >>> >> >> > > of validating runners? We are already running > > the @ValidatesRunner tests > > >>>>> >> >>> >> >> > > and the Nexmark test suite, but beyond that > > what other steps do we need to > > >>>>> >> >>> >> >> > > take to get our Runner to the level it needs > > to be at? > > >>>>> >> >>> >> >> > > > > >>>>> >> >>> >> >> > > > >>> > > >>> > > >>> > > >>> -- > > >>> > > >>> This email may be confidential and privileged. If you received > > this communication by mistake, please don't forward it to anyone > > else, please erase all copies and attachments, and please let me > > know that it has gone to the wrong person. > > >>> > > >>> The above terms reflect a potential business arrangement, are > > provided solely as a basis for further discussion, and are not > > intended to be and do not constitute a legally binding obligation. > > No legally binding obligations will be created, implied, or inferred > > until an agreement in final form is executed in writing by all > > parties involved. > > > > > > > > > > > > -- > > > > > > This email may be confidential and privileged. If you received > > this communication by mistake, please don't forward it to anyone > > else, please erase all copies and attachments, and please let me > > know that it has gone to the wrong person. > > > > > > The above terms reflect a potential business arrangement, are > > provided solely as a basis for further discussion, and are not > > intended to be and do not constitute a legally binding obligation. > > No legally binding obligations will be created, implied, or inferred > > until an agreement in final form is executed in writing by all > > parties involved. > > > > > >
