if it's technically feasible, I am also in favor of requiring experimental features to be (per-tag, Python should be updated) opt-in only. We should probably regularly audit the set of experimental features we ship (I'd say as part of the release, but that process is laborious enough, perhaps we should do it on a half-release cycle?) I think imposing hard deadlines (chosen when a feature is introduced) is too extreme, but might be valuable if opt-in plus regular audit is insufficient.
On Thu, Apr 4, 2019 at 5:28 AM Kenneth Knowles <k...@apache.org> wrote: > This all makes me think that we should rethink how we ship experimental > features. My experience is also that (1) users don't know if something is > experimental or don't think hard about it and (2) we don't use experimental > time period to gather feedback and make changes. > > How can we change both of these? Perhaps we could require experimental > features to be opt-in. Flags work and also clearly marked experimental > dependencies that a user has to add. Changing the core is sometimes tricky > to put behind a flag but rarely impossible. This way a contributor is also > motivated to gather feedback to mature their feature to become default > instead of opt-in. > > The need that @Experimental was trying to address is real. We *do* need a > way to try things and get feedback prior to committing to forever support. > We have discovered real problems far too late, or not had the will to fix > the issue we did find: > - many trigger combinators should probably be deleted > - many triggers cannot meet a good spec with merging windows > - the continuation trigger idea doesn't work well > - CombineFn had to have its spec changed in order to be both correct and > efficient > - OutputTimeFn as a UDF is convenient for Java but it turns out an enum > is better for portability > - Coder contexts turned out to be a major usability problem > - The built-in data types for schemas are evolving (luckily these are > really being worked on!) > > That's just what I can think of off the top of my head. I expect the > examples from IOs are more numerous; in that case it is pretty easy to fork > and make a new and better IO. > > And as an extreme view, I would prefer if we add a deadline for > experimental features, then our default action is to remove them, not > declare them stable. If noone is trying to mature it and get it out of > opt-in status, then it probably has not matured. And perhaps if noone care > enough to do that work it also isn't that important. > > Kenn > > On Wed, Apr 3, 2019 at 5:57 PM Ahmet Altay <al...@google.com> wrote: > >> I agree with Reuven that our experimental annotation is not useful any >> more. For example Datastore IO in python sdk is experimental for 2 years >> now. Even though it is marked as experimental an upgrade is carefully >> planned [1] as if it is not experimental. Given that I do not think we can >> remove features within a small number of minor releases. (Exception to this >> would be, if we have a clear knowledge of very low usage of a certain IO.) >> >> I am worried that tagging experimental features with release versions >> will add toil to the release process as mentioned and will also add to the >> user confusion. What would be the signal to a user if they see an >> experimental feature target release bumped between releases? How about >> tagging experimental features with JIRAs (similar to TODOs) with an action >> to either promote them as supported features or remove them? These JIRAs >> could have fix version targets as any other release blocking JIRAs. It will >> also clarify who is responsible for a given experimental feature. >> >> [1] >> https://lists.apache.org/thread.html/5ec88967aa4a382db07a60e0101c4eb36165909076867155ab3546a6@%3Cdev.beam.apache.org%3E >> >> On Wed, Apr 3, 2019 at 5:24 PM Reuven Lax <re...@google.com> wrote: >> >>> Experiments are already tagged with a Kind enum >>> (e.g. @Experimental(Kind.Schemas)). >>> >> >> This not the case for python's annotations. It will be a good idea to add >> there as well. >> >> >>> >>> On Wed, Apr 3, 2019 at 4:56 PM Ankur Goenka <goe...@google.com> wrote: >>> >>>> I think a release version with Experimental flag makes sense. >>>> In addition, I think many of our user start to rely on experimental >>>> features because they are not even aware that these features are >>>> experimental and its really hard to find the experimental features used >>>> without giving a good look at the Beam code and having some knowledge about >>>> it. >>>> >>>> It will be good it we can have a step at the pipeline submission time >>>> which can print all the experiments used in verbose mode. This might also >>>> require to add a meaningful group name for the experiment example >>>> >>>> @Experimental("SDF", 2.15.0) >>>> >>>> This will of-course add additional effort and require additional >>>> context while tagging experiments. >>>> >>>> On Wed, Apr 3, 2019 at 4:43 PM Reuven Lax <re...@google.com> wrote: >>>> >>>>> Our Experimental annotation has become almost useless. Many core, >>>>> widely-used parts of the API (e.g. triggers) are still all marked as >>>>> experimental. So many users use these features that we couldn't really >>>>> change them (in a backwards-incompatible) without hurting many users, so >>>>> the fact they are marked Experimental has become a fiction. >>>>> >>>>> Could we add a deadline to the Experimental tag - a release version >>>>> when it will be removed? e.g. >>>>> >>>>> @Experimental(2.15.0) >>>>> >>>>> We can have a test that ensure that the tag is removed at this >>>>> version. Of course if we're not ready to remove experimental by that >>>>> version, it's fine - we can always bump the tagged version. However this >>>>> forces us to think about each one. >>>>> >>>>> Downside - it might add more toil to the existing release process. >>>>> >>>>> Reuven >>>>> >>>>> >>>>> On Wed, Apr 3, 2019 at 4:00 PM Kyle Weaver <kcwea...@google.com> >>>>> wrote: >>>>> >>>>>> > We might also want to get in the habit of reviewing if something >>>>>> should no longer be experimental. >>>>>> >>>>>> +1 >>>>>> >>>>>> Kyle Weaver | Software Engineer | kcwea...@google.com | >>>>>> +16502035555 >>>>>> >>>>>> >>>>>> On Wed, Apr 3, 2019 at 3:53 PM Kenneth Knowles <k...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> I think option 2 with n=1 minor version seems OK. So users get the >>>>>>> message for one release and it is gone the next. We should make sure the >>>>>>> deprecation warning says "this is an experimental feature, so it will be >>>>>>> removed after 1 minor version". And we need a process for doing it so it >>>>>>> doesn't sit around. I think we should also leave room for using our own >>>>>>> judgment about whether the user pain is very little and then it is not >>>>>>> needed to have a deprecation cycle. >>>>>>> >>>>>>> We might also want to get in the habit of reviewing if something >>>>>>> should no longer be experimental. >>>>>>> >>>>>>> Kenn >>>>>>> >>>>>>> On Wed, Apr 3, 2019 at 2:33 PM Ismaël Mejía <ieme...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> When we did the first stable release of Beam (2.0.0) we decided to >>>>>>>> annotate most of the Beam IOs as @Experimental because we were >>>>>>>> cautious about not getting the APIs right in the first try. This >>>>>>>> was a >>>>>>>> really good decision because we could do serious improvements and >>>>>>>> refactorings to them in the first releases without the hassle of >>>>>>>> keeping backwards compatibility. However after some more releases >>>>>>>> users started to rely on features and supported versions, so we >>>>>>>> ended >>>>>>>> up in a situation where we could not change them arbitrarily without >>>>>>>> consequences to the final users. >>>>>>>> >>>>>>>> So we started to deprecate some features and parts of the API >>>>>>>> without >>>>>>>> removing them, e.g. the introduction of HadoopFormatIO deprecated >>>>>>>> HadoopInputFormatIO, we deprecated methods of MongoDbIO and MqttIO >>>>>>>> to >>>>>>>> improve the APIs (in most cases with valid/improved replacements), >>>>>>>> and >>>>>>>> recently it was discussed to removal of support for older versions >>>>>>>> in >>>>>>>> KafkaIO. >>>>>>>> >>>>>>>> Keeping deprecated stuff in experimental APIs does not seem to make >>>>>>>> sense, but it is what he have started to do to be ‘user friendly’, >>>>>>>> but >>>>>>>> it is probably a good moment to define, what should be the clear >>>>>>>> path >>>>>>>> for removal and breaking changes of experimental features, some >>>>>>>> options: >>>>>>>> >>>>>>>> 1. Stay as we were, do not mark things as deprecated and remove them >>>>>>>> at will because this is the contract of @Experimental. >>>>>>>> 2. Deprecate stuff and remove it after n versions (where n could be >>>>>>>> 3 releases). >>>>>>>> 3. Deprecate stuff and remove it just after a new LTS is decided to >>>>>>>> ensure users who need these features may still have them for some >>>>>>>> time. >>>>>>>> >>>>>>>> I would like to know your opinions about this, or if you have other >>>>>>>> ideas. Notice that in discussion I refer only to @Experimental >>>>>>>> features. >>>>>>>> >>>>>>>
