It seems we mostly agree that @Experimental is important, and that API changes (removals) on experimental features should happen quickly but still give some time to users so the Experimental purpose is not lost.
Ahmet proposal given our current release calendar is close to 2 releases. Can we settle this on 2 releases as a 'minimum time' before removal? (This will let maintainers the option to choose to support it more time if they want as discussed in the related KafkaIO thread but still be friendly with users). Do we agree? Note: for the other subjects (e.g. when an Experimental feature should become not experimental) I think we will hardly find an agreement so I think this should be treated in a per case basis by the maintainers, but if you want to follow up on that discussion we can open another thread for this. On Sat, Apr 6, 2019 at 1:04 AM Ahmet Altay <al...@google.com> wrote: > I agree that Experimental feature is still very useful. I was trying to > argue that we diluted its value so +1 to reclaim that. > > Back to the original question, in my opinion removing existing > "experimental and deprecated" features in n=1 release will confuse users. > This will likely be a surprise to them because we have been maintaining > this state release after release now. I would propose in the next release > warning users of such a change happening and give them at least 3 months to > upgrade to suggested newer paths. In the future we can have a shorter > timelines assuming that we will set the user expectations right. > > On Fri, Apr 5, 2019 at 3:01 PM Ismaël Mejía <ieme...@gmail.com> wrote: > >> I agree 100% with Kenneth on the multiple advantages that the >> Experimental feature gave us. I also can count multiple places where this >> has been essential in other modules than core. I disagree on the fact that >> the @Experimental annotation has lost sense, it is simply ill defined, and >> probably it is by design because its advantages come from it. >> >> Most of the topics in this thread are a consequence of the this loose >> definition, e.g. (1) not defining how a feature becomes stable, and (2) >> what to do when we want to remove an experimental feature, are ideas that >> we need to decide if we define just continue to handle as we do today. >> >> Defining a target for graduating an Experimental feature is a bit too >> aggressive with not much benefit, in this case we could be losing the >> advantages of Experimental (save if we could change the proposed version in >> the future). This probably makes sense for the removal of features but >> makes less sense to decide when some feature becomes stable. Of course in >> the case of the core SDKs packages this is probably more critical but >> nothing guarantees that things will be ready when we expect too. When will >> we tag for stability things like SDF or portability APIs?. We cannot >> predict the future for completion of features. >> >> Nobody has mentioned the LTS releases couldn’t be these like the middle >> points for these decisions? That at least will give LTS some value because >> so far I still have issues to understand the value of this idea given that >> we can do a minor release of any pre-released version. >> >> This debate is super important and nice to have, but we lost focus on my >> initial question. I like the proposal to remove a deprecated experimental >> feature (or part of it) after one release, in particular if the feature has >> a clear replacement path, however for cases like the removal of previously >> supported versions of Kafka one release may be too short. Other opinions on >> this? (or the other topics). >> >> On Fri, Apr 5, 2019 at 10:52 AM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> if it's technically feasible, I am also in favor of requiring >>> experimental features to be (per-tag, Python should be updated) opt-in >>> only. We should probably regularly audit the set of experimental features >>> we ship (I'd say as part of the release, but that process is laborious >>> enough, perhaps we should do it on a half-release cycle?) I think imposing >>> hard deadlines (chosen when a feature is introduced) is too extreme, but >>> might be valuable if opt-in plus regular audit is insufficient. >>> >>> On Thu, Apr 4, 2019 at 5:28 AM Kenneth Knowles <k...@apache.org> wrote: >>> >>>> This all makes me think that we should rethink how we ship experimental >>>> features. My experience is also that (1) users don't know if something is >>>> experimental or don't think hard about it and (2) we don't use experimental >>>> time period to gather feedback and make changes. >>>> >>>> How can we change both of these? Perhaps we could require experimental >>>> features to be opt-in. Flags work and also clearly marked experimental >>>> dependencies that a user has to add. Changing the core is sometimes tricky >>>> to put behind a flag but rarely impossible. This way a contributor is also >>>> motivated to gather feedback to mature their feature to become default >>>> instead of opt-in. >>>> >>>> The need that @Experimental was trying to address is real. We *do* need >>>> a way to try things and get feedback prior to committing to forever >>>> support. We have discovered real problems far too late, or not had the will >>>> to fix the issue we did find: >>>> - many trigger combinators should probably be deleted >>>> - many triggers cannot meet a good spec with merging windows >>>> - the continuation trigger idea doesn't work well >>>> - CombineFn had to have its spec changed in order to be both correct >>>> and efficient >>>> - OutputTimeFn as a UDF is convenient for Java but it turns out an >>>> enum is better for portability >>>> - Coder contexts turned out to be a major usability problem >>>> - The built-in data types for schemas are evolving (luckily these are >>>> really being worked on!) >>>> >>>> That's just what I can think of off the top of my head. I expect the >>>> examples from IOs are more numerous; in that case it is pretty easy to fork >>>> and make a new and better IO. >>>> >>>> And as an extreme view, I would prefer if we add a deadline for >>>> experimental features, then our default action is to remove them, not >>>> declare them stable. If noone is trying to mature it and get it out of >>>> opt-in status, then it probably has not matured. And perhaps if noone care >>>> enough to do that work it also isn't that important. >>>> >>>> Kenn >>>> >>>> On Wed, Apr 3, 2019 at 5:57 PM Ahmet Altay <al...@google.com> wrote: >>>> >>>>> I agree with Reuven that our experimental annotation is not useful any >>>>> more. For example Datastore IO in python sdk is experimental for 2 years >>>>> now. Even though it is marked as experimental an upgrade is carefully >>>>> planned [1] as if it is not experimental. Given that I do not think we can >>>>> remove features within a small number of minor releases. (Exception to >>>>> this >>>>> would be, if we have a clear knowledge of very low usage of a certain IO.) >>>>> >>>>> I am worried that tagging experimental features with release versions >>>>> will add toil to the release process as mentioned and will also add to the >>>>> user confusion. What would be the signal to a user if they see an >>>>> experimental feature target release bumped between releases? How about >>>>> tagging experimental features with JIRAs (similar to TODOs) with an action >>>>> to either promote them as supported features or remove them? These JIRAs >>>>> could have fix version targets as any other release blocking JIRAs. It >>>>> will >>>>> also clarify who is responsible for a given experimental feature. >>>>> >>>>> [1] >>>>> https://lists.apache.org/thread.html/5ec88967aa4a382db07a60e0101c4eb36165909076867155ab3546a6@%3Cdev.beam.apache.org%3E >>>>> >>>>> On Wed, Apr 3, 2019 at 5:24 PM Reuven Lax <re...@google.com> wrote: >>>>> >>>>>> Experiments are already tagged with a Kind enum >>>>>> (e.g. @Experimental(Kind.Schemas)). >>>>>> >>>>> >>>>> This not the case for python's annotations. It will be a good idea to >>>>> add there as well. >>>>> >>>>> >>>>>> >>>>>> On Wed, Apr 3, 2019 at 4:56 PM Ankur Goenka <goe...@google.com> >>>>>> wrote: >>>>>> >>>>>>> I think a release version with Experimental flag makes sense. >>>>>>> In addition, I think many of our user start to rely on experimental >>>>>>> features because they are not even aware that these features are >>>>>>> experimental and its really hard to find the experimental features used >>>>>>> without giving a good look at the Beam code and having some knowledge >>>>>>> about >>>>>>> it. >>>>>>> >>>>>>> It will be good it we can have a step at the pipeline submission >>>>>>> time which can print all the experiments used in verbose mode. This >>>>>>> might >>>>>>> also require to add a meaningful group name for the experiment example >>>>>>> >>>>>>> @Experimental("SDF", 2.15.0) >>>>>>> >>>>>>> This will of-course add additional effort and require additional >>>>>>> context while tagging experiments. >>>>>>> >>>>>>> On Wed, Apr 3, 2019 at 4:43 PM Reuven Lax <re...@google.com> wrote: >>>>>>> >>>>>>>> Our Experimental annotation has become almost useless. Many core, >>>>>>>> widely-used parts of the API (e.g. triggers) are still all marked as >>>>>>>> experimental. So many users use these features that we couldn't really >>>>>>>> change them (in a backwards-incompatible) without hurting many users, >>>>>>>> so >>>>>>>> the fact they are marked Experimental has become a fiction. >>>>>>>> >>>>>>>> Could we add a deadline to the Experimental tag - a release version >>>>>>>> when it will be removed? e.g. >>>>>>>> >>>>>>>> @Experimental(2.15.0) >>>>>>>> >>>>>>>> We can have a test that ensure that the tag is removed at this >>>>>>>> version. Of course if we're not ready to remove experimental by that >>>>>>>> version, it's fine - we can always bump the tagged version. However >>>>>>>> this >>>>>>>> forces us to think about each one. >>>>>>>> >>>>>>>> Downside - it might add more toil to the existing release process. >>>>>>>> >>>>>>>> Reuven >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Apr 3, 2019 at 4:00 PM Kyle Weaver <kcwea...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> > We might also want to get in the habit of reviewing if something >>>>>>>>> should no longer be experimental. >>>>>>>>> >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> Kyle Weaver | Software Engineer | kcwea...@google.com | >>>>>>>>> +16502035555 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Apr 3, 2019 at 3:53 PM Kenneth Knowles <k...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I think option 2 with n=1 minor version seems OK. So users get >>>>>>>>>> the message for one release and it is gone the next. We should make >>>>>>>>>> sure >>>>>>>>>> the deprecation warning says "this is an experimental feature, so it >>>>>>>>>> will >>>>>>>>>> be removed after 1 minor version". And we need a process for doing >>>>>>>>>> it so it >>>>>>>>>> doesn't sit around. I think we should also leave room for using our >>>>>>>>>> own >>>>>>>>>> judgment about whether the user pain is very little and then it is >>>>>>>>>> not >>>>>>>>>> needed to have a deprecation cycle. >>>>>>>>>> >>>>>>>>>> We might also want to get in the habit of reviewing if something >>>>>>>>>> should no longer be experimental. >>>>>>>>>> >>>>>>>>>> Kenn >>>>>>>>>> >>>>>>>>>> On Wed, Apr 3, 2019 at 2:33 PM Ismaël Mejía <ieme...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> When we did the first stable release of Beam (2.0.0) we decided >>>>>>>>>>> to >>>>>>>>>>> annotate most of the Beam IOs as @Experimental because we were >>>>>>>>>>> cautious about not getting the APIs right in the first try. This >>>>>>>>>>> was a >>>>>>>>>>> really good decision because we could do serious improvements and >>>>>>>>>>> refactorings to them in the first releases without the hassle of >>>>>>>>>>> keeping backwards compatibility. However after some more releases >>>>>>>>>>> users started to rely on features and supported versions, so we >>>>>>>>>>> ended >>>>>>>>>>> up in a situation where we could not change them arbitrarily >>>>>>>>>>> without >>>>>>>>>>> consequences to the final users. >>>>>>>>>>> >>>>>>>>>>> So we started to deprecate some features and parts of the API >>>>>>>>>>> without >>>>>>>>>>> removing them, e.g. the introduction of HadoopFormatIO deprecated >>>>>>>>>>> HadoopInputFormatIO, we deprecated methods of MongoDbIO and >>>>>>>>>>> MqttIO to >>>>>>>>>>> improve the APIs (in most cases with valid/improved >>>>>>>>>>> replacements), and >>>>>>>>>>> recently it was discussed to removal of support for older >>>>>>>>>>> versions in >>>>>>>>>>> KafkaIO. >>>>>>>>>>> >>>>>>>>>>> Keeping deprecated stuff in experimental APIs does not seem to >>>>>>>>>>> make >>>>>>>>>>> sense, but it is what he have started to do to be ‘user >>>>>>>>>>> friendly’, but >>>>>>>>>>> it is probably a good moment to define, what should be the clear >>>>>>>>>>> path >>>>>>>>>>> for removal and breaking changes of experimental features, some >>>>>>>>>>> options: >>>>>>>>>>> >>>>>>>>>>> 1. Stay as we were, do not mark things as deprecated and remove >>>>>>>>>>> them >>>>>>>>>>> at will because this is the contract of @Experimental. >>>>>>>>>>> 2. Deprecate stuff and remove it after n versions (where n could >>>>>>>>>>> be 3 releases). >>>>>>>>>>> 3. Deprecate stuff and remove it just after a new LTS is decided >>>>>>>>>>> to >>>>>>>>>>> ensure users who need these features may still have them for some >>>>>>>>>>> time. >>>>>>>>>>> >>>>>>>>>>> I would like to know your opinions about this, or if you have >>>>>>>>>>> other >>>>>>>>>>> ideas. Notice that in discussion I refer only to @Experimental >>>>>>>>>>> features. >>>>>>>>>>> >>>>>>>>>>
