Thank for your getting us started on annotating the API. The list looks good so far. I have the feeling it could even be extended a bit. Just curious, how did you choose which classes you annotate? Did you go through all the classes in flink-core, flink-java, and flink-clients Maven projects?
What about flink-scala? Shouldn't it be annotated as well? On Fri, Nov 27, 2015 at 4:47 PM, Robert Metzger <rmetz...@apache.org> wrote: > Okay, I'll introduce an annotation for experimental interfaces and I'll > make everything we have deprecated experimental. > > On Fri, Nov 27, 2015 at 10:40 AM, Aljoscha Krettek <aljos...@apache.org> > wrote: > >> I still think we also need an annotation to mark public interfaces as >> experimental. For example for the windowing/triggers I would like to use >> that. >> > On 25 Nov 2015, at 01:23, Robert Metzger <rmetz...@apache.org> wrote: >> > >> > Thank you Nick. I'll look into the check_compatiblilty.sh script to see >> > which tools are used. >> > I think finding breaking API changes immediately is a better process then >> > reworking the APIs before a release. >> > >> > As you can see from my email response times (2 days since your email), >> I'm >> > probably too overloaded right now to participate in the Yetus project ... >> > Sadly. >> > >> > >> > @others: I know its not the most interesting thing to go through my list >> of >> > stable interfaces, but keep in mind that we have to maintain the stuff >> for >> > probably quite some time, so it would be good to have more than one pair >> of >> > eyes looking at it :) >> > >> > >> > On Mon, Nov 23, 2015 at 6:20 PM, Nick Dimiduk <ndimi...@gmail.com> >> wrote: >> > >> >>> >> >>> Do you know if Hadoop/HBase is also using a maven plugin to fail a >> build >> >> on >> >>> breaking API changes? I would really like to have such a functionality >> in >> >>> Flink, because we can spot breaking changes very early. >> >> >> >> >> >> I don't think we have maven integration for this as of yet. We release >> >> managers run a script $HBASE/dev-support/check_compatibility.sh that >> >> generates a source and binary compatibility report. Issues are then >> >> resolved during the period leading up to the release candidate. >> >> >> >> I think Hadoop is relying on a "QA bot" which reads patches from JIRA >> and >> >>> then does these >> >>> checks? >> >>> >> >> >> >> The "QA bot" is just a collection of shell scripts used during "Patch >> >> Available" status when a patch has been attached to JIRA or when a PR >> has >> >> been submitted through github. The check_compatibility script could be >> >> included in that automation, I don't see why not. Maybe you'd like to >> open >> >> a YETUS ticket :) >> >> >> >> I've pushed a branch to my own GitHub account with all classes I would >> make >> >>> public annotated: >> >>> >> >>> >> >> >> https://github.com/apache/flink/compare/master...rmetzger:interface_stability_revapi?expand=1 >> >>> Since this is really hard to read, I (half-automated) generated the >> >>> following list of annotated classes: >> >>> >> >>> >> >> >> https://github.com/rmetzger/flink/blob/interface_stability_revapi/annotations.md >> >>> >> >>> Please let me know if you would like to include or exclude classes from >> >>> that list. >> >>> Also, let me know which methods (in stable classes) you would mark as >> >>> experimental. >> >>> >> >>> >> >>> >> >>> >> >>> On Wed, Nov 11, 2015 at 10:17 AM, Aljoscha Krettek < >> aljos...@apache.org> >> >>> wrote: >> >>> >> >>>> +1 for some way of declaring public interfaces as experimental. >> >>>> >> >>>>> On 10 Nov 2015, at 22:24, Stephan Ewen <se...@apache.org> wrote: >> >>>>> >> >>>>> I think we need anyways an annotation "@PublicExperimental". >> >>>>> >> >>>>> We can make this annotation such that it can be added to methods and >> >>> can >> >>>>> use that to declare >> >>>>> Methods in an otherwise public class (such as DataSet) as >> >> experimental. >> >>>>> >> >>>>> On Tue, Nov 10, 2015 at 10:19 PM, Fabian Hueske <fhue...@gmail.com> >> >>>> wrote: >> >>>>> >> >>>>>> I am not sure if we always should declare complete classes as >> >>>>>> @PublicInterface. >> >>>>>> This does definitely make sense for interfaces and abstract classes >> >>>> such as >> >>>>>> MapFunction or InputFormat but not necessarily for classes such as >> >>>> DataSet >> >>>>>> that we might want to extend by methods which should not immediately >> >>> be >> >>>>>> considered as stable. >> >>>>>> >> >>>>>> >> >>>>>> 2015-11-10 21:36 GMT+01:00 Vasiliki Kalavri < >> >>> vasilikikala...@gmail.com >> >>>>> : >> >>>>>> >> >>>>>>> Yes, my opinion is that we shouldn't declare the Gelly API frozen >> >>> yet. >> >>>>>>> We can reconsider when we're closer to the 1.0 release, but if >> >>>> possible, >> >>>>>> I >> >>>>>>> would give it some more time. >> >>>>>>> >> >>>>>>> -V. >> >>>>>>> >> >>>>>>> On 10 November 2015 at 21:06, Stephan Ewen <se...@apache.org> >> >> wrote: >> >>>>>>> >> >>>>>>>> I think no component should be forced to be stable. It should be >> >> an >> >>>>>>>> individual decision for each component, and in some cases even for >> >>>>>>>> individual classes. >> >>>>>>>> >> >>>>>>>> @Vasia If you think Gelly should not be declared interface-frozen, >> >>>> then >> >>>>>>>> this is a good point to raise and this should definitely be >> >>> reflected. >> >>>>>>>> There is no point in declaring certain APIs as frozen when we are >> >>> not >> >>>>>> yet >> >>>>>>>> confident they have converged. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Tue, Nov 10, 2015 at 8:39 PM, Vasiliki Kalavri < >> >>>>>>>> vasilikikala...@gmail.com >> >>>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> Hi Robert, >> >>>>>>>>> >> >>>>>>>>> thanks for bringing this up! >> >>>>>>>>> >> >>>>>>>>> I generally like the idea, but I wouldn't rush to annotate the >> >>> Gelly >> >>>>>>>>> classes yet. Gelly hasn't had that many users and I'm quite sure >> >>>>>> we'll >> >>>>>>>> find >> >>>>>>>>> things to improve as it gets more exposure. >> >>>>>>>>> TBH, I think it's quite unfair to force Gelly (also e.g. ML, >> >> Table) >> >>>>>> to >> >>>>>>> a >> >>>>>>>>> "1.0" status (from an API stability point of view) since they're >> >>>>>> really >> >>>>>>>>> young compared to the other Flink APIs. >> >>>>>>>>> >> >>>>>>>>> Cheers, >> >>>>>>>>> Vasia. >> >>>>>>>>> On Nov 10, 2015 8:04 PM, "Robert Metzger" <rmetz...@apache.org> >> >>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Hi everyone, >> >>>>>>>>>> >> >>>>>>>>>> I would like to bring this discussion back to your attention as >> >> we >> >>>>>>> seem >> >>>>>>>>> to >> >>>>>>>>>> approach the 1.0 release of Flink. >> >>>>>>>>>> >> >>>>>>>>>> My suggestion back in January was to annotate all classes, but I >> >>>>>>> think >> >>>>>>>>>> it'll be more feasible to just annotate public classes. >> >>>>>>>>>> So how about adding an annotation @PublicInterface >> >>>>>>>>>> >> >>>>>>>>>> For @PublicInterface, I would annotate classes such as: DataSet, >> >>>>>>>>>> DataStream, ExecutionEnvironment, InputFormat, MapFunction, >> >>>>>>> FileSystems >> >>>>>>>>> but >> >>>>>>>>>> also Gelly for example. >> >>>>>>>>>> >> >>>>>>>>>> I would not annotate as public components such as ML, Storm >> >>>>>>>>> compatibility, >> >>>>>>>>>> internals from runtime, yarn, optimizer. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> From a tooling perspective, I've looked into different maven >> >>>>>> plugins >> >>>>>>>> and >> >>>>>>>>>> java libraries and I found https://github.com/siom79/japicmp to >> >>> be >> >>>>>>> the >> >>>>>>>>>> closest to our needs. I actually opened a pull request to the >> >>>>>> project >> >>>>>>>> to >> >>>>>>>>>> allow inclusion/exclusion of classes based on annotations. Lets >> >>>>>> hope >> >>>>>>> it >> >>>>>>>>>> gets merged. >> >>>>>>>>>> >> >>>>>>>>>> Does everybody agree with adding just the @PublicInterface >> >>>>>>> annotation? >> >>>>>>>>>> >> >>>>>>>>>> Note that I'll add the annotation on a class-level, making the >> >>>>>> entire >> >>>>>>>>> class >> >>>>>>>>>> either public or private (from a stability point of view). If we >> >>>>>>> need a >> >>>>>>>>>> more fine-grained annotation, we have to add a second >> >>>>>>> @PrivateInterface >> >>>>>>>>>> annotation which we'll only apply to certain methods. >> >>>>>>>>>> >> >>>>>>>>>> The next step is that I'm going to open a pull request with all >> >>>>>>> classes >> >>>>>>>>>> annotated that I consider public. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Fri, Jan 30, 2015 at 7:10 PM, Henry Saputra < >> >>>>>>>> henry.sapu...@gmail.com> >> >>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> I like the idea. But would love to have different name for the >> >>>>>>>>>>> "LimitedPrivate" to make it easier to distinguish. >> >>>>>>>>>>> How about "Module" or "Package" ? >> >>>>>>>>>>> >> >>>>>>>>>>> - Henry >> >>>>>>>>>>> >> >>>>>>>>>>> On Wed, Jan 28, 2015 at 1:29 AM, Robert Metzger < >> >>>>>>> rmetz...@apache.org >> >>>>>>>>> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>> I think in Hadoop they use LimitedPrivate for the different >> >>>>>>>>> components >> >>>>>>>>>> of >> >>>>>>>>>>>> the project. >> >>>>>>>>>>>> For example LimitedPrivate("yarn"). >> >>>>>>>>>>>> Here is a very good documentation on the topic: >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>> >> >>> >> >> >> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/InterfaceClassification.html >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Tue, Jan 27, 2015 at 3:58 PM, Alexander Alexandrov < >> >>>>>>>>>>>> alexander.s.alexand...@gmail.com> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> I don't get the difference between Private and >> >> LimitedPrivate, >> >>>>>>> but >> >>>>>>>>>>>>> otherwise seems like quite a nice idea. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> It will be also good if we can agree upon what these tags >> >>>>>>> actually >> >>>>>>>>>> mean >> >>>>>>>>>>> and >> >>>>>>>>>>>>> add this meaning to the documentation. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> 2015-01-27 15:46 GMT+01:00 Robert Metzger < >> >>>>>> rmetz...@apache.org >> >>>>>>>> : >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> Hi, >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Hadoop has annotations for tagging the stability and >> >>>>>> audience >> >>>>>>> of >> >>>>>>>>>>> classes >> >>>>>>>>>>>>>> and methods. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Through that, you can have @InterfaceAudience.Public, >> >>>>>> Private, >> >>>>>>>>>>>>>> LimitedPrivate >> >>>>>>>>>>>>>> and also @InterfaceStability.Evolving, Unstable, and Stable. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> I guess there are tools which allow to automatically check >> >>>>>> if >> >>>>>>>>>>> interfaces, >> >>>>>>>>>>>>>> which are marked as Stable have been changed between >> >>>>>> releases >> >>>>>>>> (or >> >>>>>>>>> by >> >>>>>>>>>>> pull >> >>>>>>>>>>>>>> requests). >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> I think such annotations are crucial if we want to guarantee >> >>>>>>>>>> interface >> >>>>>>>>>>>>>> stability between releases. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> What do you think? Should we add those annotations? Which >> >>>>>> one >> >>>>>>>>> would >> >>>>>>>>>>> you >> >>>>>>>>>>>>>> like to add? >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Robert >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>> >> >>>> >> >>> >> >> >> >>
