Thanks Sonke, that's great. I filed an initial JIRA: https://issues.apache.org/jira/browse/KAFKA-5637
As per our offline conversation, you captured the thread discussion already, so feel free to flesh out the JIRA. Ismael On Mon, Jul 24, 2017 at 8:45 PM, Sönke Liebau < soenke.lie...@opencore.com.invalid> wrote: > I am happy to volunteer for working on documenting the release logistics > and related topics if someone is needed for that part. > > Best regards, > Sönke > > Am 24.07.2017 9:28 nachm. schrieb "Guozhang Wang" <wangg...@gmail.com>: > > > Thanks for everyone who has shared their thoughts and feedbacks to this > > discussion. Here is my conclusion: > > > > 1. So far everyone has agreed on naming the next major release version to > > 1.0; if I did not hear any other opinions by the end of today PDT time I > > will name the next release as 1.0.0 and update the release wiki page > > correspondingly. > > > > 2. People have also expressed expectations to some of the release > logistics > > while we are releasing 1.0.0, including old version support period, > upgrade > > support period, bug fix / security fix backport policies etc. Ismael has > a > > detailed summary on these in this email thread and we will look for > someone > > to drive the documentation of these in parallel with the 1.0.0 release > > process. > > > > > > Guozhang > > > > > > On Mon, Jul 24, 2017 at 11:45 AM, Becket Qin <becket....@gmail.com> > wrote: > > > > > +1 on 1.0. I think it makes a lot of sense given the point Kafka is > now. > > To > > > me Kafka has absolutely reached 1.0 in terms of features/functions. > That > > > said, I do share the same opinion with others that the usablity and > > > policies might still need some improvement to meet the 1.0 standard. > > > > > > On Sat, Jul 22, 2017 at 7:21 AM, Matthias J. Sax < > matth...@confluent.io> > > > wrote: > > > > > > > I agree with Ismael. If we go for 1.0 (what I do support), we need to > > > > meet people's expectations and should document all policies and > > > > guarantees explicitly. We might also consider to support older > releases > > > > longer and do bug fix release for older releases, too. > > > > > > > > Other projects (like Flink) do a fantastic job with this regard and > we > > > > should learn from them. > > > > > > > > -Matthias > > > > > > > > On 7/21/17 9:50 PM, Guozhang Wang wrote: > > > > > That's fair enough too. > > > > > > > > > > Guozhang > > > > > > > > > > On Fri, Jul 21, 2017 at 12:13 PM, Ismael Juma <isma...@gmail.com> > > > wrote: > > > > > > > > > >> Yes, I agree that the choice of version number can be done > > separately. > > > > >> That's why I said I'd file a separate JIRA for the documentation > > > > >> improvements. Having said that, there are some expectations that > > > people > > > > >> have for projects that have reached 1.0.0 and we should try to > > > allocate > > > > >> time for the important ones. > > > > >> > > > > >> Ismael > > > > >> > > > > >> On 21 Jul 2017 8:07 pm, "Guozhang Wang" <wangg...@gmail.com> > wrote: > > > > >> > > > > >>> Thanks Ismael. I agree with you on all these points, and for some > > of > > > > >> these > > > > >>> points like 3) we never have a written-down policy though in > > practice > > > > we > > > > >>> tend to follow some patterns. > > > > >>> > > > > >>> To me deciding what's the version number of the next major > release > > > does > > > > >> not > > > > >>> necessarily mean we need now to change any of these or to set the > > > hard > > > > >>> rules along with it; I'd like to keep them as two separate > > > discussions > > > > as > > > > >>> they seem semi-orthogonal to me. > > > > >>> > > > > >>> > > > > >>> Guozhang > > > > >>> > > > > >>> > > > > >>> On Fri, Jul 21, 2017 at 8:44 AM, Ismael Juma <ism...@juma.me.uk> > > > > wrote: > > > > >>> > > > > >>>> On the topic of documentation, we should also document which > > > releases > > > > >> are > > > > >>>> still supported and which are not. There a few factors to > > consider: > > > > >>>> > > > > >>>> 1. Which branches receive bug fixes. We typically backport fixes > > to > > > > the > > > > >>> two > > > > >>>> most recent stable branches (the most recent stable branch > > typically > > > > >> gets > > > > >>>> more backports than the older one). > > > > >>>> > > > > >>>> 2. Which branches receive security fixes. This could be the same > > as > > > > >> `1`, > > > > >>>> but we could attempt to backport more aggressively for security > > > fixes > > > > >> as > > > > >>>> they tend to be rare (so far at least) and the impact could be > > > severe. > > > > >>>> > > > > >>>> 3. The release policy for stable branches. We tend to stop > > releasing > > > > >>> from a > > > > >>>> given stable branch before we stop backporting fixes. Maybe > that's > > > OK, > > > > >>> but > > > > >>>> it would be good to document how we decide that a bug fix > release > > is > > > > >>>> needed. > > > > >>>> > > > > >>>> 4. How long are direct upgrades supported for. During the > > time-based > > > > >>>> releases discussion, we agreed to support direct upgrades for 2 > > > years. > > > > >> As > > > > >>>> it happens, direct upgrades from 0.8.2 to 1.0.0 would not be > > > supported > > > > >> if > > > > >>>> we follow this strictly. Not clear if we want to do that. > > > > >>>> > > > > >>>> 5. How long are older clients supported for. Current brokers > > support > > > > >>>> clients all the way to 0.8.x. > > > > >>>> > > > > >>>> 6. How long are older brokers supported for. Current clients > > support > > > > >>> 0.10.x > > > > >>>> and newer brokers. > > > > >>>> > > > > >>>> 7. How long are message formats supported for. We never > discussed > > > > >> this. I > > > > >>>> think 5 years would probably be the minimum. > > > > >>>> > > > > >>>> Ismael > > > > >>>> > > > > >>>> P.S. I'll file a JIRA to capture this information. > > > > >>>> > > > > >>>> On Wed, Jul 19, 2017 at 10:12 AM, Ismael Juma < > ism...@juma.me.uk> > > > > >> wrote: > > > > >>>> > > > > >>>>> Hi Stevo, > > > > >>>>> > > > > >>>>> Thanks for your feedback. We should definitely do a better job > of > > > > >>>>> documenting things. We basically follow semantic versioning, > but > > > it's > > > > >>>>> currently a bit confusing because: > > > > >>>>> > > > > >>>>> 1. There are 4 segments in the version. The "0." part should be > > > > >> ignored > > > > >>>>> when deciding what is major, minor and patch at the moment, but > > > many > > > > >>>> people > > > > >>>>> don't know this. Once we move to 1.0.0, that problem goes away. > > > > >>>>> > > > > >>>>> 2. To know what is a public API, you must check the Javadoc ( > > > > >>>>> https://kafka.apache.org/0110/javadoc/index.html?org/ > > > > >>>>> apache/kafka/clients/consumer/KafkaConsumer.html). If it's not > > > > >> listed > > > > >>>>> there, it's not public API. Ideally, it would be obvious from > the > > > > >>> package > > > > >>>>> name (i.e. there would be "internals" in the name), but we are > > not > > > > >>> there > > > > >>>>> yet. The exception are the old Scala APIs, but they have all > been > > > > >>>>> deprecated and they will be removed eventually (the old Scala > > > > >> consumers > > > > >>>>> won't be removed until the June 2018 release at the earliest in > > > order > > > > >>> to > > > > >>>>> give people time to migrate). > > > > >>>>> > > > > >>>>> 3. Even though we are following reasonably common practices, we > > > > >> haven't > > > > >>>>> documented them all in one place. It would be great to do it > > during > > > > >> the > > > > >>>>> next release cycle. > > > > >>>>> > > > > >>>>> A few comments below. > > > > >>>>> > > > > >>>>> On Wed, Jul 19, 2017 at 1:31 AM, Stevo Slavić < > ssla...@gmail.com > > > > > > > >>> wrote: > > > > >>>>> > > > > >>>>>> - APIs not labeled or labeled as stable > > > > >>>>>> -- change in major version is only one that can break backward > > > > >>>>>> compatibility (client APIs or behavior) > > > > >>>>>> > > > > >>>>> > > > > >>>>> To clarify, stable APIs should not be changed in an > incompatible > > > way > > > > >>>>> without a deprecation cycle. Independently of whether it's a > > major > > > > >>>> release > > > > >>>>> or not. > > > > >>>>> > > > > >>>>> > > > > >>>>>> -- change in minor version can introduce new features, but not > > > break > > > > >>>>>> backward compatibility > > > > >>>>>> -- change in patch version, is for bug fixes only. > > > > >>>>>> > > > > >>>>> > > > > >>>>> Right, this has been the case for a while already. Also see > > > > >> annotations > > > > >>>>> below. > > > > >>>>> > > > > >>>>> > > > > >>>>>> - APIs labeled as evolving can be broken in backward > > incompatible > > > > >> way > > > > >>> in > > > > >>>>>> any release, but are assumed less likely to be broken compared > > to > > > > >>>> unstable > > > > >>>>>> APIs > > > > >>>>>> - APIs labeled as unstable can be broken in backward > > incompatible > > > > >> way > > > > >>> in > > > > >>>>>> any release, major, minor or patch > > > > >>>>>> > > > > >>>>> > > > > >>>>> The relevant annotations do explain this: > > > > >>>>> > > > > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ > > > > >>>> common/annotation/ > > > > >>>>> InterfaceStability.html > > > > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ > > > > >>>> common/annotation/ > > > > >>>>> InterfaceStability.Stable.html > > > > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ > > > > >>>> common/annotation/ > > > > >>>>> InterfaceStability.Evolving.html > > > > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ > > > > >>>> common/annotation/ > > > > >>>>> InterfaceStability.Unstable.html > > > > >>>>> > > > > >>>>> But we should have a section in our documentation as well. > > > > >>>>> > > > > >>>>> > > > > >>>>>> - deprecated stable APIs are treated as any stable APIs, they > > can > > > be > > > > >>>>>> removed only in major release, are not allowed to be changed > in > > > > >>> backward > > > > >>>>>> incompatible way in either patch or minor version release > > > > >>>>>> > > > > >>>>> > > > > >>>>> Right, but note that stable non-deprecated APIs provide > stronger > > > > >>>>> guarantees in major releases (they can't be changed in an > > > > >> incompatible > > > > >>>> way). > > > > >>>>> > > > > >>>>>> > > > > >>>>>> This means one should be able to upgrade server and > > > recompile/deploy > > > > >>>> apps > > > > >>>>>> with clients to new minor.patch release with dependency > version > > > > >> change > > > > >>>>>> being only change needed and there would be no drama. > > > > >>>>>> > > > > >>>>> > > > > >>>>> That should have been the case for a while as long as you are > > using > > > > >>>> stable > > > > >>>>> public APIs. > > > > >>>>> > > > > >>>>>> > > > > >>>>>> Practice/"features" like protocol version being a parameter, > and > > > > >>>>>> defaulting > > > > >>>>>> to latest so auto updated with dependency update which > > introduces > > > > >> new > > > > >>>>>> protocol/behavior should not be used in public client APIs. To > > > > >> switch > > > > >>>>>> between backward incompatible APIs (contract and behaviors), > > > ideally > > > > >>>> user > > > > >>>>>> should explicitly have to change code and not dependency only, > > but > > > > >> at > > > > >>>>>> least > > > > >>>>>> it should be clearly communicated that there are breaking > > changes > > > to > > > > >>>>>> expect > > > > >>>>>> even with just dependency update by e.g. giving major version > > > > >> release > > > > >>>>>> clear > > > > >>>>>> meaning. If app dependency on Kafka client library minor.patch > > on > > > > >> same > > > > >>>>>> major is updated, and if there's a change in behavior or API > > > > >> requiring > > > > >>>> app > > > > >>>>>> code change - it's a bug. > > > > >>>>>> > > > > >>>>> > > > > >>>>> Hmm, if the protocol bump provides improved behaviour, that is > > not > > > a > > > > >>>>> backwards incompatible change though. So, I don't think I agree > > > with > > > > >>>> this. > > > > >>>>> Of course, > > > > >>>>> it does mean that _downgrading_ may cause loss of > functionality. > > > > >> That's > > > > >>>>> OK, in my opinion. > > > > >>>>> > > > > >>>>> Change introduced contrary to the SLO, is OK to be reported as > > bug. > > > > >>>>>> Everything else is improvement or feature request. > > > > >>>>>> > > > > >>>>>> If this was the case, and 1.0.0 was released today with APIs > as > > > they > > > > >>> are > > > > >>>>>> now, Scala client APIs even though deprecated would not break > > and > > > > >>>> require > > > > >>>>>> refactoring with every 1.* minor/patch release, and would only > > be > > > > >>>> allowed > > > > >>>>>> to be broken or removed in future major release, like 2.0.0 > > > > >>>>>> > > > > >>>>> > > > > >>>>> Yes, that is the plan for any _public_ Scala client APIs that > are > > > > >> still > > > > >>>>> present in 1.0.0. The public Scala client APIs are the producer > > and > > > > >>>>> consumer, basically. Again, we should make this clear in our > > > > >>>> documentation. > > > > >>>>> Note that we have made an effort to keep those APIs compatible > > for > > > > >>> quite > > > > >>>> a > > > > >>>>> while. It sounds like you have had some issues, were they > related > > > to > > > > >>>> usage > > > > >>>>> of internal Admin APIs by any chance (since we didn't have a > > public > > > > >>>>> AdminClient API until very recently)? > > > > >>>>> > > > > >>>>>> > > > > >>>>>> It should be also clear how long is each version supported - > > e.g. > > > if > > > > >>>>>> minor.patch had meaning that there are no backward > incompatible > > > > >>> changes, > > > > >>>>>> it's OK to file a bug only for current major.minor.patch; > > previous > > > > >>> major > > > > >>>>>> and its last minor.patch can only have patches released up to > > some > > > > >>> time > > > > >>>>>> like 1 up to 3 months. > > > > >>>>>> > > > > >>>>> > > > > >>>>> I am not sure I understood this point correctly. Can you please > > > > >>> clarify? > > > > >>>>> > > > > >>>>> If there are changes in release cadence with new versioning, it > > > > >> should > > > > >>> be > > > > >>>>>> clear too. > > > > >>>>>> > > > > >>>>> > > > > >>>>> No changes are planned. We have started time-based releases > less > > > > >> than a > > > > >>>>> year ago and they seem to be going well. > > > > >>>>> > > > > >>>>> Ismael > > > > >>>>> > > > > >>>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> -- > > > > >>> -- Guozhang > > > > >>> > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > -- Guozhang > > >
