Re: [DISCUSS] Time-based releases for Apache Kafka

[Neha Narkhede]

Thanks for volunteering, Jason!

On Wed, Sep 7, 2016 at 1:59 AM Ismael Juma <ism...@juma.me.uk> wrote:

> Thanks for volunteering Jason. Sounds good to me,
>
> Ismael
>
> On Wed, Sep 7, 2016 at 4:22 AM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hey All,
> >
> > It sounds like the general consensus is in favor of time-based releases.
> We
> > can continue the discussion about LTS, but I wanted to go ahead and get
> > things moving forward by volunteering to manage the next release, which
> is
> > currently slated for October. If that sounds OK, I'll draft a release
> plan
> > and send it out to the community for feedback and a vote.
> >
> > Thanks,
> > Jason
> >
> > On Thu, Aug 25, 2016 at 2:03 PM, Ofir Manor <ofir.ma...@equalum.io>
> wrote:
> >
> > > I happily agree that Kafka is a solid and the community is great :)
> > > But I think there is a gap in perception here.
> > > For me, LTS means that someone is actively taking care of a release -
> > > actively backporting critical fixes (security, stability, data loss,
> > > corruption, hangs etc) from trunk to that LTS version periodically for
> an
> > > extended period of time, for example 18-36 months... So people can
> really
> > > rely on the same Kafka version for a long time.
> > > Is someone doing it today for 0.9.0? When is 0.9.0.2 expected? When is
> > > 0.8.2.3 expected? Will they cover all known critical issues for whoever
> > > relies on them in production?
> > > In other words, what is the scope of support that the community want to
> > > commit for older versions? (upgrade compatibility? investigating bug
> > > reports? proactively backporting fixes?)
> > > BTW, another legit option is that the Apache Kafka project won't commit
> > to
> > > LTS releases. It could let commercial vendors compete on supporting
> very
> > > old versions. I find that actually quite reasonable as well.
> > >
> > > Ofir Manor
> > >
> > > Co-Founder & CTO | Equalum
> > >
> > > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
> > >
> > > On Thu, Aug 25, 2016 at 8:19 PM, Andrew Schofield <
> > > andrew_schofield_j...@outlook.com> wrote:
> > >
> > > > I agree that the Kafka community has managed to maintain a very high
> > > > quality level, so I'm not concerned
> > > > about the quality of non-LTS releases. If the principle is that every
> > > > release is supported for 2 years, that
> > > > would be good. I suppose that if the burden of having that many
> > > in-support
> > > > releases proves too heavy,
> > > > as you say we could reconsider.
> > > >
> > > > Andrew Schofield
> > > >
> > > > 
> > > > > From: g...@confluent.io
> > > > > Date: Thu, 25 Aug 2016 09:57:30 -0700
> > > > > Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> > > > > To: dev@kafka.apache.org
> > > > >
> > > > > I prefer Ismael's suggestion for supporting 2-years (6 releases)
> > > > > rather than have designated LTS releases.
> > > > >
> > > > > The LTS model seems to work well when some releases are high
> quality
> > > > > (LTS) and the rest are a bit more questionable. It is great for
> > > > > companies like Redhat, where they have to invest less to support
> few
> > > > > releases and let the community deal with everything else.
> > > > >
> > > > > Until now the Kafka community has managed to maintain very high
> > > > > quality level. Not just for releases, our trunk is often of better
> > > > > quality than other project's releases - we don't think of stability
> > as
> > > > > something you tuck into a release (and just some releases) but
> rather
> > > > > as an on-going concern. There are costs to doing things that way,
> but
> > > > > in general, I think it has served us well - allowing even
> > conservative
> > > > > companies to run on the latest released version.
> > > > >
> > > > > I hope we can agree to at least try maintaining last 6 releases as
> > LTS
> > > > > (i.e. every single release is supported for 2 years) rather than
> > > > > designate some releases as better than others. Of course, if this
> >

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

Thanks for volunteering Jason. Sounds good to me,

Ismael

On Wed, Sep 7, 2016 at 4:22 AM, Jason Gustafson <ja...@confluent.io> wrote:

> Hey All,
>
> It sounds like the general consensus is in favor of time-based releases. We
> can continue the discussion about LTS, but I wanted to go ahead and get
> things moving forward by volunteering to manage the next release, which is
> currently slated for October. If that sounds OK, I'll draft a release plan
> and send it out to the community for feedback and a vote.
>
> Thanks,
> Jason
>
> On Thu, Aug 25, 2016 at 2:03 PM, Ofir Manor <ofir.ma...@equalum.io> wrote:
>
> > I happily agree that Kafka is a solid and the community is great :)
> > But I think there is a gap in perception here.
> > For me, LTS means that someone is actively taking care of a release -
> > actively backporting critical fixes (security, stability, data loss,
> > corruption, hangs etc) from trunk to that LTS version periodically for an
> > extended period of time, for example 18-36 months... So people can really
> > rely on the same Kafka version for a long time.
> > Is someone doing it today for 0.9.0? When is 0.9.0.2 expected? When is
> > 0.8.2.3 expected? Will they cover all known critical issues for whoever
> > relies on them in production?
> > In other words, what is the scope of support that the community want to
> > commit for older versions? (upgrade compatibility? investigating bug
> > reports? proactively backporting fixes?)
> > BTW, another legit option is that the Apache Kafka project won't commit
> to
> > LTS releases. It could let commercial vendors compete on supporting very
> > old versions. I find that actually quite reasonable as well.
> >
> > Ofir Manor
> >
> > Co-Founder & CTO | Equalum
> >
> > Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
> >
> > On Thu, Aug 25, 2016 at 8:19 PM, Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> > > I agree that the Kafka community has managed to maintain a very high
> > > quality level, so I'm not concerned
> > > about the quality of non-LTS releases. If the principle is that every
> > > release is supported for 2 years, that
> > > would be good. I suppose that if the burden of having that many
> > in-support
> > > releases proves too heavy,
> > > as you say we could reconsider.
> > >
> > > Andrew Schofield
> > >
> > > 
> > > > From: g...@confluent.io
> > > > Date: Thu, 25 Aug 2016 09:57:30 -0700
> > > > Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> > > > To: dev@kafka.apache.org
> > > >
> > > > I prefer Ismael's suggestion for supporting 2-years (6 releases)
> > > > rather than have designated LTS releases.
> > > >
> > > > The LTS model seems to work well when some releases are high quality
> > > > (LTS) and the rest are a bit more questionable. It is great for
> > > > companies like Redhat, where they have to invest less to support few
> > > > releases and let the community deal with everything else.
> > > >
> > > > Until now the Kafka community has managed to maintain very high
> > > > quality level. Not just for releases, our trunk is often of better
> > > > quality than other project's releases - we don't think of stability
> as
> > > > something you tuck into a release (and just some releases) but rather
> > > > as an on-going concern. There are costs to doing things that way, but
> > > > in general, I think it has served us well - allowing even
> conservative
> > > > companies to run on the latest released version.
> > > >
> > > > I hope we can agree to at least try maintaining last 6 releases as
> LTS
> > > > (i.e. every single release is supported for 2 years) rather than
> > > > designate some releases as better than others. Of course, if this
> > > > totally fails, we can reconsider.
> > > >
> > > > Gwen
> > > >
> > > > On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
> > > > <andrew_schofield_j...@outlook.com> wrote:
> > > >> The proposal sounds pretty good, but the main thing currently
> missing
> > > is a proper long-term support release.
> > > >>
> > > >> Having 3 releases a year sounds OK, but if they're all equivalent
> and
> > > bugfix releases are produced for the most
>

Re: [DISCUSS] Time-based releases for Apache Kafka

[Sriram Subramanian]

Sounds good to me. 

> On Sep 6, 2016, at 8:22 PM, Jason Gustafson <ja...@confluent.io> wrote:
> 
> Hey All,
> 
> It sounds like the general consensus is in favor of time-based releases. We
> can continue the discussion about LTS, but I wanted to go ahead and get
> things moving forward by volunteering to manage the next release, which is
> currently slated for October. If that sounds OK, I'll draft a release plan
> and send it out to the community for feedback and a vote.
> 
> Thanks,
> Jason
> 
>> On Thu, Aug 25, 2016 at 2:03 PM, Ofir Manor <ofir.ma...@equalum.io> wrote:
>> 
>> I happily agree that Kafka is a solid and the community is great :)
>> But I think there is a gap in perception here.
>> For me, LTS means that someone is actively taking care of a release -
>> actively backporting critical fixes (security, stability, data loss,
>> corruption, hangs etc) from trunk to that LTS version periodically for an
>> extended period of time, for example 18-36 months... So people can really
>> rely on the same Kafka version for a long time.
>> Is someone doing it today for 0.9.0? When is 0.9.0.2 expected? When is
>> 0.8.2.3 expected? Will they cover all known critical issues for whoever
>> relies on them in production?
>> In other words, what is the scope of support that the community want to
>> commit for older versions? (upgrade compatibility? investigating bug
>> reports? proactively backporting fixes?)
>> BTW, another legit option is that the Apache Kafka project won't commit to
>> LTS releases. It could let commercial vendors compete on supporting very
>> old versions. I find that actually quite reasonable as well.
>> 
>> Ofir Manor
>> 
>> Co-Founder & CTO | Equalum
>> 
>> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>> 
>> On Thu, Aug 25, 2016 at 8:19 PM, Andrew Schofield <
>> andrew_schofield_j...@outlook.com> wrote:
>> 
>>> I agree that the Kafka community has managed to maintain a very high
>>> quality level, so I'm not concerned
>>> about the quality of non-LTS releases. If the principle is that every
>>> release is supported for 2 years, that
>>> would be good. I suppose that if the burden of having that many
>> in-support
>>> releases proves too heavy,
>>> as you say we could reconsider.
>>> 
>>> Andrew Schofield
>>> 
>>> 
>>>> From: g...@confluent.io
>>>> Date: Thu, 25 Aug 2016 09:57:30 -0700
>>>> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
>>>> To: dev@kafka.apache.org
>>>> 
>>>> I prefer Ismael's suggestion for supporting 2-years (6 releases)
>>>> rather than have designated LTS releases.
>>>> 
>>>> The LTS model seems to work well when some releases are high quality
>>>> (LTS) and the rest are a bit more questionable. It is great for
>>>> companies like Redhat, where they have to invest less to support few
>>>> releases and let the community deal with everything else.
>>>> 
>>>> Until now the Kafka community has managed to maintain very high
>>>> quality level. Not just for releases, our trunk is often of better
>>>> quality than other project's releases - we don't think of stability as
>>>> something you tuck into a release (and just some releases) but rather
>>>> as an on-going concern. There are costs to doing things that way, but
>>>> in general, I think it has served us well - allowing even conservative
>>>> companies to run on the latest released version.
>>>> 
>>>> I hope we can agree to at least try maintaining last 6 releases as LTS
>>>> (i.e. every single release is supported for 2 years) rather than
>>>> designate some releases as better than others. Of course, if this
>>>> totally fails, we can reconsider.
>>>> 
>>>> Gwen
>>>> 
>>>> On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
>>>> <andrew_schofield_j...@outlook.com> wrote:
>>>>> The proposal sounds pretty good, but the main thing currently missing
>>> is a proper long-term support release.
>>>>> 
>>>>> Having 3 releases a year sounds OK, but if they're all equivalent and
>>> bugfix releases are produced for the most
>>>>> recent 2 or 3 releases, anyone wanting to run on an "in support"
>>> release of Kafka has to upgrade every 8-12 months.
>>>>> If you don't ac

Re: [DISCUSS] Time-based releases for Apache Kafka

[Jason Gustafson]

Hey All,

It sounds like the general consensus is in favor of time-based releases. We
can continue the discussion about LTS, but I wanted to go ahead and get
things moving forward by volunteering to manage the next release, which is
currently slated for October. If that sounds OK, I'll draft a release plan
and send it out to the community for feedback and a vote.

Thanks,
Jason

On Thu, Aug 25, 2016 at 2:03 PM, Ofir Manor <ofir.ma...@equalum.io> wrote:

> I happily agree that Kafka is a solid and the community is great :)
> But I think there is a gap in perception here.
> For me, LTS means that someone is actively taking care of a release -
> actively backporting critical fixes (security, stability, data loss,
> corruption, hangs etc) from trunk to that LTS version periodically for an
> extended period of time, for example 18-36 months... So people can really
> rely on the same Kafka version for a long time.
> Is someone doing it today for 0.9.0? When is 0.9.0.2 expected? When is
> 0.8.2.3 expected? Will they cover all known critical issues for whoever
> relies on them in production?
> In other words, what is the scope of support that the community want to
> commit for older versions? (upgrade compatibility? investigating bug
> reports? proactively backporting fixes?)
> BTW, another legit option is that the Apache Kafka project won't commit to
> LTS releases. It could let commercial vendors compete on supporting very
> old versions. I find that actually quite reasonable as well.
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
> On Thu, Aug 25, 2016 at 8:19 PM, Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
> > I agree that the Kafka community has managed to maintain a very high
> > quality level, so I'm not concerned
> > about the quality of non-LTS releases. If the principle is that every
> > release is supported for 2 years, that
> > would be good. I suppose that if the burden of having that many
> in-support
> > releases proves too heavy,
> > as you say we could reconsider.
> >
> > Andrew Schofield
> >
> > --------
> > > From: g...@confluent.io
> > > Date: Thu, 25 Aug 2016 09:57:30 -0700
> > > Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> > > To: dev@kafka.apache.org
> > >
> > > I prefer Ismael's suggestion for supporting 2-years (6 releases)
> > > rather than have designated LTS releases.
> > >
> > > The LTS model seems to work well when some releases are high quality
> > > (LTS) and the rest are a bit more questionable. It is great for
> > > companies like Redhat, where they have to invest less to support few
> > > releases and let the community deal with everything else.
> > >
> > > Until now the Kafka community has managed to maintain very high
> > > quality level. Not just for releases, our trunk is often of better
> > > quality than other project's releases - we don't think of stability as
> > > something you tuck into a release (and just some releases) but rather
> > > as an on-going concern. There are costs to doing things that way, but
> > > in general, I think it has served us well - allowing even conservative
> > > companies to run on the latest released version.
> > >
> > > I hope we can agree to at least try maintaining last 6 releases as LTS
> > > (i.e. every single release is supported for 2 years) rather than
> > > designate some releases as better than others. Of course, if this
> > > totally fails, we can reconsider.
> > >
> > > Gwen
> > >
> > > On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
> > > <andrew_schofield_j...@outlook.com> wrote:
> > >> The proposal sounds pretty good, but the main thing currently missing
> > is a proper long-term support release.
> > >>
> > >> Having 3 releases a year sounds OK, but if they're all equivalent and
> > bugfix releases are produced for the most
> > >> recent 2 or 3 releases, anyone wanting to run on an "in support"
> > release of Kafka has to upgrade every 8-12 months.
> > >> If you don't actually want anything specific from the newer releases,
> > it's just unnecessary churn.
> > >>
> > >> Wouldn't it be better to designate one release every 12-18 months as a
> > long-term support release with bugfix releases
> > >> produced for those for a longer period of say 24 months. That halves
> > the upgrade work for people just wanting to ke

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ofir Manor]

I happily agree that Kafka is a solid and the community is great :)
But I think there is a gap in perception here.
For me, LTS means that someone is actively taking care of a release -
actively backporting critical fixes (security, stability, data loss,
corruption, hangs etc) from trunk to that LTS version periodically for an
extended period of time, for example 18-36 months... So people can really
rely on the same Kafka version for a long time.
Is someone doing it today for 0.9.0? When is 0.9.0.2 expected? When is
0.8.2.3 expected? Will they cover all known critical issues for whoever
relies on them in production?
In other words, what is the scope of support that the community want to
commit for older versions? (upgrade compatibility? investigating bug
reports? proactively backporting fixes?)
BTW, another legit option is that the Apache Kafka project won't commit to
LTS releases. It could let commercial vendors compete on supporting very
old versions. I find that actually quite reasonable as well.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Thu, Aug 25, 2016 at 8:19 PM, Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> I agree that the Kafka community has managed to maintain a very high
> quality level, so I'm not concerned
> about the quality of non-LTS releases. If the principle is that every
> release is supported for 2 years, that
> would be good. I suppose that if the burden of having that many in-support
> releases proves too heavy,
> as you say we could reconsider.
>
> Andrew Schofield
>
> 
> > From: g...@confluent.io
> > Date: Thu, 25 Aug 2016 09:57:30 -0700
> > Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> > To: dev@kafka.apache.org
> >
> > I prefer Ismael's suggestion for supporting 2-years (6 releases)
> > rather than have designated LTS releases.
> >
> > The LTS model seems to work well when some releases are high quality
> > (LTS) and the rest are a bit more questionable. It is great for
> > companies like Redhat, where they have to invest less to support few
> > releases and let the community deal with everything else.
> >
> > Until now the Kafka community has managed to maintain very high
> > quality level. Not just for releases, our trunk is often of better
> > quality than other project's releases - we don't think of stability as
> > something you tuck into a release (and just some releases) but rather
> > as an on-going concern. There are costs to doing things that way, but
> > in general, I think it has served us well - allowing even conservative
> > companies to run on the latest released version.
> >
> > I hope we can agree to at least try maintaining last 6 releases as LTS
> > (i.e. every single release is supported for 2 years) rather than
> > designate some releases as better than others. Of course, if this
> > totally fails, we can reconsider.
> >
> > Gwen
> >
> > On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
> > <andrew_schofield_j...@outlook.com> wrote:
> >> The proposal sounds pretty good, but the main thing currently missing
> is a proper long-term support release.
> >>
> >> Having 3 releases a year sounds OK, but if they're all equivalent and
> bugfix releases are produced for the most
> >> recent 2 or 3 releases, anyone wanting to run on an "in support"
> release of Kafka has to upgrade every 8-12 months.
> >> If you don't actually want anything specific from the newer releases,
> it's just unnecessary churn.
> >>
> >> Wouldn't it be better to designate one release every 12-18 months as a
> long-term support release with bugfix releases
> >> produced for those for a longer period of say 24 months. That halves
> the upgrade work for people just wanting to keep
> >> "in support". Now that adoption is increasing, there are plenty of
> users that just want a dependable messaging system
> >> without having to be deeply knowledgeable about its innards.
> >>
> >> LTS works nicely for plenty of open-source projects. I think it would
> work well for Kafka too.
> >>
> >> Andrew Schofield
> >>
> >> 
> >>> From: ofir.ma...@equalum.io
> >>> Date: Thu, 25 Aug 2016 16:07:07 +0300
> >>> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> >>> To: dev@kafka.apache.org
> >>>
> >>> Regarding bug fixes, you may want to consider to have an LTS release
> once a
> >>> year - designating it for longer-term support / better for 

RE: [DISCUSS] Time-based releases for Apache Kafka

[Andrew Schofield]

I agree that the Kafka community has managed to maintain a very high quality 
level, so I'm not concerned
about the quality of non-LTS releases. If the principle is that every release 
is supported for 2 years, that
would be good. I suppose that if the burden of having that many in-support 
releases proves too heavy,
as you say we could reconsider.

Andrew Schofield


> From: g...@confluent.io
> Date: Thu, 25 Aug 2016 09:57:30 -0700
> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> To: dev@kafka.apache.org
>
> I prefer Ismael's suggestion for supporting 2-years (6 releases)
> rather than have designated LTS releases.
>
> The LTS model seems to work well when some releases are high quality
> (LTS) and the rest are a bit more questionable. It is great for
> companies like Redhat, where they have to invest less to support few
> releases and let the community deal with everything else.
>
> Until now the Kafka community has managed to maintain very high
> quality level. Not just for releases, our trunk is often of better
> quality than other project's releases - we don't think of stability as
> something you tuck into a release (and just some releases) but rather
> as an on-going concern. There are costs to doing things that way, but
> in general, I think it has served us well - allowing even conservative
> companies to run on the latest released version.
>
> I hope we can agree to at least try maintaining last 6 releases as LTS
> (i.e. every single release is supported for 2 years) rather than
> designate some releases as better than others. Of course, if this
> totally fails, we can reconsider.
>
> Gwen
>
> On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
> <andrew_schofield_j...@outlook.com> wrote:
>> The proposal sounds pretty good, but the main thing currently missing is a 
>> proper long-term support release.
>>
>> Having 3 releases a year sounds OK, but if they're all equivalent and bugfix 
>> releases are produced for the most
>> recent 2 or 3 releases, anyone wanting to run on an "in support" release of 
>> Kafka has to upgrade every 8-12 months.
>> If you don't actually want anything specific from the newer releases, it's 
>> just unnecessary churn.
>>
>> Wouldn't it be better to designate one release every 12-18 months as a 
>> long-term support release with bugfix releases
>> produced for those for a longer period of say 24 months. That halves the 
>> upgrade work for people just wanting to keep
>> "in support". Now that adoption is increasing, there are plenty of users 
>> that just want a dependable messaging system
>> without having to be deeply knowledgeable about its innards.
>>
>> LTS works nicely for plenty of open-source projects. I think it would work 
>> well for Kafka too.
>>
>> Andrew Schofield
>>
>> 
>>> From: ofir.ma...@equalum.io
>>> Date: Thu, 25 Aug 2016 16:07:07 +0300
>>> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
>>> To: dev@kafka.apache.org
>>>
>>> Regarding bug fixes, you may want to consider to have an LTS release once a
>>> year - designating it for longer-term support / better for the masses.
>>> If you like that - then fix bugs in trunk, backport important ones to
>>> latest release + the last two LTS releases.
>>> Even if you don't - if a downstream distribution picks a Kafka version and
>>> plans to support it over a few years, it could be nice of them to "own"
>>> that older release - volunteer to be a release manager for bug backports to
>>> that version over a longer period...
>>> Just my two cents :)
>>>
>>> Ofir Manor
>>>
>>> Co-Founder & CTO | Equalum
>>>
>>> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>>>
>>> On Thu, Aug 25, 2016 at 12:32 PM, Ismael Juma <ism...@juma.me.uk> wrote:
>>>
>>>> Thanks for putting this together Gwen. I think it sounds reasonable and
>>>> instead of trying to optimise every aspect of it ahead of time (which is
>>>> hard, subjective and time-consuming), I am happy to try what is being
>>>> proposed and tweak based on experience. One thing we should pay particular
>>>> attention to is how the stabilisation process works out in practice.
>>>>
>>>> A couple of comments:
>>>>
>>>> "Given 3 releases a year and the fact that no one upgrades three times a
>>>> year, we propose making sure (by testing!) that rolling upgrade c

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

I prefer Ismael's suggestion for supporting 2-years (6 releases)
rather than have designated LTS releases.

The LTS model seems to work well when some releases are high quality
(LTS) and the rest are a bit more questionable. It is great for
companies like Redhat, where they have to invest less to support few
releases and let the community deal with everything else.

Until now the Kafka community has managed to maintain very high
quality level. Not just for releases, our trunk is often of better
quality than other project's releases - we don't think of stability as
something you tuck into a release (and just some releases) but rather
as an on-going concern. There are costs to doing things that way, but
in general, I think it has served us well - allowing even conservative
companies to run on the latest released version.

I hope we can agree to at least try maintaining last 6 releases as LTS
(i.e. every single release is supported for 2 years) rather than
designate some releases as better than others. Of course, if this
totally fails, we can reconsider.

Gwen

On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
<andrew_schofield_j...@outlook.com> wrote:
> The proposal sounds pretty good, but the main thing currently missing is a 
> proper long-term support release.
>
> Having 3 releases a year sounds OK, but if they're all equivalent and bugfix 
> releases are produced for the most
> recent 2 or 3 releases, anyone wanting to run on an "in support" release of 
> Kafka has to upgrade every 8-12 months.
> If you don't actually want anything specific from the newer releases, it's 
> just unnecessary churn.
>
> Wouldn't it be better to designate one release every 12-18 months as a 
> long-term support release with bugfix releases
> produced for those for a longer period of say 24 months. That halves the 
> upgrade work for people just wanting to keep
> "in support". Now that adoption is increasing, there are plenty of users that 
> just want a dependable messaging system
> without having to be deeply knowledgeable about its innards.
>
> LTS works nicely for plenty of open-source projects. I think it would work 
> well for Kafka too.
>
> Andrew Schofield
>
> ----
>> From: ofir.ma...@equalum.io
>> Date: Thu, 25 Aug 2016 16:07:07 +0300
>> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
>> To: dev@kafka.apache.org
>>
>> Regarding bug fixes, you may want to consider to have an LTS release once a
>> year - designating it for longer-term support / better for the masses.
>> If you like that - then fix bugs in trunk, backport important ones to
>> latest release + the last two LTS releases.
>> Even if you don't - if a downstream distribution picks a Kafka version and
>> plans to support it over a few years, it could be nice of them to "own"
>> that older release - volunteer to be a release manager for bug backports to
>> that version over a longer period...
>> Just my two cents :)
>>
>> Ofir Manor
>>
>> Co-Founder & CTO | Equalum
>>
>> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>>
>> On Thu, Aug 25, 2016 at 12:32 PM, Ismael Juma <ism...@juma.me.uk> wrote:
>>
>>> Thanks for putting this together Gwen. I think it sounds reasonable and
>>> instead of trying to optimise every aspect of it ahead of time (which is
>>> hard, subjective and time-consuming), I am happy to try what is being
>>> proposed and tweak based on experience. One thing we should pay particular
>>> attention to is how the stabilisation process works out in practice.
>>>
>>> A couple of comments:
>>>
>>> "Given 3 releases a year and the fact that no one upgrades three times a
>>> year, we propose making sure (by testing!) that rolling upgrade can be done
>>> from each release in the past year (i.e. last 3 releases) to the latest
>>> version."
>>>
>>> Because the cost of doing this for a larger number of releases is
>>> relatively low, I still think we should go for 6 here (our code currently
>>> supports 5 versions as I said in a previous message, so we're close to that
>>> target already). I'm generally very keen to make upgrades as easy as
>>> possible so that people have no reason not to upgrade. :)
>>>
>>> "We will also attempt, as a community to do bugfix releases as needed for
>>> the last 3 releases."
>>>
>>> I would suggest 2, personally, but since this is a bit fuzzy, I am OK with
>>> 3 if people prefer that.
>>>
>>> Ismael
>>>
>>> On Thu, Aug 25, 2016 at 6:22 AM, Gwen S

RE: [DISCUSS] Time-based releases for Apache Kafka

[Andrew Schofield]

The proposal sounds pretty good, but the main thing currently missing is a 
proper long-term support release.

Having 3 releases a year sounds OK, but if they're all equivalent and bugfix 
releases are produced for the most
recent 2 or 3 releases, anyone wanting to run on an "in support" release of 
Kafka has to upgrade every 8-12 months.
If you don't actually want anything specific from the newer releases, it's just 
unnecessary churn.

Wouldn't it be better to designate one release every 12-18 months as a 
long-term support release with bugfix releases
produced for those for a longer period of say 24 months. That halves the 
upgrade work for people just wanting to keep
"in support". Now that adoption is increasing, there are plenty of users that 
just want a dependable messaging system
without having to be deeply knowledgeable about its innards.

LTS works nicely for plenty of open-source projects. I think it would work well 
for Kafka too.

Andrew Schofield


> From: ofir.ma...@equalum.io
> Date: Thu, 25 Aug 2016 16:07:07 +0300
> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
> To: dev@kafka.apache.org
>
> Regarding bug fixes, you may want to consider to have an LTS release once a
> year - designating it for longer-term support / better for the masses.
> If you like that - then fix bugs in trunk, backport important ones to
> latest release + the last two LTS releases.
> Even if you don't - if a downstream distribution picks a Kafka version and
> plans to support it over a few years, it could be nice of them to "own"
> that older release - volunteer to be a release manager for bug backports to
> that version over a longer period...
> Just my two cents :)
>
> Ofir Manor
>
> Co-Founder & CTO | Equalum
>
> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>
> On Thu, Aug 25, 2016 at 12:32 PM, Ismael Juma <ism...@juma.me.uk> wrote:
>
>> Thanks for putting this together Gwen. I think it sounds reasonable and
>> instead of trying to optimise every aspect of it ahead of time (which is
>> hard, subjective and time-consuming), I am happy to try what is being
>> proposed and tweak based on experience. One thing we should pay particular
>> attention to is how the stabilisation process works out in practice.
>>
>> A couple of comments:
>>
>> "Given 3 releases a year and the fact that no one upgrades three times a
>> year, we propose making sure (by testing!) that rolling upgrade can be done
>> from each release in the past year (i.e. last 3 releases) to the latest
>> version."
>>
>> Because the cost of doing this for a larger number of releases is
>> relatively low, I still think we should go for 6 here (our code currently
>> supports 5 versions as I said in a previous message, so we're close to that
>> target already). I'm generally very keen to make upgrades as easy as
>> possible so that people have no reason not to upgrade. :)
>>
>> "We will also attempt, as a community to do bugfix releases as needed for
>> the last 3 releases."
>>
>> I would suggest 2, personally, but since this is a bit fuzzy, I am OK with
>> 3 if people prefer that.
>>
>> Ismael
>>
>> On Thu, Aug 25, 2016 at 6:22 AM, Gwen Shapira <g...@confluent.io> wrote:
>>
>>> Hi Team Kafka,
>>>
>>> As per the KIP meeting, I created a wiki:
>>> https://cwiki.apache.org/confluence/display/KAFKA/Time+
>> Based+Release+Plan
>>> Summarizing most of the discussion so far.
>>>
>>> Comments and additional discussion is welcome :)
>>>
>>> Gwen
>>>
>>> On Wed, Aug 17, 2016 at 12:31 PM, Vahid S Hashemian
>>> <vahidhashem...@us.ibm.com> wrote:
>>>> Time-based releases is a good idea and something that has proved to be
>>>> working in a number of open source projects. One successful example is
>>>> Node.js, that goes through two major releases a year. The interesting
>>> fact
>>>> about the two releases is that only one (the even-number release) comes
>>>> with a long term support (LTS) plan (30 months). More can be read here:
>>>> https://github.com/nodejs/LTS. The odd-number releases still come with
>>>> major changes and help build the ecosystem, but as far as LTS goes,
>> there
>>>> is only one per year. This LTS plan makes most enterprises want to
>> stick
>>>> to even-number releases, which is okay since frequent upgrades is not
>>>> something they are normally interested in anyway.
>>>>
>>>> There could be several m

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ofir Manor]

Regarding bug fixes, you may want to consider to have an LTS release once a
year - designating it for longer-term support / better for the masses.
If you like that - then fix bugs in trunk, backport important ones to
latest release + the last two LTS releases.
Even if you don't - if a downstream distribution picks a Kafka version and
plans to support it over a few years, it could be nice of them to "own"
that older release - volunteer to be a release manager for bug backports to
that version over a longer period...
Just my two cents :)

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io

On Thu, Aug 25, 2016 at 12:32 PM, Ismael Juma <ism...@juma.me.uk> wrote:

> Thanks for putting this together Gwen. I think it sounds reasonable and
> instead of trying to optimise every aspect of it ahead of time (which is
> hard, subjective and time-consuming), I am happy to try what is being
> proposed and tweak based on experience. One thing we should pay particular
> attention to is how the stabilisation process works out in practice.
>
> A couple of comments:
>
> "Given 3 releases a year and the fact that no one upgrades three times a
> year, we propose making sure (by testing!) that rolling upgrade can be done
> from each release in the past year (i.e. last 3 releases) to the latest
> version."
>
> Because the cost of doing this for a larger number of releases is
> relatively low, I still think we should go for 6 here (our code currently
> supports 5 versions as I said in a previous message, so we're close to that
> target already). I'm generally very keen to make upgrades as easy as
> possible so that people have no reason not to upgrade. :)
>
> "We will also attempt, as a community to do bugfix releases as needed for
> the last 3 releases."
>
> I would suggest 2, personally, but since this is a bit fuzzy, I am OK with
> 3 if people prefer that.
>
> Ismael
>
> On Thu, Aug 25, 2016 at 6:22 AM, Gwen Shapira <g...@confluent.io> wrote:
>
> > Hi Team Kafka,
> >
> > As per the KIP meeting, I created a wiki:
> > https://cwiki.apache.org/confluence/display/KAFKA/Time+
> Based+Release+Plan
> > Summarizing most of the discussion so far.
> >
> > Comments and additional discussion is welcome :)
> >
> > Gwen
> >
> > On Wed, Aug 17, 2016 at 12:31 PM, Vahid S Hashemian
> > <vahidhashem...@us.ibm.com> wrote:
> > > Time-based releases is a good idea and something that has proved to be
> > > working in a number of open source projects. One successful example is
> > > Node.js, that goes through two major releases a year. The interesting
> > fact
> > > about the two releases is that only one (the even-number release) comes
> > > with a long term support (LTS) plan (30 months). More can be read here:
> > > https://github.com/nodejs/LTS. The odd-number releases still come with
> > > major changes and help build the ecosystem, but as far as LTS goes,
> there
> > > is only one per year. This LTS plan makes most enterprises want to
> stick
> > > to even-number releases, which is okay since frequent upgrades is not
> > > something they are normally interested in anyway.
> > >
> > > There could be several minor releases (non-breaking) in between major
> > > releases. A major release contains all the features / bug fixes in the
> > > master branch a month before the release date, with the potential
> > addition
> > > of (non-breaking) bug fixes until the release day. The deprecation
> cycle
> > > is one major release: any functionality that is decided to be removed
> is
> > > deprecated in the next major release, and removed in the major release
> > > after that.
> > >
> > > Because of the success of LTS models in this and other open source
> > > projects, I would suggest implementing a formal LTS plan for Kafka too.
> > >
> > > Regards,
> > > --Vahid
> > >
> > >
> > >
> > > From:   Gwen Shapira <g...@confluent.io>
> > > To: dev@kafka.apache.org
> > > Date:   08/09/2016 04:49 PM
> > > Subject:[DISCUSS] Time-based releases for Apache Kafka
> > >
> > >
> > >
> > > Dear Kafka Developers and Users,
> > >
> > > In the past, our releases have been quite unpredictable. We'll notice
> > > that a large number of nice features made it in (or are close),
> > > someone would suggest a release and we'd do it. This is fun, but makes
> > > planning really hard - we saw it during the last release which we

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

Thanks for putting this together Gwen. I think it sounds reasonable and
instead of trying to optimise every aspect of it ahead of time (which is
hard, subjective and time-consuming), I am happy to try what is being
proposed and tweak based on experience. One thing we should pay particular
attention to is how the stabilisation process works out in practice.

A couple of comments:

"Given 3 releases a year and the fact that no one upgrades three times a
year, we propose making sure (by testing!) that rolling upgrade can be done
from each release in the past year (i.e. last 3 releases) to the latest
version."

Because the cost of doing this for a larger number of releases is
relatively low, I still think we should go for 6 here (our code currently
supports 5 versions as I said in a previous message, so we're close to that
target already). I'm generally very keen to make upgrades as easy as
possible so that people have no reason not to upgrade. :)

"We will also attempt, as a community to do bugfix releases as needed for
the last 3 releases."

I would suggest 2, personally, but since this is a bit fuzzy, I am OK with
3 if people prefer that.

Ismael

On Thu, Aug 25, 2016 at 6:22 AM, Gwen Shapira <g...@confluent.io> wrote:

> Hi Team Kafka,
>
> As per the KIP meeting, I created a wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> Summarizing most of the discussion so far.
>
> Comments and additional discussion is welcome :)
>
> Gwen
>
> On Wed, Aug 17, 2016 at 12:31 PM, Vahid S Hashemian
> <vahidhashem...@us.ibm.com> wrote:
> > Time-based releases is a good idea and something that has proved to be
> > working in a number of open source projects. One successful example is
> > Node.js, that goes through two major releases a year. The interesting
> fact
> > about the two releases is that only one (the even-number release) comes
> > with a long term support (LTS) plan (30 months). More can be read here:
> > https://github.com/nodejs/LTS. The odd-number releases still come with
> > major changes and help build the ecosystem, but as far as LTS goes, there
> > is only one per year. This LTS plan makes most enterprises want to stick
> > to even-number releases, which is okay since frequent upgrades is not
> > something they are normally interested in anyway.
> >
> > There could be several minor releases (non-breaking) in between major
> > releases. A major release contains all the features / bug fixes in the
> > master branch a month before the release date, with the potential
> addition
> > of (non-breaking) bug fixes until the release day. The deprecation cycle
> > is one major release: any functionality that is decided to be removed is
> > deprecated in the next major release, and removed in the major release
> > after that.
> >
> > Because of the success of LTS models in this and other open source
> > projects, I would suggest implementing a formal LTS plan for Kafka too.
> >
> > Regards,
> > --Vahid
> >
> >
> >
> > From:   Gwen Shapira <g...@confluent.io>
> > To: dev@kafka.apache.org
> > Date:   08/09/2016 04:49 PM
> > Subject:[DISCUSS] Time-based releases for Apache Kafka
> >
> >
> >
> > Dear Kafka Developers and Users,
> >
> > In the past, our releases have been quite unpredictable. We'll notice
> > that a large number of nice features made it in (or are close),
> > someone would suggest a release and we'd do it. This is fun, but makes
> > planning really hard - we saw it during the last release which we
> > decided to delay by a few weeks to allow more features to "land".
> >
> > Many other communities have adopted time-based releases successfully
> > (Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
> > will make sense for the Apache Kafka community to try doing the same.
> >
> > The benefits of this approach are:
> >
> > 1. A quicker feedback cycle and users can benefit from features
> > quicker (assuming for reasonably short time between releases - I was
> > thinking 4 months)
> >
> > 2. Predictability for contributors and users:
> > * Developers and reviewers can decide in advance what release they are
> > aiming for with specific features.
> > * If a feature misses a release we have a good idea of when it will show
> > up.
> > * Users know when to expect their features
> >
> > 3. Transparency - There will be a published cut-off date (AKA feature
> > freeze) for the release and people will know about it in advance.
> > Hopefully this will remove the contention around which features ma

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

Hi Team Kafka,

As per the KIP meeting, I created a wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
Summarizing most of the discussion so far.

Comments and additional discussion is welcome :)

Gwen

On Wed, Aug 17, 2016 at 12:31 PM, Vahid S Hashemian
<vahidhashem...@us.ibm.com> wrote:
> Time-based releases is a good idea and something that has proved to be
> working in a number of open source projects. One successful example is
> Node.js, that goes through two major releases a year. The interesting fact
> about the two releases is that only one (the even-number release) comes
> with a long term support (LTS) plan (30 months). More can be read here:
> https://github.com/nodejs/LTS. The odd-number releases still come with
> major changes and help build the ecosystem, but as far as LTS goes, there
> is only one per year. This LTS plan makes most enterprises want to stick
> to even-number releases, which is okay since frequent upgrades is not
> something they are normally interested in anyway.
>
> There could be several minor releases (non-breaking) in between major
> releases. A major release contains all the features / bug fixes in the
> master branch a month before the release date, with the potential addition
> of (non-breaking) bug fixes until the release day. The deprecation cycle
> is one major release: any functionality that is decided to be removed is
> deprecated in the next major release, and removed in the major release
> after that.
>
> Because of the success of LTS models in this and other open source
> projects, I would suggest implementing a formal LTS plan for Kafka too.
>
> Regards,
> --Vahid
>
>
>
> From:   Gwen Shapira <g...@confluent.io>
> To: dev@kafka.apache.org
> Date:   08/09/2016 04:49 PM
> Subject:[DISCUSS] Time-based releases for Apache Kafka
>
>
>
> Dear Kafka Developers and Users,
>
> In the past, our releases have been quite unpredictable. We'll notice
> that a large number of nice features made it in (or are close),
> someone would suggest a release and we'd do it. This is fun, but makes
> planning really hard - we saw it during the last release which we
> decided to delay by a few weeks to allow more features to "land".
>
> Many other communities have adopted time-based releases successfully
> (Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
> will make sense for the Apache Kafka community to try doing the same.
>
> The benefits of this approach are:
>
> 1. A quicker feedback cycle and users can benefit from features
> quicker (assuming for reasonably short time between releases - I was
> thinking 4 months)
>
> 2. Predictability for contributors and users:
> * Developers and reviewers can decide in advance what release they are
> aiming for with specific features.
> * If a feature misses a release we have a good idea of when it will show
> up.
> * Users know when to expect their features
>
> 3. Transparency - There will be a published cut-off date (AKA feature
> freeze) for the release and people will know about it in advance.
> Hopefully this will remove the contention around which features make
> it.
>
> 4. Quality - we've seen issues pop up in release candidates due to
> last-minute features that didn't have proper time to bake in. More
> time between feature freeze and release will let us test more,
> document more and resolve more issues.
>
> Since nothing is ever perfect, there will be some downsides:
>
> 1. Most notably, features that miss the feature-freeze date for a
> release will have to wait few month for the next release. Features
> will reach users faster overall as per benefit #1, but individual
> features that just miss the cut will lose out
>
> 2. More releases a year mean that being a committer is more work -
> release management is still some headache and we'll have more of
> those. Hopefully we'll get better at it. Also, the committer list is
> growing and hopefully it will be less than once-a-year effort for each
> committer.
>
> 3. For users, figuring out which release to use and having frequent
> new releases to upgrade to may be a bit confusing.
>
> 4. Frequent releases mean we need to do bugfix releases for older
> branches. Right now we only do bugfix releases to latest release.
>
> I think the benefits outweigh the drawbacks. Or at least suggest that
> its worth trying - we can have another discussion in few releases to
> see if we want to keep it that way or try something else.
>
> My suggestion for the process:
>
> 1. We decide on a reasonable release cadence
> 2. We decide on release dates (even rough estimate such as "end of
> February" or something) and 

Re: [DISCUSS] Time-based releases for Apache Kafka

[Vahid S Hashemian]

Time-based releases is a good idea and something that has proved to be 
working in a number of open source projects. One successful example is 
Node.js, that goes through two major releases a year. The interesting fact 
about the two releases is that only one (the even-number release) comes 
with a long term support (LTS) plan (30 months). More can be read here: 
https://github.com/nodejs/LTS. The odd-number releases still come with 
major changes and help build the ecosystem, but as far as LTS goes, there 
is only one per year. This LTS plan makes most enterprises want to stick 
to even-number releases, which is okay since frequent upgrades is not 
something they are normally interested in anyway.

There could be several minor releases (non-breaking) in between major 
releases. A major release contains all the features / bug fixes in the 
master branch a month before the release date, with the potential addition 
of (non-breaking) bug fixes until the release day. The deprecation cycle 
is one major release: any functionality that is decided to be removed is 
deprecated in the next major release, and removed in the major release 
after that.

Because of the success of LTS models in this and other open source 
projects, I would suggest implementing a formal LTS plan for Kafka too.

Regards,
--Vahid



From:   Gwen Shapira <g...@confluent.io>
To: dev@kafka.apache.org
Date:   08/09/2016 04:49 PM
Subject:    [DISCUSS] Time-based releases for Apache Kafka



Dear Kafka Developers and Users,

In the past, our releases have been quite unpredictable. We'll notice
that a large number of nice features made it in (or are close),
someone would suggest a release and we'd do it. This is fun, but makes
planning really hard - we saw it during the last release which we
decided to delay by a few weeks to allow more features to "land".

Many other communities have adopted time-based releases successfully
(Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
will make sense for the Apache Kafka community to try doing the same.

The benefits of this approach are:

1. A quicker feedback cycle and users can benefit from features
quicker (assuming for reasonably short time between releases - I was
thinking 4 months)

2. Predictability for contributors and users:
* Developers and reviewers can decide in advance what release they are
aiming for with specific features.
* If a feature misses a release we have a good idea of when it will show 
up.
* Users know when to expect their features

3. Transparency - There will be a published cut-off date (AKA feature
freeze) for the release and people will know about it in advance.
Hopefully this will remove the contention around which features make
it.

4. Quality - we've seen issues pop up in release candidates due to
last-minute features that didn't have proper time to bake in. More
time between feature freeze and release will let us test more,
document more and resolve more issues.

Since nothing is ever perfect, there will be some downsides:

1. Most notably, features that miss the feature-freeze date for a
release will have to wait few month for the next release. Features
will reach users faster overall as per benefit #1, but individual
features that just miss the cut will lose out

2. More releases a year mean that being a committer is more work -
release management is still some headache and we'll have more of
those. Hopefully we'll get better at it. Also, the committer list is
growing and hopefully it will be less than once-a-year effort for each
committer.

3. For users, figuring out which release to use and having frequent
new releases to upgrade to may be a bit confusing.

4. Frequent releases mean we need to do bugfix releases for older
branches. Right now we only do bugfix releases to latest release.

I think the benefits outweigh the drawbacks. Or at least suggest that
its worth trying - we can have another discussion in few releases to
see if we want to keep it that way or try something else.

My suggestion for the process:

1. We decide on a reasonable release cadence
2. We decide on release dates (even rough estimate such as "end of
February" or something) and work back feature freeze dates.
3. Committers volunteer to be "release managers" for specific
releases. We can coordinate on the list or on a wiki. If no committer
volunteers, we assume the community doesn't need a release and skip
it.
4. At the "feature freeze" date, the release manager announces the
contents of the release (which KIPs made it in on time), creates the
release branch and starts the release process as usual. From this
point onwards, only bug fixes should be double-committed to the
release branch while trunk can start collecting features for the
subsequent release.

Comments and improvements are appreciated.

Gwen Shapira
Former-release-manager

Re: [DISCUSS] Time-based releases for Apache Kafka

[Mickael Maison]

+1

Having better predictability when features will land is a huge benefit.

On Tue, Aug 16, 2016 at 5:34 PM, Jim Jagielski  wrote:
> I'm following along on the thread so for sure! :)
>
>> On Aug 16, 2016, at 12:19 PM, Gwen Shapira  wrote:
>>
>> Absolutely!
>>
>> If you have any concrete suggestions for steps we can take to improve
>> the process, this will be most awesome. We'd love to learn from your
>> long experience in Apache :)
>>
>> Gwen
>>
>> On Tue, Aug 16, 2016 at 6:59 AM, Jim Jagielski  wrote:
>>> By being aware of the potential issues, it's easier to address
>>> them at the start, and to create a process which does what
>>> it can to "ensure" the problems don't pop up :)
>>>
 On Aug 16, 2016, at 9:48 AM, Ismael Juma  wrote:

 Hi Jim,

 Thanks for your feedback. We value the community and we definitely want
 Kafka to remain a fun and friendly place to participate. Under this
 proposal, volunteers will still be able to do the work when they can. The
 benefit is that it is likely to reach users faster since the next release
 is never far away.

 Ismael

 On Tue, Aug 16, 2016 at 2:42 PM, Jim Jagielski  wrote:

> The idea of time-based releases make sense. The issue is
> when they become the tail wagging the dog.
>
> Recall that all developers and contributors are assumed to
> be doing this because they are personally invested in the
> project. Their is also the assumption that, as such, they
> are volunteers and do the work "when they can". And finally,
> there is the fact that working on Apache projects should be
> FUN. It should be someplace where you aren't beholden to,
> or under, some artificial schedule.
>
> If time-based releases are put in place, and held to under
> unforgiving standards, all the above are put at risk. And
> when that happens it puts the project and the community at
> risk as well.
>
> So having a set schedule is fine... it's how "we" do it that
> is key.
>
>>>
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>

Re: [DISCUSS] Time-based releases for Apache Kafka

[Jim Jagielski]

I'm following along on the thread so for sure! :)

> On Aug 16, 2016, at 12:19 PM, Gwen Shapira  wrote:
> 
> Absolutely!
> 
> If you have any concrete suggestions for steps we can take to improve
> the process, this will be most awesome. We'd love to learn from your
> long experience in Apache :)
> 
> Gwen
> 
> On Tue, Aug 16, 2016 at 6:59 AM, Jim Jagielski  wrote:
>> By being aware of the potential issues, it's easier to address
>> them at the start, and to create a process which does what
>> it can to "ensure" the problems don't pop up :)
>> 
>>> On Aug 16, 2016, at 9:48 AM, Ismael Juma  wrote:
>>> 
>>> Hi Jim,
>>> 
>>> Thanks for your feedback. We value the community and we definitely want
>>> Kafka to remain a fun and friendly place to participate. Under this
>>> proposal, volunteers will still be able to do the work when they can. The
>>> benefit is that it is likely to reach users faster since the next release
>>> is never far away.
>>> 
>>> Ismael
>>> 
>>> On Tue, Aug 16, 2016 at 2:42 PM, Jim Jagielski  wrote:
>>> 
 The idea of time-based releases make sense. The issue is
 when they become the tail wagging the dog.
 
 Recall that all developers and contributors are assumed to
 be doing this because they are personally invested in the
 project. Their is also the assumption that, as such, they
 are volunteers and do the work "when they can". And finally,
 there is the fact that working on Apache projects should be
 FUN. It should be someplace where you aren't beholden to,
 or under, some artificial schedule.
 
 If time-based releases are put in place, and held to under
 unforgiving standards, all the above are put at risk. And
 when that happens it puts the project and the community at
 risk as well.
 
 So having a set schedule is fine... it's how "we" do it that
 is key.
 
>> 
> 
> 
> 
> -- 
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

Absolutely!

If you have any concrete suggestions for steps we can take to improve
the process, this will be most awesome. We'd love to learn from your
long experience in Apache :)

Gwen

On Tue, Aug 16, 2016 at 6:59 AM, Jim Jagielski  wrote:
> By being aware of the potential issues, it's easier to address
> them at the start, and to create a process which does what
> it can to "ensure" the problems don't pop up :)
>
>> On Aug 16, 2016, at 9:48 AM, Ismael Juma  wrote:
>>
>> Hi Jim,
>>
>> Thanks for your feedback. We value the community and we definitely want
>> Kafka to remain a fun and friendly place to participate. Under this
>> proposal, volunteers will still be able to do the work when they can. The
>> benefit is that it is likely to reach users faster since the next release
>> is never far away.
>>
>> Ismael
>>
>> On Tue, Aug 16, 2016 at 2:42 PM, Jim Jagielski  wrote:
>>
>>> The idea of time-based releases make sense. The issue is
>>> when they become the tail wagging the dog.
>>>
>>> Recall that all developers and contributors are assumed to
>>> be doing this because they are personally invested in the
>>> project. Their is also the assumption that, as such, they
>>> are volunteers and do the work "when they can". And finally,
>>> there is the fact that working on Apache projects should be
>>> FUN. It should be someplace where you aren't beholden to,
>>> or under, some artificial schedule.
>>>
>>> If time-based releases are put in place, and held to under
>>> unforgiving standards, all the above are put at risk. And
>>> when that happens it puts the project and the community at
>>> risk as well.
>>>
>>> So having a set schedule is fine... it's how "we" do it that
>>> is key.
>>>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Time-based releases for Apache Kafka

[Jim Jagielski]

By being aware of the potential issues, it's easier to address
them at the start, and to create a process which does what
it can to "ensure" the problems don't pop up :)

> On Aug 16, 2016, at 9:48 AM, Ismael Juma  wrote:
> 
> Hi Jim,
> 
> Thanks for your feedback. We value the community and we definitely want
> Kafka to remain a fun and friendly place to participate. Under this
> proposal, volunteers will still be able to do the work when they can. The
> benefit is that it is likely to reach users faster since the next release
> is never far away.
> 
> Ismael
> 
> On Tue, Aug 16, 2016 at 2:42 PM, Jim Jagielski  wrote:
> 
>> The idea of time-based releases make sense. The issue is
>> when they become the tail wagging the dog.
>> 
>> Recall that all developers and contributors are assumed to
>> be doing this because they are personally invested in the
>> project. Their is also the assumption that, as such, they
>> are volunteers and do the work "when they can". And finally,
>> there is the fact that working on Apache projects should be
>> FUN. It should be someplace where you aren't beholden to,
>> or under, some artificial schedule.
>> 
>> If time-based releases are put in place, and held to under
>> unforgiving standards, all the above are put at risk. And
>> when that happens it puts the project and the community at
>> risk as well.
>> 
>> So having a set schedule is fine... it's how "we" do it that
>> is key.
>> 

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

Hi Jim,

Thanks for your feedback. We value the community and we definitely want
Kafka to remain a fun and friendly place to participate. Under this
proposal, volunteers will still be able to do the work when they can. The
benefit is that it is likely to reach users faster since the next release
is never far away.

Ismael

On Tue, Aug 16, 2016 at 2:42 PM, Jim Jagielski  wrote:

> The idea of time-based releases make sense. The issue is
> when they become the tail wagging the dog.
>
> Recall that all developers and contributors are assumed to
> be doing this because they are personally invested in the
> project. Their is also the assumption that, as such, they
> are volunteers and do the work "when they can". And finally,
> there is the fact that working on Apache projects should be
> FUN. It should be someplace where you aren't beholden to,
> or under, some artificial schedule.
>
> If time-based releases are put in place, and held to under
> unforgiving standards, all the above are put at risk. And
> when that happens it puts the project and the community at
> risk as well.
>
> So having a set schedule is fine... it's how "we" do it that
> is key.
>

Re: [DISCUSS] Time-based releases for Apache Kafka

[Jim Jagielski]

The idea of time-based releases make sense. The issue is
when they become the tail wagging the dog.

Recall that all developers and contributors are assumed to
be doing this because they are personally invested in the
project. Their is also the assumption that, as such, they
are volunteers and do the work "when they can". And finally,
there is the fact that working on Apache projects should be
FUN. It should be someplace where you aren't beholden to,
or under, some artificial schedule.

If time-based releases are put in place, and held to under
unforgiving standards, all the above are put at risk. And
when that happens it puts the project and the community at
risk as well.

So having a set schedule is fine... it's how "we" do it that
is key.

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

Exactly :)

The goal is to have stable releases. We are hoping that time-based
will help achieve this goal as well as introduce some stability to the
planning process.

On Mon, Aug 15, 2016 at 12:55 PM, Nacho Solis
 wrote:
> To clear up, I'm not against time-based releases, I just think that the
> goals that were stated are not intrinsic to time-based releases but the
> release process (whether it's time-based or not).
>
> The goal of "when will my code get into a release" and the goal of getting
> features faster in a release (vs just in trunk) seem (imho) secondary to
> providing stable releases.  If the releases happen every 4 months then
> we're saying every 4 months we're providing a stable release, right?
>
> Nacho
>
> On Sat, Aug 13, 2016 at 12:17 PM, Neha Narkhede  wrote:
>
>> I'm supportive of this for 2 reasons -
>>
>> 1. The community has been looking for predictability and this allows us to
>> offer that to Kafka users
>> 2. Trunk stability and the ability to release from trunk. This is important
>> for several companies and more frequent releases means higher quality and
>> faster detection of regressions.
>>
>> This does mean we are signing up for more work in the following areas --
>>
>> 1. Release management by committers.
>> 2. Discipline amongst contributors to pull put features that are not ready
>> by the code freeze.
>>
>> We'd also have to learn what cadence works for Kafka. We can start with the
>> one that Gwen suggested and see what works.
>>
>> I'd give it a try.
>> On Sat, Aug 13, 2016 at 11:25 AM Kartik Paramasivam
>>  wrote:
>>
>> > Plus one.  This is a good direction to move towards.
>> >
>> > The frequency of releases would probably depend on how long it takes
>> > to certify the release.
>> >
>> > > On Aug 13, 2016, at 10:18 AM, Jay Kreps  wrote:
>> > >
>> > > I'm +1.
>> > >
>> > > I've seen this both ways and I really do think that time-based releases
>> > > tend to scale better with more developers doing parallel work (I think
>> > the
>> > > probability of at least one feature slipping as you have more and more
>> > > developers gets very high, and if that means the release slips then the
>> > > release will frequently slip quite a lot). I think between the clients,
>> > the
>> > > server, streams, connect, etc there is enough parallelism that this
>> will
>> > be
>> > > important.
>> > >
>> > > I think this also gives a lot more predictability to people who
>> > contribute
>> > > code or want to use a feature they see on trunk as to when it will be
>> > > available in a release (to date our answer has been "eventually", which
>> > is
>> > > a bit unsatisfying).
>> > >
>> > > -Jay
>> > >
>> > >> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira 
>> wrote:
>> > >>
>> > >> Dear Kafka Developers and Users,
>> > >>
>> > >> In the past, our releases have been quite unpredictable. We'll notice
>> > >> that a large number of nice features made it in (or are close),
>> > >> someone would suggest a release and we'd do it. This is fun, but makes
>> > >> planning really hard - we saw it during the last release which we
>> > >> decided to delay by a few weeks to allow more features to "land".
>> > >>
>> > >> Many other communities have adopted time-based releases successfully
>> > >> (Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
>> > >> will make sense for the Apache Kafka community to try doing the same.
>> > >>
>> > >> The benefits of this approach are:
>> > >>
>> > >> 1. A quicker feedback cycle and users can benefit from features
>> > >> quicker (assuming for reasonably short time between releases - I was
>> > >> thinking 4 months)
>> > >>
>> > >> 2. Predictability for contributors and users:
>> > >> * Developers and reviewers can decide in advance what release they are
>> > >> aiming for with specific features.
>> > >> * If a feature misses a release we have a good idea of when it will
>> show
>> > >> up.
>> > >> * Users know when to expect their features
>> > >>
>> > >> 3. Transparency - There will be a published cut-off date (AKA feature
>> > >> freeze) for the release and people will know about it in advance.
>> > >> Hopefully this will remove the contention around which features make
>> > >> it.
>> > >>
>> > >> 4. Quality - we've seen issues pop up in release candidates due to
>> > >> last-minute features that didn't have proper time to bake in. More
>> > >> time between feature freeze and release will let us test more,
>> > >> document more and resolve more issues.
>> > >>
>> > >> Since nothing is ever perfect, there will be some downsides:
>> > >>
>> > >> 1. Most notably, features that miss the feature-freeze date for a
>> > >> release will have to wait few month for the next release. Features
>> > >> will reach users faster overall as per benefit #1, but individual
>> > >> features that just miss the cut will lose out
>> > >>
>> > >> 

Re: [DISCUSS] Time-based releases for Apache Kafka

[Nacho Solis]

To clear up, I'm not against time-based releases, I just think that the
goals that were stated are not intrinsic to time-based releases but the
release process (whether it's time-based or not).

The goal of "when will my code get into a release" and the goal of getting
features faster in a release (vs just in trunk) seem (imho) secondary to
providing stable releases.  If the releases happen every 4 months then
we're saying every 4 months we're providing a stable release, right?

Nacho

On Sat, Aug 13, 2016 at 12:17 PM, Neha Narkhede  wrote:

> I'm supportive of this for 2 reasons -
>
> 1. The community has been looking for predictability and this allows us to
> offer that to Kafka users
> 2. Trunk stability and the ability to release from trunk. This is important
> for several companies and more frequent releases means higher quality and
> faster detection of regressions.
>
> This does mean we are signing up for more work in the following areas --
>
> 1. Release management by committers.
> 2. Discipline amongst contributors to pull put features that are not ready
> by the code freeze.
>
> We'd also have to learn what cadence works for Kafka. We can start with the
> one that Gwen suggested and see what works.
>
> I'd give it a try.
> On Sat, Aug 13, 2016 at 11:25 AM Kartik Paramasivam
>  wrote:
>
> > Plus one.  This is a good direction to move towards.
> >
> > The frequency of releases would probably depend on how long it takes
> > to certify the release.
> >
> > > On Aug 13, 2016, at 10:18 AM, Jay Kreps  wrote:
> > >
> > > I'm +1.
> > >
> > > I've seen this both ways and I really do think that time-based releases
> > > tend to scale better with more developers doing parallel work (I think
> > the
> > > probability of at least one feature slipping as you have more and more
> > > developers gets very high, and if that means the release slips then the
> > > release will frequently slip quite a lot). I think between the clients,
> > the
> > > server, streams, connect, etc there is enough parallelism that this
> will
> > be
> > > important.
> > >
> > > I think this also gives a lot more predictability to people who
> > contribute
> > > code or want to use a feature they see on trunk as to when it will be
> > > available in a release (to date our answer has been "eventually", which
> > is
> > > a bit unsatisfying).
> > >
> > > -Jay
> > >
> > >> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira 
> wrote:
> > >>
> > >> Dear Kafka Developers and Users,
> > >>
> > >> In the past, our releases have been quite unpredictable. We'll notice
> > >> that a large number of nice features made it in (or are close),
> > >> someone would suggest a release and we'd do it. This is fun, but makes
> > >> planning really hard - we saw it during the last release which we
> > >> decided to delay by a few weeks to allow more features to "land".
> > >>
> > >> Many other communities have adopted time-based releases successfully
> > >> (Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
> > >> will make sense for the Apache Kafka community to try doing the same.
> > >>
> > >> The benefits of this approach are:
> > >>
> > >> 1. A quicker feedback cycle and users can benefit from features
> > >> quicker (assuming for reasonably short time between releases - I was
> > >> thinking 4 months)
> > >>
> > >> 2. Predictability for contributors and users:
> > >> * Developers and reviewers can decide in advance what release they are
> > >> aiming for with specific features.
> > >> * If a feature misses a release we have a good idea of when it will
> show
> > >> up.
> > >> * Users know when to expect their features
> > >>
> > >> 3. Transparency - There will be a published cut-off date (AKA feature
> > >> freeze) for the release and people will know about it in advance.
> > >> Hopefully this will remove the contention around which features make
> > >> it.
> > >>
> > >> 4. Quality - we've seen issues pop up in release candidates due to
> > >> last-minute features that didn't have proper time to bake in. More
> > >> time between feature freeze and release will let us test more,
> > >> document more and resolve more issues.
> > >>
> > >> Since nothing is ever perfect, there will be some downsides:
> > >>
> > >> 1. Most notably, features that miss the feature-freeze date for a
> > >> release will have to wait few month for the next release. Features
> > >> will reach users faster overall as per benefit #1, but individual
> > >> features that just miss the cut will lose out
> > >>
> > >> 2. More releases a year mean that being a committer is more work -
> > >> release management is still some headache and we'll have more of
> > >> those. Hopefully we'll get better at it. Also, the committer list is
> > >> growing and hopefully it will be less than once-a-year effort for each
> > >> committer.
> > >>
> > >> 3. For users, figuring out which release to 

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

To add to what Gwen said (which makes sense to me), here's a link to the
Cassandra release model:

http://www.planetcassandra.org/blog/cassandra-2-2-3-0-and-beyond/

It involves monthly releases, so more aggressive than what is being
proposed, but they follow a tick tock model where a feature release is
followed by a bug fix release (so a 2 months cycle for combined features +
bug fixes, which is half the time than what is being proposed for Kafka).

I also want to mention that the proposed model is likely to result in more
effective testing than the previous model since we have a freeze period of
similar duration, but a smaller period to accumulate changes.

Ismael

On Sat, Aug 13, 2016 at 3:02 AM, Gwen Shapira  wrote:

> Thats good feedback, thanks.
>
> I don't think you were involved in the previous release, but we did
> try to follow the model you suggest.
> We announced few weeks before cutting branches and gave time for
> features and stabilization. This started a mad last-minute rush of
> semi-baked features that forced 6 (or 7) release candidates and
> significant delays.
>
> I think (but can't know) that part of the reason is that people felt
> compelled to get features in to the nearest release, because who knows
> when the next one will happen. Scheduled releases can take some of the
> stress and pressure off the process (we hope) - you miss this train
> and the next will show up.
>
> This is obviously just a reasoned guess, we won't know without trying.
> But since we did try feature releases and ran into difficulties, I
> thought we can experiment with something different.
>
> After 3 releases (i.e. this time next year), we can circle back and
> see how everyone feels and whether we need to adjust course.
>
> I am fairly confident that with 4 month per release and one month for
> stability we'll have both features and sufficient testing (already the
> next release will have the admin protocols, which is huge, time-based
> indexes, connector improvements, throttling... it will be a good
> one!). But we won't know for sure until we try
>
> I do agree with you that Ubuntu is not the right model for us. I am
> more inspired by Cassandra that has similar product with a rather
> similar community, and are doing time-based releases successfully.
>
> Gwen
>
> On Fri, Aug 12, 2016 at 2:35 PM, Nacho Solis
>  wrote:
> > I'm not convinced that time-based releases for this type of development
> are
> > the right thing.
> >
> > Something like Ubuntu, where all components are moving targets needs to
> > decide to freeze and release without having full coordination from the
> > participants.  There is no guarantee from Canonical that the intermediate
> > product works well together.   Debian takes a different (and more stable)
> > approach and releases on features.  They also have to corral a bunch of
> > separate components and make them work together.
> >
> > I would argue that the benefit of a release is that it's tested and
> > coherent. From a kafka standpoint I would rather see releases on
> > feature[set] with some testing and then have some notion of bug support
> for
> > a little while.  If I want to have cutting edge or more frequent releases
> > then I should be able to grab stuff off of github and run that.
> >
> >
> > I don't agree that the benefits we're looking for are what we're going to
> > get.
> >
> > We can achieve some of the benefits listed without having to have timed
> > releases.  What it seems you're looking for is more of a "heads up"
> > announcement that we plan to do a release.  We could have a release
> process
> > that gives 4 weeks of warning.  2 weeks to get features in, 2 weeks of
> > freeze and then a release.  There is no need to fix how often (and when)
> > this process happens.  We just know that for a release to happen we'll
> have
> > 4 weeks advance notice.
> >
> > Nacho
> >
> >
> >
> > On Fri, Aug 12, 2016 at 1:17 PM, Gwen Shapira  wrote:
> >
> >> Good question!
> >>
> >> My thoughts are... bugfix releases will be done "as needed" based on
> >> community feedback
> >>
> >> Feature releases will be a minor release by default 0.11, 0.12 - unless:
> >> 1. We manage to break compatibility completely (we shouldn't) in which
> >> case we need to bump to 1.X
> >> 2. We do something totally amazing (exactly once?) and decide to bump
> >> to 1.X to celebrate
> >> 3. The release manager decides that the features in the release are
> >> not very exciting and we can go with 0.10.1 (i.e very minor release)
> >>
> >> Does that make sense?
> >>
> >> On Fri, Aug 12, 2016 at 10:25 AM, Nacho Solis
> >>  wrote:
> >> > How would time releases relate to versions? (Major, minor, API
> >> > compatibility, etc).
> >> >
> >> > On Thu, Aug 11, 2016 at 9:37 AM, Guozhang Wang 
> >> wrote:
> >> >
> >> >> I think we do not need to make the same guarantee as for "how old of
> >> 

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

Thats good feedback, thanks.

I don't think you were involved in the previous release, but we did
try to follow the model you suggest.
We announced few weeks before cutting branches and gave time for
features and stabilization. This started a mad last-minute rush of
semi-baked features that forced 6 (or 7) release candidates and
significant delays.

I think (but can't know) that part of the reason is that people felt
compelled to get features in to the nearest release, because who knows
when the next one will happen. Scheduled releases can take some of the
stress and pressure off the process (we hope) - you miss this train
and the next will show up.

This is obviously just a reasoned guess, we won't know without trying.
But since we did try feature releases and ran into difficulties, I
thought we can experiment with something different.

After 3 releases (i.e. this time next year), we can circle back and
see how everyone feels and whether we need to adjust course.

I am fairly confident that with 4 month per release and one month for
stability we'll have both features and sufficient testing (already the
next release will have the admin protocols, which is huge, time-based
indexes, connector improvements, throttling... it will be a good
one!). But we won't know for sure until we try

I do agree with you that Ubuntu is not the right model for us. I am
more inspired by Cassandra that has similar product with a rather
similar community, and are doing time-based releases successfully.

Gwen

On Fri, Aug 12, 2016 at 2:35 PM, Nacho Solis
 wrote:
> I'm not convinced that time-based releases for this type of development are
> the right thing.
>
> Something like Ubuntu, where all components are moving targets needs to
> decide to freeze and release without having full coordination from the
> participants.  There is no guarantee from Canonical that the intermediate
> product works well together.   Debian takes a different (and more stable)
> approach and releases on features.  They also have to corral a bunch of
> separate components and make them work together.
>
> I would argue that the benefit of a release is that it's tested and
> coherent. From a kafka standpoint I would rather see releases on
> feature[set] with some testing and then have some notion of bug support for
> a little while.  If I want to have cutting edge or more frequent releases
> then I should be able to grab stuff off of github and run that.
>
>
> I don't agree that the benefits we're looking for are what we're going to
> get.
>
> We can achieve some of the benefits listed without having to have timed
> releases.  What it seems you're looking for is more of a "heads up"
> announcement that we plan to do a release.  We could have a release process
> that gives 4 weeks of warning.  2 weeks to get features in, 2 weeks of
> freeze and then a release.  There is no need to fix how often (and when)
> this process happens.  We just know that for a release to happen we'll have
> 4 weeks advance notice.
>
> Nacho
>
>
>
> On Fri, Aug 12, 2016 at 1:17 PM, Gwen Shapira  wrote:
>
>> Good question!
>>
>> My thoughts are... bugfix releases will be done "as needed" based on
>> community feedback
>>
>> Feature releases will be a minor release by default 0.11, 0.12 - unless:
>> 1. We manage to break compatibility completely (we shouldn't) in which
>> case we need to bump to 1.X
>> 2. We do something totally amazing (exactly once?) and decide to bump
>> to 1.X to celebrate
>> 3. The release manager decides that the features in the release are
>> not very exciting and we can go with 0.10.1 (i.e very minor release)
>>
>> Does that make sense?
>>
>> On Fri, Aug 12, 2016 at 10:25 AM, Nacho Solis
>>  wrote:
>> > How would time releases relate to versions? (Major, minor, API
>> > compatibility, etc).
>> >
>> > On Thu, Aug 11, 2016 at 9:37 AM, Guozhang Wang 
>> wrote:
>> >
>> >> I think we do not need to make the same guarantee as for "how old of
>> your
>> >> Kafka version that you can upgrade to the latest in one shot" (just
>> call it
>> >> "upgrade maintenance" for short) and "how old of your Kafka version that
>> >> you can enjoy backport critical bug fixes from the latest version"
>> (call it
>> >> "bugfix backport maintenance" for short).
>> >>
>> >> Upgrade maintenance: I think 2 years is a good cadence, and we can use
>> the
>> >> same policy for getting rid of obsolete APIs / protocols as well. I.e.
>> say
>> >> we release 0.10.1.0 in 2017FEB then we can in that release drop
>> obsoleted
>> >> code in 0.8.2 (released in 2015FEB), such that users cannot directly
>> >> upgrade from 0.8.2.x to 0.10.1.x, but needs to have another hop.
>> >>
>> >> Bugfix backports maintenance: if we release every 4 months, then
>> current +
>> >> 2 older means we have one year period for back-port critical bug fixes,
>> >> which sounds good to me; if we release every 3 months, then 

Re: [DISCUSS] Time-based releases for Apache Kafka

[Nacho Solis]

I'm not convinced that time-based releases for this type of development are
the right thing.

Something like Ubuntu, where all components are moving targets needs to
decide to freeze and release without having full coordination from the
participants.  There is no guarantee from Canonical that the intermediate
product works well together.   Debian takes a different (and more stable)
approach and releases on features.  They also have to corral a bunch of
separate components and make them work together.

I would argue that the benefit of a release is that it's tested and
coherent. From a kafka standpoint I would rather see releases on
feature[set] with some testing and then have some notion of bug support for
a little while.  If I want to have cutting edge or more frequent releases
then I should be able to grab stuff off of github and run that.


I don't agree that the benefits we're looking for are what we're going to
get.

We can achieve some of the benefits listed without having to have timed
releases.  What it seems you're looking for is more of a "heads up"
announcement that we plan to do a release.  We could have a release process
that gives 4 weeks of warning.  2 weeks to get features in, 2 weeks of
freeze and then a release.  There is no need to fix how often (and when)
this process happens.  We just know that for a release to happen we'll have
4 weeks advance notice.

Nacho



On Fri, Aug 12, 2016 at 1:17 PM, Gwen Shapira  wrote:

> Good question!
>
> My thoughts are... bugfix releases will be done "as needed" based on
> community feedback
>
> Feature releases will be a minor release by default 0.11, 0.12 - unless:
> 1. We manage to break compatibility completely (we shouldn't) in which
> case we need to bump to 1.X
> 2. We do something totally amazing (exactly once?) and decide to bump
> to 1.X to celebrate
> 3. The release manager decides that the features in the release are
> not very exciting and we can go with 0.10.1 (i.e very minor release)
>
> Does that make sense?
>
> On Fri, Aug 12, 2016 at 10:25 AM, Nacho Solis
>  wrote:
> > How would time releases relate to versions? (Major, minor, API
> > compatibility, etc).
> >
> > On Thu, Aug 11, 2016 at 9:37 AM, Guozhang Wang 
> wrote:
> >
> >> I think we do not need to make the same guarantee as for "how old of
> your
> >> Kafka version that you can upgrade to the latest in one shot" (just
> call it
> >> "upgrade maintenance" for short) and "how old of your Kafka version that
> >> you can enjoy backport critical bug fixes from the latest version"
> (call it
> >> "bugfix backport maintenance" for short).
> >>
> >> Upgrade maintenance: I think 2 years is a good cadence, and we can use
> the
> >> same policy for getting rid of obsolete APIs / protocols as well. I.e.
> say
> >> we release 0.10.1.0 in 2017FEB then we can in that release drop
> obsoleted
> >> code in 0.8.2 (released in 2015FEB), such that users cannot directly
> >> upgrade from 0.8.2.x to 0.10.1.x, but needs to have another hop.
> >>
> >> Bugfix backports maintenance: if we release every 4 months, then
> current +
> >> 2 older means we have one year period for back-port critical bug fixes,
> >> which sounds good to me; if we release every 3 months, then probably we
> >> should have current + 3 older.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Thu, Aug 11, 2016 at 12:14 AM, Ismael Juma 
> wrote:
> >>
> >> > Do we need to make a decision on this particular point? Could we gauge
> >> > community demand (people tend to ask for fixes to be backported in
> JIRA)
> >> > and decide then?
> >> >
> >> > If we make a good job of avoiding regressions, then it seems that each
> >> > branch should really only need one or or a maximum of two bug fix
> >> releases.
> >> > After that, users are welcome to upgrade to a new feature release (and
> >> > hopefully to the most current) to get non-critical fixes and
> features. An
> >> > exception is if we get security fixes. We probably do need a policy
> for
> >> how
> >> > far we backport those.
> >> >
> >> > Ismael
> >> >
> >> > On Thu, Aug 11, 2016 at 4:35 AM, Gwen Shapira 
> wrote:
> >> >
> >> > > Yeah, I agree that maintaining 6 release branches is not really
> >> > > sustainable...
> >> > >
> >> > > Maybe 3 (current and 2 older) makes sense?
> >> > >
> >> > > On Wed, Aug 10, 2016 at 7:35 PM, Joel Koshy 
> >> wrote:
> >> > > > On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy 
> >> > wrote:
> >> > > >
> >> > > >>
> >> > > >>
> >> > > >> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira 
> >> > wrote:
> >> > > >>
> >> > > >>>
> >> > > >>> 4. Frequent releases mean we need to do bugfix releases for
> older
> >> > > >>> branches. Right now we only do bugfix releases to latest
> release.
> >> > > >>>
> >> > > >>
> >> > > >> I'm a bit unclear on how the above is a side-effect of time-based
> >> > > >> 

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

Interesting, so you are suggesting that we drop Kafka's current versioning
scheme (0.major.minor.patch)? I can see the reasoning for doing so now.

However, I think I'd prefer to do that when we do the 1.x bump, which I
think we should do once exactly-once lands. That way people only have to
re-learn what Kafka version numbers mean exactly once (and the 1.x bump is
a good signal that something has changed with regards to that). I don't
feel too strongly though, so I would be OK with the other way if others
feel that it's better.

If we were to keep the existing versioning, it would look like:

Feature releases will be a minor release by default 0.10.1, 0.10.2 - unless:
1. We break compatibility (i.e. remove deprecated public methods after a
reasonable period), in which case we bump to 0.11.x
2. We change the message format, in which case we bump to 0.11.x
3. We do something totally amazing (exactly once?) and decide to bump to
1.X to celebrate

It's worth noting that 0.9.0.0 was an exception in that we bumped to the
next version because it had lots of new features even though it was not
incompatible. Also, we consider new protocol versions as compatible in
minor releases even though it's not entirely true (the Java clients cannot
talk to older brokers if there are new protocol versions that they use). We
don't introduce new protocol versions in patch releases.

Ismael

P.S. I wrote this in parallel to Jun's message

On Fri, Aug 12, 2016 at 9:17 PM, Gwen Shapira  wrote:

> Good question!
>
> My thoughts are... bugfix releases will be done "as needed" based on
> community feedback
>
> Feature releases will be a minor release by default 0.11, 0.12 - unless:
> 1. We manage to break compatibility completely (we shouldn't) in which
> case we need to bump to 1.X
> 2. We do something totally amazing (exactly once?) and decide to bump
> to 1.X to celebrate
> 3. The release manager decides that the features in the release are
> not very exciting and we can go with 0.10.1 (i.e very minor release)
>
> Does that make sense?
>
> On Fri, Aug 12, 2016 at 10:25 AM, Nacho Solis
>  wrote:
> > How would time releases relate to versions? (Major, minor, API
> > compatibility, etc).
> >
> > On Thu, Aug 11, 2016 at 9:37 AM, Guozhang Wang 
> wrote:
> >
> >> I think we do not need to make the same guarantee as for "how old of
> your
> >> Kafka version that you can upgrade to the latest in one shot" (just
> call it
> >> "upgrade maintenance" for short) and "how old of your Kafka version that
> >> you can enjoy backport critical bug fixes from the latest version"
> (call it
> >> "bugfix backport maintenance" for short).
> >>
> >> Upgrade maintenance: I think 2 years is a good cadence, and we can use
> the
> >> same policy for getting rid of obsolete APIs / protocols as well. I.e.
> say
> >> we release 0.10.1.0 in 2017FEB then we can in that release drop
> obsoleted
> >> code in 0.8.2 (released in 2015FEB), such that users cannot directly
> >> upgrade from 0.8.2.x to 0.10.1.x, but needs to have another hop.
> >>
> >> Bugfix backports maintenance: if we release every 4 months, then
> current +
> >> 2 older means we have one year period for back-port critical bug fixes,
> >> which sounds good to me; if we release every 3 months, then probably we
> >> should have current + 3 older.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Thu, Aug 11, 2016 at 12:14 AM, Ismael Juma 
> wrote:
> >>
> >> > Do we need to make a decision on this particular point? Could we gauge
> >> > community demand (people tend to ask for fixes to be backported in
> JIRA)
> >> > and decide then?
> >> >
> >> > If we make a good job of avoiding regressions, then it seems that each
> >> > branch should really only need one or or a maximum of two bug fix
> >> releases.
> >> > After that, users are welcome to upgrade to a new feature release (and
> >> > hopefully to the most current) to get non-critical fixes and
> features. An
> >> > exception is if we get security fixes. We probably do need a policy
> for
> >> how
> >> > far we backport those.
> >> >
> >> > Ismael
> >> >
> >> > On Thu, Aug 11, 2016 at 4:35 AM, Gwen Shapira 
> wrote:
> >> >
> >> > > Yeah, I agree that maintaining 6 release branches is not really
> >> > > sustainable...
> >> > >
> >> > > Maybe 3 (current and 2 older) makes sense?
> >> > >
> >> > > On Wed, Aug 10, 2016 at 7:35 PM, Joel Koshy 
> >> wrote:
> >> > > > On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy 
> >> > wrote:
> >> > > >
> >> > > >>
> >> > > >>
> >> > > >> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira 
> >> > wrote:
> >> > > >>
> >> > > >>>
> >> > > >>> 4. Frequent releases mean we need to do bugfix releases for
> older
> >> > > >>> branches. Right now we only do bugfix releases to latest
> release.
> >> > > >>>
> >> > > >>
> >> > > >> I'm a bit unclear on how the above is a 

Re: [DISCUSS] Time-based releases for Apache Kafka

[Jun Rao]

Right now, our major releases are really indicated in the second digit, not
in the leading 0. So 0.10 is a major, 0.10.1 is a minor and 0.10.0.1 is a
bug fix.

Thanks,

Jun

On Fri, Aug 12, 2016 at 1:17 PM, Gwen Shapira  wrote:

> Good question!
>
> My thoughts are... bugfix releases will be done "as needed" based on
> community feedback
>
> Feature releases will be a minor release by default 0.11, 0.12 - unless:
> 1. We manage to break compatibility completely (we shouldn't) in which
> case we need to bump to 1.X
> 2. We do something totally amazing (exactly once?) and decide to bump
> to 1.X to celebrate
> 3. The release manager decides that the features in the release are
> not very exciting and we can go with 0.10.1 (i.e very minor release)
>
> Does that make sense?
>
> On Fri, Aug 12, 2016 at 10:25 AM, Nacho Solis
>  wrote:
> > How would time releases relate to versions? (Major, minor, API
> > compatibility, etc).
> >
> > On Thu, Aug 11, 2016 at 9:37 AM, Guozhang Wang 
> wrote:
> >
> >> I think we do not need to make the same guarantee as for "how old of
> your
> >> Kafka version that you can upgrade to the latest in one shot" (just
> call it
> >> "upgrade maintenance" for short) and "how old of your Kafka version that
> >> you can enjoy backport critical bug fixes from the latest version"
> (call it
> >> "bugfix backport maintenance" for short).
> >>
> >> Upgrade maintenance: I think 2 years is a good cadence, and we can use
> the
> >> same policy for getting rid of obsolete APIs / protocols as well. I.e.
> say
> >> we release 0.10.1.0 in 2017FEB then we can in that release drop
> obsoleted
> >> code in 0.8.2 (released in 2015FEB), such that users cannot directly
> >> upgrade from 0.8.2.x to 0.10.1.x, but needs to have another hop.
> >>
> >> Bugfix backports maintenance: if we release every 4 months, then
> current +
> >> 2 older means we have one year period for back-port critical bug fixes,
> >> which sounds good to me; if we release every 3 months, then probably we
> >> should have current + 3 older.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Thu, Aug 11, 2016 at 12:14 AM, Ismael Juma 
> wrote:
> >>
> >> > Do we need to make a decision on this particular point? Could we gauge
> >> > community demand (people tend to ask for fixes to be backported in
> JIRA)
> >> > and decide then?
> >> >
> >> > If we make a good job of avoiding regressions, then it seems that each
> >> > branch should really only need one or or a maximum of two bug fix
> >> releases.
> >> > After that, users are welcome to upgrade to a new feature release (and
> >> > hopefully to the most current) to get non-critical fixes and
> features. An
> >> > exception is if we get security fixes. We probably do need a policy
> for
> >> how
> >> > far we backport those.
> >> >
> >> > Ismael
> >> >
> >> > On Thu, Aug 11, 2016 at 4:35 AM, Gwen Shapira 
> wrote:
> >> >
> >> > > Yeah, I agree that maintaining 6 release branches is not really
> >> > > sustainable...
> >> > >
> >> > > Maybe 3 (current and 2 older) makes sense?
> >> > >
> >> > > On Wed, Aug 10, 2016 at 7:35 PM, Joel Koshy 
> >> wrote:
> >> > > > On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy 
> >> > wrote:
> >> > > >
> >> > > >>
> >> > > >>
> >> > > >> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira 
> >> > wrote:
> >> > > >>
> >> > > >>>
> >> > > >>> 4. Frequent releases mean we need to do bugfix releases for
> older
> >> > > >>> branches. Right now we only do bugfix releases to latest
> release.
> >> > > >>>
> >> > > >>
> >> > > >> I'm a bit unclear on how the above is a side-effect of time-based
> >> > > >> releases. IIUC this just changes how frequently we release off
> the
> >> > > current
> >> > > >> release branch right? Or put another way, are you also proposing
> any
> >> > > >> fundamental change to our versioning/branching scheme?
> >> > > >>
> >> > > >
> >> > > > Actually nm - so what you're saying is we cut more frequently off
> >> trunk
> >> > > > which means having to maintaining multiple release branches.
> However,
> >> > the
> >> > > > more frequently we release then it should be less difficult to
> >> upgrade
> >> > > from
> >> > > > one release to another which means it should be reasonable to
> expect
> >> > that
> >> > > > we EOL release branches sooner than later.
> >> > > >
> >> > > > However, if we are expected to maintain release branches for up to
> >> two
> >> > > > years then that means potential bugfix releases for up to eight
> >> release
> >> > > > branches at any given time? i.e., it seems that a short
> inter-release
> >> > > > interval would require us to EOL release branches sooner than
> that to
> >> > > make
> >> > > > things manageable.
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Gwen Shapira
> >> > > Product Manager | Confluent
> >> > > 650.450.2760 | @gwenshap
> >> > > Follow us: 

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

Good question!

My thoughts are... bugfix releases will be done "as needed" based on
community feedback

Feature releases will be a minor release by default 0.11, 0.12 - unless:
1. We manage to break compatibility completely (we shouldn't) in which
case we need to bump to 1.X
2. We do something totally amazing (exactly once?) and decide to bump
to 1.X to celebrate
3. The release manager decides that the features in the release are
not very exciting and we can go with 0.10.1 (i.e very minor release)

Does that make sense?

On Fri, Aug 12, 2016 at 10:25 AM, Nacho Solis
 wrote:
> How would time releases relate to versions? (Major, minor, API
> compatibility, etc).
>
> On Thu, Aug 11, 2016 at 9:37 AM, Guozhang Wang  wrote:
>
>> I think we do not need to make the same guarantee as for "how old of your
>> Kafka version that you can upgrade to the latest in one shot" (just call it
>> "upgrade maintenance" for short) and "how old of your Kafka version that
>> you can enjoy backport critical bug fixes from the latest version" (call it
>> "bugfix backport maintenance" for short).
>>
>> Upgrade maintenance: I think 2 years is a good cadence, and we can use the
>> same policy for getting rid of obsolete APIs / protocols as well. I.e. say
>> we release 0.10.1.0 in 2017FEB then we can in that release drop obsoleted
>> code in 0.8.2 (released in 2015FEB), such that users cannot directly
>> upgrade from 0.8.2.x to 0.10.1.x, but needs to have another hop.
>>
>> Bugfix backports maintenance: if we release every 4 months, then current +
>> 2 older means we have one year period for back-port critical bug fixes,
>> which sounds good to me; if we release every 3 months, then probably we
>> should have current + 3 older.
>>
>>
>> Guozhang
>>
>>
>> On Thu, Aug 11, 2016 at 12:14 AM, Ismael Juma  wrote:
>>
>> > Do we need to make a decision on this particular point? Could we gauge
>> > community demand (people tend to ask for fixes to be backported in JIRA)
>> > and decide then?
>> >
>> > If we make a good job of avoiding regressions, then it seems that each
>> > branch should really only need one or or a maximum of two bug fix
>> releases.
>> > After that, users are welcome to upgrade to a new feature release (and
>> > hopefully to the most current) to get non-critical fixes and features. An
>> > exception is if we get security fixes. We probably do need a policy for
>> how
>> > far we backport those.
>> >
>> > Ismael
>> >
>> > On Thu, Aug 11, 2016 at 4:35 AM, Gwen Shapira  wrote:
>> >
>> > > Yeah, I agree that maintaining 6 release branches is not really
>> > > sustainable...
>> > >
>> > > Maybe 3 (current and 2 older) makes sense?
>> > >
>> > > On Wed, Aug 10, 2016 at 7:35 PM, Joel Koshy 
>> wrote:
>> > > > On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy 
>> > wrote:
>> > > >
>> > > >>
>> > > >>
>> > > >> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira 
>> > wrote:
>> > > >>
>> > > >>>
>> > > >>> 4. Frequent releases mean we need to do bugfix releases for older
>> > > >>> branches. Right now we only do bugfix releases to latest release.
>> > > >>>
>> > > >>
>> > > >> I'm a bit unclear on how the above is a side-effect of time-based
>> > > >> releases. IIUC this just changes how frequently we release off the
>> > > current
>> > > >> release branch right? Or put another way, are you also proposing any
>> > > >> fundamental change to our versioning/branching scheme?
>> > > >>
>> > > >
>> > > > Actually nm - so what you're saying is we cut more frequently off
>> trunk
>> > > > which means having to maintaining multiple release branches. However,
>> > the
>> > > > more frequently we release then it should be less difficult to
>> upgrade
>> > > from
>> > > > one release to another which means it should be reasonable to expect
>> > that
>> > > > we EOL release branches sooner than later.
>> > > >
>> > > > However, if we are expected to maintain release branches for up to
>> two
>> > > > years then that means potential bugfix releases for up to eight
>> release
>> > > > branches at any given time? i.e., it seems that a short inter-release
>> > > > interval would require us to EOL release branches sooner than that to
>> > > make
>> > > > things manageable.
>> > >
>> > >
>> > >
>> > > --
>> > > Gwen Shapira
>> > > Product Manager | Confluent
>> > > 650.450.2760 | @gwenshap
>> > > Follow us: Twitter | blog
>> > >
>> >
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
>
> --
> Nacho (Ignacio) Solis
> nso...@linkedin.com



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Time-based releases for Apache Kafka

[Nacho Solis]

​How would time releases relate to versions? (Major, minor, API
compatibility, etc).​

On Thu, Aug 11, 2016 at 9:37 AM, Guozhang Wang  wrote:

> I think we do not need to make the same guarantee as for "how old of your
> Kafka version that you can upgrade to the latest in one shot" (just call it
> "upgrade maintenance" for short) and "how old of your Kafka version that
> you can enjoy backport critical bug fixes from the latest version" (call it
> "bugfix backport maintenance" for short).
>
> Upgrade maintenance: I think 2 years is a good cadence, and we can use the
> same policy for getting rid of obsolete APIs / protocols as well. I.e. say
> we release 0.10.1.0 in 2017FEB then we can in that release drop obsoleted
> code in 0.8.2 (released in 2015FEB), such that users cannot directly
> upgrade from 0.8.2.x to 0.10.1.x, but needs to have another hop.
>
> Bugfix backports maintenance: if we release every 4 months, then current +
> 2 older means we have one year period for back-port critical bug fixes,
> which sounds good to me; if we release every 3 months, then probably we
> should have current + 3 older.
>
>
> Guozhang
>
>
> On Thu, Aug 11, 2016 at 12:14 AM, Ismael Juma  wrote:
>
> > Do we need to make a decision on this particular point? Could we gauge
> > community demand (people tend to ask for fixes to be backported in JIRA)
> > and decide then?
> >
> > If we make a good job of avoiding regressions, then it seems that each
> > branch should really only need one or or a maximum of two bug fix
> releases.
> > After that, users are welcome to upgrade to a new feature release (and
> > hopefully to the most current) to get non-critical fixes and features. An
> > exception is if we get security fixes. We probably do need a policy for
> how
> > far we backport those.
> >
> > Ismael
> >
> > On Thu, Aug 11, 2016 at 4:35 AM, Gwen Shapira  wrote:
> >
> > > Yeah, I agree that maintaining 6 release branches is not really
> > > sustainable...
> > >
> > > Maybe 3 (current and 2 older) makes sense?
> > >
> > > On Wed, Aug 10, 2016 at 7:35 PM, Joel Koshy 
> wrote:
> > > > On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy 
> > wrote:
> > > >
> > > >>
> > > >>
> > > >> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira 
> > wrote:
> > > >>
> > > >>>
> > > >>> 4. Frequent releases mean we need to do bugfix releases for older
> > > >>> branches. Right now we only do bugfix releases to latest release.
> > > >>>
> > > >>
> > > >> I'm a bit unclear on how the above is a side-effect of time-based
> > > >> releases. IIUC this just changes how frequently we release off the
> > > current
> > > >> release branch right? Or put another way, are you also proposing any
> > > >> fundamental change to our versioning/branching scheme?
> > > >>
> > > >
> > > > Actually nm - so what you're saying is we cut more frequently off
> trunk
> > > > which means having to maintaining multiple release branches. However,
> > the
> > > > more frequently we release then it should be less difficult to
> upgrade
> > > from
> > > > one release to another which means it should be reasonable to expect
> > that
> > > > we EOL release branches sooner than later.
> > > >
> > > > However, if we are expected to maintain release branches for up to
> two
> > > > years then that means potential bugfix releases for up to eight
> release
> > > > branches at any given time? i.e., it seems that a short inter-release
> > > > interval would require us to EOL release branches sooner than that to
> > > make
> > > > things manageable.
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
>
>
>
> --
> -- Guozhang
>



-- 
Nacho (Ignacio) Solis
nso...@linkedin.com

Re: [DISCUSS] Time-based releases for Apache Kafka

[Guozhang Wang]

I think we do not need to make the same guarantee as for "how old of your
Kafka version that you can upgrade to the latest in one shot" (just call it
"upgrade maintenance" for short) and "how old of your Kafka version that
you can enjoy backport critical bug fixes from the latest version" (call it
"bugfix backport maintenance" for short).

Upgrade maintenance: I think 2 years is a good cadence, and we can use the
same policy for getting rid of obsolete APIs / protocols as well. I.e. say
we release 0.10.1.0 in 2017FEB then we can in that release drop obsoleted
code in 0.8.2 (released in 2015FEB), such that users cannot directly
upgrade from 0.8.2.x to 0.10.1.x, but needs to have another hop.

Bugfix backports maintenance: if we release every 4 months, then current +
2 older means we have one year period for back-port critical bug fixes,
which sounds good to me; if we release every 3 months, then probably we
should have current + 3 older.


Guozhang


On Thu, Aug 11, 2016 at 12:14 AM, Ismael Juma  wrote:

> Do we need to make a decision on this particular point? Could we gauge
> community demand (people tend to ask for fixes to be backported in JIRA)
> and decide then?
>
> If we make a good job of avoiding regressions, then it seems that each
> branch should really only need one or or a maximum of two bug fix releases.
> After that, users are welcome to upgrade to a new feature release (and
> hopefully to the most current) to get non-critical fixes and features. An
> exception is if we get security fixes. We probably do need a policy for how
> far we backport those.
>
> Ismael
>
> On Thu, Aug 11, 2016 at 4:35 AM, Gwen Shapira  wrote:
>
> > Yeah, I agree that maintaining 6 release branches is not really
> > sustainable...
> >
> > Maybe 3 (current and 2 older) makes sense?
> >
> > On Wed, Aug 10, 2016 at 7:35 PM, Joel Koshy  wrote:
> > > On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy 
> wrote:
> > >
> > >>
> > >>
> > >> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira 
> wrote:
> > >>
> > >>>
> > >>> 4. Frequent releases mean we need to do bugfix releases for older
> > >>> branches. Right now we only do bugfix releases to latest release.
> > >>>
> > >>
> > >> I'm a bit unclear on how the above is a side-effect of time-based
> > >> releases. IIUC this just changes how frequently we release off the
> > current
> > >> release branch right? Or put another way, are you also proposing any
> > >> fundamental change to our versioning/branching scheme?
> > >>
> > >
> > > Actually nm - so what you're saying is we cut more frequently off trunk
> > > which means having to maintaining multiple release branches. However,
> the
> > > more frequently we release then it should be less difficult to upgrade
> > from
> > > one release to another which means it should be reasonable to expect
> that
> > > we EOL release branches sooner than later.
> > >
> > > However, if we are expected to maintain release branches for up to two
> > > years then that means potential bugfix releases for up to eight release
> > > branches at any given time? i.e., it seems that a short inter-release
> > > interval would require us to EOL release branches sooner than that to
> > make
> > > things manageable.
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>



-- 
-- Guozhang

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

Do we need to make a decision on this particular point? Could we gauge
community demand (people tend to ask for fixes to be backported in JIRA)
and decide then?

If we make a good job of avoiding regressions, then it seems that each
branch should really only need one or or a maximum of two bug fix releases.
After that, users are welcome to upgrade to a new feature release (and
hopefully to the most current) to get non-critical fixes and features. An
exception is if we get security fixes. We probably do need a policy for how
far we backport those.

Ismael

On Thu, Aug 11, 2016 at 4:35 AM, Gwen Shapira  wrote:

> Yeah, I agree that maintaining 6 release branches is not really
> sustainable...
>
> Maybe 3 (current and 2 older) makes sense?
>
> On Wed, Aug 10, 2016 at 7:35 PM, Joel Koshy  wrote:
> > On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy  wrote:
> >
> >>
> >>
> >> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira  wrote:
> >>
> >>>
> >>> 4. Frequent releases mean we need to do bugfix releases for older
> >>> branches. Right now we only do bugfix releases to latest release.
> >>>
> >>
> >> I'm a bit unclear on how the above is a side-effect of time-based
> >> releases. IIUC this just changes how frequently we release off the
> current
> >> release branch right? Or put another way, are you also proposing any
> >> fundamental change to our versioning/branching scheme?
> >>
> >
> > Actually nm - so what you're saying is we cut more frequently off trunk
> > which means having to maintaining multiple release branches. However, the
> > more frequently we release then it should be less difficult to upgrade
> from
> > one release to another which means it should be reasonable to expect that
> > we EOL release branches sooner than later.
> >
> > However, if we are expected to maintain release branches for up to two
> > years then that means potential bugfix releases for up to eight release
> > branches at any given time? i.e., it seems that a short inter-release
> > interval would require us to EOL release branches sooner than that to
> make
> > things manageable.
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

Hi Joel,

I think my suggestion was misunderstood. :) I suggested that we should
support upgrades to the latest release for a reasonable period (and I used
2 years as an example). That doesn't mean supporting all of those branches
for that period. It simply means that we maintain the code necessary to
support upgrading in trunk along with relevant system tests. For example,
we currently support direct upgrades from 0.8.x to the unreleased 0.10.1.x
(although we only have tests from 0.8.2.x onwards, 0.8.0.x and 0.8.1.x are
missing). That's 5 releases, so we sort of do it already. I just wanted to
emphasise the importance of making it easier for people to catch up to
current releases (which is a different point from supporting people who
want to stay on older releases).

Ismael

On Thu, Aug 11, 2016 at 3:35 AM, Joel Koshy  wrote:

> On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy  wrote:
>
> >
> >
> > On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira  wrote:
> >
> >>
> >> 4. Frequent releases mean we need to do bugfix releases for older
> >> branches. Right now we only do bugfix releases to latest release.
> >>
> >
> > I'm a bit unclear on how the above is a side-effect of time-based
> > releases. IIUC this just changes how frequently we release off the
> current
> > release branch right? Or put another way, are you also proposing any
> > fundamental change to our versioning/branching scheme?
> >
>
> Actually nm - so what you're saying is we cut more frequently off trunk
> which means having to maintaining multiple release branches. However, the
> more frequently we release then it should be less difficult to upgrade from
> one release to another which means it should be reasonable to expect that
> we EOL release branches sooner than later.
>
> However, if we are expected to maintain release branches for up to two
> years then that means potential bugfix releases for up to eight release
> branches at any given time? i.e., it seems that a short inter-release
> interval would require us to EOL release branches sooner than that to make
> things manageable.
>

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

Yeah, I agree that maintaining 6 release branches is not really sustainable...

Maybe 3 (current and 2 older) makes sense?

On Wed, Aug 10, 2016 at 7:35 PM, Joel Koshy  wrote:
> On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy  wrote:
>
>>
>>
>> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira  wrote:
>>
>>>
>>> 4. Frequent releases mean we need to do bugfix releases for older
>>> branches. Right now we only do bugfix releases to latest release.
>>>
>>
>> I'm a bit unclear on how the above is a side-effect of time-based
>> releases. IIUC this just changes how frequently we release off the current
>> release branch right? Or put another way, are you also proposing any
>> fundamental change to our versioning/branching scheme?
>>
>
> Actually nm - so what you're saying is we cut more frequently off trunk
> which means having to maintaining multiple release branches. However, the
> more frequently we release then it should be less difficult to upgrade from
> one release to another which means it should be reasonable to expect that
> we EOL release branches sooner than later.
>
> However, if we are expected to maintain release branches for up to two
> years then that means potential bugfix releases for up to eight release
> branches at any given time? i.e., it seems that a short inter-release
> interval would require us to EOL release branches sooner than that to make
> things manageable.



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

Good question, let me clarify my thinking:

We were used to doing every year (or even at lower frequency). So the
expectation was that users will just upgrade once a year and we
wouldn't worry about backporting bugfixes to bugs that were over a
year old. It seemed pretty reasonable.

But if we are trying for 3 releases a year... well, almost no one
(except Todd Palino) upgrades 3 times a year. Its like running 50
miles: Doable for some, but definitely not for everyone :)

Basically, the same reasoning behind the desire to support upgrades
for two years: We think it isn't reasonable to ask people to upgrade
every 4 month, so we need to make sure that staying on a year-old
version is still feasible and this includes fixing critical bugs that
are found in the older release (that really isn't all that old) and
publishing bugfix releases every once in a while.

This doesn't really change the way we do versioning or branching
(although our versioning could be slightly broken too, its a different
story). It just means that I propose backporting more patches for
critical bugs and doing some releases off older branches (which we
didn't in the past).

Does that make sense?

Gwen

On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy  wrote:
> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira  wrote:
>
>>
>> 4. Frequent releases mean we need to do bugfix releases for older
>> branches. Right now we only do bugfix releases to latest release.
>>
>
> I'm a bit unclear on how the above is a side-effect of time-based releases.
> IIUC this just changes how frequently we release off the current release
> branch right? Or put another way, are you also proposing any fundamental
> change to our versioning/branching scheme?



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Time-based releases for Apache Kafka

[Joel Koshy]

On Wed, Aug 10, 2016 at 5:44 PM, Joel Koshy  wrote:

>
>
> On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira  wrote:
>
>>
>> 4. Frequent releases mean we need to do bugfix releases for older
>> branches. Right now we only do bugfix releases to latest release.
>>
>
> I'm a bit unclear on how the above is a side-effect of time-based
> releases. IIUC this just changes how frequently we release off the current
> release branch right? Or put another way, are you also proposing any
> fundamental change to our versioning/branching scheme?
>

Actually nm - so what you're saying is we cut more frequently off trunk
which means having to maintaining multiple release branches. However, the
more frequently we release then it should be less difficult to upgrade from
one release to another which means it should be reasonable to expect that
we EOL release branches sooner than later.

However, if we are expected to maintain release branches for up to two
years then that means potential bugfix releases for up to eight release
branches at any given time? i.e., it seems that a short inter-release
interval would require us to EOL release branches sooner than that to make
things manageable.

Re: [DISCUSS] Time-based releases for Apache Kafka

[Joel Koshy]

On Tue, Aug 9, 2016 at 4:49 PM, Gwen Shapira  wrote:

>
> 4. Frequent releases mean we need to do bugfix releases for older
> branches. Right now we only do bugfix releases to latest release.
>

I'm a bit unclear on how the above is a side-effect of time-based releases.
IIUC this just changes how frequently we release off the current release
branch right? Or put another way, are you also proposing any fundamental
change to our versioning/branching scheme?

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

Hi Gwen,

Comments inline.

On Wed, Aug 10, 2016 at 6:21 PM, Gwen Shapira  wrote:

> I hear what you are saying (enterprises upgrade every 2-years
> more-or-less). It seems reasonable - this basically means maintaining
> 10 compatibility tests at any point in time.


Indeed. Although it's up to 6 tests in each branch because we only have
upgrade tests to the latest release in the given branch (i.e. if the
release branch is 0.10.0, we only have upgrade tests with 0.10.0.x as the
target in that branch). This provides full coverage without having
unnecessary tests in each branch.

We will need to be
> disciplined about maintaining those tests though - or it will get
> painful.
>

Definitely.

Another thing, hurrying up with implementing full forward-and-back
> compatibility for clients (i.e. full KIP-35 support in clients) would
> go a long way toward making upgrades less painful.
>

Yes, I think this would be awesome. It would also reduce the need to
backport client fixes to older branches.

Ismael

Re: [DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

I hear what you are saying (enterprises upgrade every 2-years
more-or-less). It seems reasonable - this basically means maintaining
10 compatibility tests at any point in time. We will need to be
disciplined about maintaining those tests though - or it will get
painful.

Another thing, hurrying up with implementing full forward-and-back
compatibility for clients (i.e. full KIP-35 support in clients) would
go a long way toward making upgrades less painful.

Gwen

On Wed, Aug 10, 2016 at 6:34 AM, Ismael Juma  wrote:
> Hi Gwen,
>
> The proposal sounds good to me. With regards to the cadence, 3 releases a
> year (every 4 months as you said) sounds reasonable. One thing that I think
> is very important if we release more often is that users should be able to
> upgrade directly to the latest release for a reasonable period. For
> example, we could say that we support direct upgrades for 2 years (6
> releases).
>
> Ismael
>
> On Wed, Aug 10, 2016 at 12:49 AM, Gwen Shapira  wrote:
>
>> Dear Kafka Developers and Users,
>>
>> In the past, our releases have been quite unpredictable. We'll notice
>> that a large number of nice features made it in (or are close),
>> someone would suggest a release and we'd do it. This is fun, but makes
>> planning really hard - we saw it during the last release which we
>> decided to delay by a few weeks to allow more features to "land".
>>
>> Many other communities have adopted time-based releases successfully
>> (Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
>> will make sense for the Apache Kafka community to try doing the same.
>>
>> The benefits of this approach are:
>>
>> 1. A quicker feedback cycle and users can benefit from features
>> quicker (assuming for reasonably short time between releases - I was
>> thinking 4 months)
>>
>> 2. Predictability for contributors and users:
>> * Developers and reviewers can decide in advance what release they are
>> aiming for with specific features.
>> * If a feature misses a release we have a good idea of when it will show
>> up.
>> * Users know when to expect their features
>>
>> 3. Transparency - There will be a published cut-off date (AKA feature
>> freeze) for the release and people will know about it in advance.
>> Hopefully this will remove the contention around which features make
>> it.
>>
>> 4. Quality - we've seen issues pop up in release candidates due to
>> last-minute features that didn't have proper time to bake in. More
>> time between feature freeze and release will let us test more,
>> document more and resolve more issues.
>>
>> Since nothing is ever perfect, there will be some downsides:
>>
>> 1. Most notably, features that miss the feature-freeze date for a
>> release will have to wait few month for the next release. Features
>> will reach users faster overall as per benefit #1, but individual
>> features that just miss the cut will lose out
>>
>> 2. More releases a year mean that being a committer is more work -
>> release management is still some headache and we'll have more of
>> those. Hopefully we'll get better at it. Also, the committer list is
>> growing and hopefully it will be less than once-a-year effort for each
>> committer.
>>
>> 3. For users, figuring out which release to use and having frequent
>> new releases to upgrade to may be a bit confusing.
>>
>> 4. Frequent releases mean we need to do bugfix releases for older
>> branches. Right now we only do bugfix releases to latest release.
>>
>> I think the benefits outweigh the drawbacks. Or at least suggest that
>> its worth trying - we can have another discussion in few releases to
>> see if we want to keep it that way or try something else.
>>
>> My suggestion for the process:
>>
>> 1. We decide on a reasonable release cadence
>> 2. We decide on release dates (even rough estimate such as "end of
>> February" or something) and work back feature freeze dates.
>> 3. Committers volunteer to be "release managers" for specific
>> releases. We can coordinate on the list or on a wiki. If no committer
>> volunteers, we assume the community doesn't need a release and skip
>> it.
>> 4. At the "feature freeze" date, the release manager announces the
>> contents of the release (which KIPs made it in on time), creates the
>> release branch and starts the release process as usual. From this
>> point onwards, only bug fixes should be double-committed to the
>> release branch while trunk can start collecting features for the
>> subsequent release.
>>
>> Comments and improvements are appreciated.
>>
>> Gwen Shapira
>> Former-release-manager
>>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] Time-based releases for Apache Kafka

[Ismael Juma]

Hi Gwen,

The proposal sounds good to me. With regards to the cadence, 3 releases a
year (every 4 months as you said) sounds reasonable. One thing that I think
is very important if we release more often is that users should be able to
upgrade directly to the latest release for a reasonable period. For
example, we could say that we support direct upgrades for 2 years (6
releases).

Ismael

On Wed, Aug 10, 2016 at 12:49 AM, Gwen Shapira  wrote:

> Dear Kafka Developers and Users,
>
> In the past, our releases have been quite unpredictable. We'll notice
> that a large number of nice features made it in (or are close),
> someone would suggest a release and we'd do it. This is fun, but makes
> planning really hard - we saw it during the last release which we
> decided to delay by a few weeks to allow more features to "land".
>
> Many other communities have adopted time-based releases successfully
> (Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
> will make sense for the Apache Kafka community to try doing the same.
>
> The benefits of this approach are:
>
> 1. A quicker feedback cycle and users can benefit from features
> quicker (assuming for reasonably short time between releases - I was
> thinking 4 months)
>
> 2. Predictability for contributors and users:
> * Developers and reviewers can decide in advance what release they are
> aiming for with specific features.
> * If a feature misses a release we have a good idea of when it will show
> up.
> * Users know when to expect their features
>
> 3. Transparency - There will be a published cut-off date (AKA feature
> freeze) for the release and people will know about it in advance.
> Hopefully this will remove the contention around which features make
> it.
>
> 4. Quality - we've seen issues pop up in release candidates due to
> last-minute features that didn't have proper time to bake in. More
> time between feature freeze and release will let us test more,
> document more and resolve more issues.
>
> Since nothing is ever perfect, there will be some downsides:
>
> 1. Most notably, features that miss the feature-freeze date for a
> release will have to wait few month for the next release. Features
> will reach users faster overall as per benefit #1, but individual
> features that just miss the cut will lose out
>
> 2. More releases a year mean that being a committer is more work -
> release management is still some headache and we'll have more of
> those. Hopefully we'll get better at it. Also, the committer list is
> growing and hopefully it will be less than once-a-year effort for each
> committer.
>
> 3. For users, figuring out which release to use and having frequent
> new releases to upgrade to may be a bit confusing.
>
> 4. Frequent releases mean we need to do bugfix releases for older
> branches. Right now we only do bugfix releases to latest release.
>
> I think the benefits outweigh the drawbacks. Or at least suggest that
> its worth trying - we can have another discussion in few releases to
> see if we want to keep it that way or try something else.
>
> My suggestion for the process:
>
> 1. We decide on a reasonable release cadence
> 2. We decide on release dates (even rough estimate such as "end of
> February" or something) and work back feature freeze dates.
> 3. Committers volunteer to be "release managers" for specific
> releases. We can coordinate on the list or on a wiki. If no committer
> volunteers, we assume the community doesn't need a release and skip
> it.
> 4. At the "feature freeze" date, the release manager announces the
> contents of the release (which KIPs made it in on time), creates the
> release branch and starts the release process as usual. From this
> point onwards, only bug fixes should be double-committed to the
> release branch while trunk can start collecting features for the
> subsequent release.
>
> Comments and improvements are appreciated.
>
> Gwen Shapira
> Former-release-manager
>

[DISCUSS] Time-based releases for Apache Kafka

[Gwen Shapira]

Dear Kafka Developers and Users,

In the past, our releases have been quite unpredictable. We'll notice
that a large number of nice features made it in (or are close),
someone would suggest a release and we'd do it. This is fun, but makes
planning really hard - we saw it during the last release which we
decided to delay by a few weeks to allow more features to "land".

Many other communities have adopted time-based releases successfully
(Cassandra, GCC, LLVM, Fedora, Gnome, Ubuntu, etc.). And I thought it
will make sense for the Apache Kafka community to try doing the same.

The benefits of this approach are:

1. A quicker feedback cycle and users can benefit from features
quicker (assuming for reasonably short time between releases - I was
thinking 4 months)

2. Predictability for contributors and users:
* Developers and reviewers can decide in advance what release they are
aiming for with specific features.
* If a feature misses a release we have a good idea of when it will show up.
* Users know when to expect their features

3. Transparency - There will be a published cut-off date (AKA feature
freeze) for the release and people will know about it in advance.
Hopefully this will remove the contention around which features make
it.

4. Quality - we've seen issues pop up in release candidates due to
last-minute features that didn't have proper time to bake in. More
time between feature freeze and release will let us test more,
document more and resolve more issues.

Since nothing is ever perfect, there will be some downsides:

1. Most notably, features that miss the feature-freeze date for a
release will have to wait few month for the next release. Features
will reach users faster overall as per benefit #1, but individual
features that just miss the cut will lose out

2. More releases a year mean that being a committer is more work -
release management is still some headache and we'll have more of
those. Hopefully we'll get better at it. Also, the committer list is
growing and hopefully it will be less than once-a-year effort for each
committer.

3. For users, figuring out which release to use and having frequent
new releases to upgrade to may be a bit confusing.

4. Frequent releases mean we need to do bugfix releases for older
branches. Right now we only do bugfix releases to latest release.

I think the benefits outweigh the drawbacks. Or at least suggest that
its worth trying - we can have another discussion in few releases to
see if we want to keep it that way or try something else.

My suggestion for the process:

1. We decide on a reasonable release cadence
2. We decide on release dates (even rough estimate such as "end of
February" or something) and work back feature freeze dates.
3. Committers volunteer to be "release managers" for specific
releases. We can coordinate on the list or on a wiki. If no committer
volunteers, we assume the community doesn't need a release and skip
it.
4. At the "feature freeze" date, the release manager announces the
contents of the release (which KIPs made it in on time), creates the
release branch and starts the release process as usual. From this
point onwards, only bug fixes should be double-committed to the
release branch while trunk can start collecting features for the
subsequent release.

Comments and improvements are appreciated.

Gwen Shapira
Former-release-manager