Re: [ceph-users] Changing the release cadence

2019-07-21 Thread Brent Kennedy 
12 months sounds good to me, I like the idea of march as well since we plan
on doing upgrades in June/July each year.  Gives it time to be discussed and
marinate before we decide to upgrade.

-Brent

-Original Message-
From: ceph-users  On Behalf Of Sage Weil
Sent: Wednesday, June 5, 2019 11:58 AM
To: ceph-us...@ceph.com; ceph-de...@vger.kernel.org; d...@ceph.io
Subject: [ceph-users] Changing the release cadence

Hi everyone,

Since luminous, we have had the follow release cadence and policy:   
 - release every 9 months
 - maintain backports for the last two releases
 - enable upgrades to move either 1 or 2 releases heads
   (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)

This has mostly worked out well, except that the mimic release received less
attention that we wanted due to the fact that multiple downstream Ceph
products (from Red Has and SUSE) decided to based their next release on
nautilus.  Even though upstream every release is an "LTS" release, as a
practical matter mimic got less attention than luminous or nautilus.

We've had several requests/proposals to shift to a 12 month cadence. This
has several advantages:

 - Stable/conservative clusters only have to be upgraded every 2 years
   (instead of every 18 months)
 - Yearly releases are more likely to intersect with downstream
   distribution release (e.g., Debian).  In the past there have been 
   problems where the Ceph releases included in consecutive releases of a 
   distro weren't easily upgradeable.
 - Vendors that make downstream Ceph distributions/products tend to
   release yearly.  Aligning with those vendors means they are more likely 
   to productize *every* Ceph release.  This will help make every Ceph 
   release an "LTS" release (not just in name but also in terms of 
   maintenance attention).

So far the balance of opinion seems to favor a shift to a 12 month cycle[1],
especially among developers, so it seems pretty likely we'll make that
shift.  (If you do have strong concerns about such a move, now is the time
to raise them.)

That brings us to an important decision: what time of year should we
release?  Once we pick the timing, we'll be releasing at that time *every
year* for each release (barring another schedule shift, which we want to
avoid), so let's choose carefully!

A few options:

 - November: If we release Octopus 9 months from the Nautilus release
   (planned for Feb, released in Mar) then we'd target this November.  We 
   could shift to a 12 months candence after that.
 - February: That's 12 months from the Nautilus target.
 - March: That's 12 months from when Nautilus was *actually* released.

November is nice in the sense that we'd wrap things up before the holidays.
It's less good in that users may not be inclined to install the new release
when many developers will be less available in December.

February kind of sucked in that the scramble to get the last few things done
happened during the holidays.  OTOH, we should be doing what we can to avoid
such scrambles, so that might not be something we should factor in.  March
may be a bit more balanced, with a solid 3 months before when people are
productive, and 3 months after before they disappear on holiday to address
any post-release issues.

People tend to be somewhat less available over the summer months due to
holidays etc, so an early or late summer release might also be less than
ideal.

Thoughts?  If we can narrow it down to a few options maybe we could do a
poll to gauge user preferences.

Thanks!
sage


[1] https://twitter.com/larsmb/status/1130010208971952129

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-07-18 Thread Konstantin Shalygin 

Arch Linux packager for Ceph here o/


I take this opportunity to consider the possibility of the appearance 
not in Ceph packaging, but Archlinux+Ceph related.
Currently with Archlinux packaging impossible to build "Samba CTDB 
Cluster with CephFS backend". This caused by lack of build options, 
ticket requests for this ignored for a years: [1], [2]. AFAIK all 
distro's lack of full RADOS support, only exception is SUSE - because 
most of RADOS features comes to Samba from SUSE employees. For CentOS7 
this covered here [3].


May be Thore can raise this question among samba package maintainers.



[1] https://bugs.archlinux.org/task/53467
[2] https://bugs.archlinux.org/task/49356
[3] https://lists.samba.org/archive/samba/2019-July/224288.html

Thanks,
k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-07-15 Thread Sage Weil 
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
> On Mon, Jul 15, 2019 at 10:10 AM Sage Weil  wrote:
> 
> > On Mon, 15 Jul 2019, Kaleb Keithley wrote:
> > >
> > > If Octopus is really an LTS release like all the others, and you want
> > > bleeding edge users to test/use it and give early feedback, then Fedora
> > is
> > > probably one of the better places to get that feedback.
> >
> > I think the first release worth testing outside of the dev community is
> > the release candidate.  I don't like the idea of having any distro carry
> > an untested dev checkpoint or else someone will lose data... even the rc
> > should be tested cautiously and, since it is only relevant for a week or
> > two, I'm not sure that distros can play much of a role there?
> >
> 
> To be clear, when I talk about packaging a new version Ceph, it starts out
> by _only_ going into Fedora Rawhide. It would be extremely foolish, IMO,
> for anyone to run Rawhide for anything that's mission critical.
> 
> Sometimes it takes a long time to work through build and packaging issues.
> The switch to gcc-8 and gcc-9 are good illustrations of the kinds of issues
> that we, as packagers, run into. Those kinds of things are why I — at least
> — like to get as early a start as I can. Even short-lived release
> candidates are useful stepping stones to the eventual GA.
> 
> Personally I'd say that any fears anyone has of people losing data by using
> an early dev checkpoint of Ceph on Rawhide are probably a teeny bit
> misplaced.

Okay, it sounds like the rc x.1.z releases are a good fit, then.  They 
should start about ~2 months before the first stable release.  I'm not so 
sure about the dev checkpoints since they are quite arbitrary.  I suspect 
less effort would be needed to just do a manual build a few times 
partway through the cycle (e.g., now), identify any issues, and open PRs 
with build fixes.

sage___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-07-15 Thread Thore Bödecker 
Hey,

On 15.07.19 09:58, Kaleb Keithley wrote:
> Speaking as (one of) the Ceph packager(s) in Fedora:

Arch Linux packager for Ceph here o/

> If Octopus is really an LTS release like all the others, and you want
> bleeding edge users to test/use it and give early feedback, then Fedora is
> probably one of the better places to get that feedback.
> 
> FWIW, In the absence of a ceph-14.0.z.tar.gz from
> http://download.ceph.com/tarballs/ branto came up with one (possibly from
> https://github.com/ceph/ceph/releases/...???)  but I haven't had any luck
> building ceph-15 with the tarball from there.

I've been watching the RSS feed of the GitHub releases and use the
GitHub release tarballs for building it.

Due to unforseen complications and lack of time (similar situation
like you) the last packaged version of ceph in the Arch Linux repos is
13.2.1 right now.
Over the last couple of weeks I have spent numerous hours and worked
with people in oftc.#ceph-devel to work on the issues I was
experiencing while building it.
Right now I'm finalizing the 14.2.1 build and have some last kinks to
work out, for some reason the zstd compression tests are failing for
me, gonna look into that over the coming days.

Seeing that Arch Linux and Fedora should have a few more things common
then, let's say Ubuntu or Debian, I would like to offer my help
however I can. I'm not that familiar with building packages on/for
Fedora but I've had my fair share of debugging building ceph with
bleeding edge components (gcc, cmake, boost, python) and could
possibly help out there.


Feel free to get in touch, maybe we can help each other out.

Cheers,
Thore


PS: I was thinking about maybe getting Arch Linux (and Fedora)
onboarded to the ceph-dev-build matrix on jenkins.ceph.com.
Thinking about the possibility to spot arising issues and/or
incompatibilities with upstream / bleeding edge compilers, helpers and
dependencies that should have common intereset with Ceph upstream.
Although I haven't talked to anyone upstream about this and am just
throwing it out here.

-- 
Thore Bödecker

GPG ID: 0xD622431AF8DB80F3
GPG FP: 0F96 559D 3556 24FC 2226  A864 D622 431A F8DB 80F3


signature.asc
Description: PGP signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-07-15 Thread Sage Weil 
On Mon, 15 Jul 2019, Kaleb Keithley wrote:
> On Wed, Jun 5, 2019 at 11:58 AM Sage Weil  wrote:
> 
> > ...
> >
> > This has mostly worked out well, except that the mimic release received
> > less attention that we wanted due to the fact that multiple downstream
> > Ceph products (from Red Has and SUSE) decided to based their next release
> > on nautilus.  Even though upstream every release is an "LTS" release, as a
> > practical matter mimic got less attention than luminous or nautilus.
> >
> 
> Speaking as (one of) the Ceph packager(s) in Fedora:
> 
> 1) I have been told (don't recall by who now) that odd numbered releases
> are experimental or short term support or red haired step child releases.

In principle that hasn't been the case since 12.2.z (luminous).  13 has 
gotten somewhat less backport attention because the people doing backports 
work for Red Hat and SUSE and neither shipped a product based on 13.

> 2) even though there have been ceph-{10,11,12}.0.z tarballs on
> http://download.ceph.com/tarballs/ in the past, there still isn't a
> ceph-15.0.0.tar.gz there. And I was told that there wouldn't be one. Yes,
> I'm aware that X.0.z are alpha releases, X.1.z are beta releases, and X.2.z
> are GA releases.

In the past we haven't published tarballs until x.1.z (rc), since the dev 
checkpoints are not meant to be used by anyone other than developers.  
(They're really just a 'pause and make sure the qa suites are nice and 
clean', nothing more.)

> 3) ceph-13 was skipped over in Fedora because I could never get it to build
> in Fedora in the limited time I make to do upstream packaging of Ceph and
> other things (because it's not my $dayjob.) And there seemed to be no
> interest on the part of anyone in the Ceph devel community to fix it
> (because see #1?)
> 
> If Octopus is really an LTS release like all the others, and you want
> bleeding edge users to test/use it and give early feedback, then Fedora is
> probably one of the better places to get that feedback.

I think the first release worth testing outside of the dev community is 
the release candidate.  I don't like the idea of having any distro carry 
an untested dev checkpoint or else someone will lose data... even the rc 
should be tested cautiously and, since it is only relevant for a week or 
two, I'm not sure that distros can play much of a role there?

sage


> FWIW, In the absence of a ceph-14.0.z.tar.gz from
> http://download.ceph.com/tarballs/ branto came up with one (possibly from
> https://github.com/ceph/ceph/releases/...???)  but I haven't had any luck
> building ceph-15 with the tarball from there.
> 
> My 2¢.
> ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-26 Thread Lars Marowsky-Bree 
On 2019-06-26T14:45:31, Sage Weil  wrote:

Hi Sage,

I think that makes sense. I'd have preferred the Oct/Nov target, but
that'd have made Octopus quite short.

Unsure whether freezing in December with a release in March is too long
though. But given how much people scramble, setting that as a goal
probably will help with stabilization.

I'm also hoping that one day, we can move towards a more agile
continuously integration model (like the Linux kernel does) instead of
massive yearly forklifts. But hey, that's just me ;-)



Regards,
Lars

-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG 
Nürnberg)
"Architects should open possibilities and not determine everything." (Ueli 
Zbinden)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-26 Thread Bob Farrell 
March seems sensible to me for the reasons you stated. If a release gets
delayed, I'd prefer it to be on the spring side of Christmas (again for the
reasons already mentioned).

That aside, I'm now very impatient to install Octopus on my 8-node cluster.
: )

On Wed, 26 Jun 2019 at 15:46, Sage Weil  wrote:

> Hi everyone,
>
> We talked a bit about this during the CLT meeting this morning.  How about
> the following proposal:
>
> - Target release date of Mar 1 each year.
> - Target freeze in Dec.  That will allow us to use the holidays to do a
>   lot of testing when the lab infrastructure tends to be somewhat idle.
>
> If we get an early build out at the point of the freeze (or even earlier),
> perhaps this capture some of the time that the retailers have during their
> lockdown to identify structural issues with release.  It is probably
> better to do more of this testing at this point in the cycle so that we
> have time to properly fix any big issues (like performance or scaling
> regressions).  It is of course a challenge to motivate testing on
> something that is too far from the final a release, but we can try.
>
> This avoids an abbreviated octopus cycle, and avoids placing August (which
> also often has people out for vacations) right in the middle of the
> lead-up to the freeze.
>
> Thoughts?
> sage
>
>
>
> On Wed, 26 Jun 2019, Sage Weil wrote:
>
> > On Wed, 26 Jun 2019, Alfonso Martinez Hidalgo wrote:
> > > I think March is a good idea.
> >
> > Spring had a slight edge over fall in the twitter poll (for whatever
> > that's worth).  I see the appeal for fall when it comes to down time
> for
> > retailers, but as a practical matter for Octopus specifically, a target
> of
> > say October means freezing in August, which means we only have 2
> > more months of development time.  I'm worried that will turn Octopus
> > in another weak (aka lightly adopted) release.
> >
> > March would mean freezing in January again, which would give us July to
> > Dec... 6 more months.
> >
> > sage
> >
> >
> >
> > >
> > > On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza  wrote:
> > >
> > > > On Mon, Jun 17, 2019 at 4:09 PM David Turner 
> > > > wrote:
> > > > >
> > > > > This was a little long to respond with on Twitter, so I thought I'd
> > > > share my thoughts here. I love the idea of a 12 month cadence. I like
> > > > October because admins aren't upgrading production within the first
> few
> > > > months of a new release. It gives it plenty of time to be stable for
> the OS
> > > > distros as well as giving admins something low-key to work on over
> the
> > > > holidays with testing the new releases in stage/QA.
> > > >
> > > > October sounds ideal, but in reality, we haven't been able to release
> > > > right on time as long as I can remember. Realistically, if we set
> > > > October, we are probably going to get into November/December.
> > > >
> > > > For example, Nautilus was set to release in February and we got it
> out
> > > > late in late March (Almost April)
> > > >
> > > > Would love to see more of a discussion around solving the problem of
> > > > releasing when we say we are going to - so that we can then choose
> > > > what the cadence is.
> > > >
> > > > >
> > > > > On Mon, Jun 17, 2019 at 12:22 PM Sage Weil 
> wrote:
> > > > >>
> > > > >> On Wed, 5 Jun 2019, Sage Weil wrote:
> > > > >> > That brings us to an important decision: what time of year
> should we
> > > > >> > release?  Once we pick the timing, we'll be releasing at that
> time
> > > > *every
> > > > >> > year* for each release (barring another schedule shift, which
> we want
> > > > to
> > > > >> > avoid), so let's choose carefully!
> > > > >>
> > > > >> I've put up a twitter poll:
> > > > >>
> > > > >> https://twitter.com/liewegas/status/1140655233430970369
> > > > >>
> > > > >> Thanks!
> > > > >> sage
> > > > >> ___
> > > > >> ceph-users mailing list
> > > > >> ceph-users@lists.ceph.com
> > > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > > >
> > > > > ___
> > > > > ceph-users mailing list
> > > > > ceph-users@lists.ceph.com
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > >
> > >
> > >
> > > --
> > >
> > > Alfonso Martínez
> > >
> > > Senior Software Engineer, Ceph Storage
> > >
> > > Red Hat 
> > > 
> > > ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-26 Thread Sage Weil 
Hi everyone,

We talked a bit about this during the CLT meeting this morning.  How about 
the following proposal:

- Target release date of Mar 1 each year.
- Target freeze in Dec.  That will allow us to use the holidays to do a 
  lot of testing when the lab infrastructure tends to be somewhat idle.

If we get an early build out at the point of the freeze (or even earlier), 
perhaps this capture some of the time that the retailers have during their 
lockdown to identify structural issues with release.  It is probably 
better to do more of this testing at this point in the cycle so that we 
have time to properly fix any big issues (like performance or scaling 
regressions).  It is of course a challenge to motivate testing on 
something that is too far from the final a release, but we can try.

This avoids an abbreviated octopus cycle, and avoids placing August (which 
also often has people out for vacations) right in the middle of the 
lead-up to the freeze.

Thoughts?
sage



On Wed, 26 Jun 2019, Sage Weil wrote:

> On Wed, 26 Jun 2019, Alfonso Martinez Hidalgo wrote:
> > I think March is a good idea.
> 
> Spring had a slight edge over fall in the twitter poll (for whatever 
> that's worth).  I see the appeal for fall when it comes to down time for  
> retailers, but as a practical matter for Octopus specifically, a target of
> say October means freezing in August, which means we only have 2
> more months of development time.  I'm worried that will turn Octopus 
> in another weak (aka lightly adopted) release.
> 
> March would mean freezing in January again, which would give us July to 
> Dec... 6 more months.
> 
> sage
> 
> 
> 
> > 
> > On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza  wrote:
> > 
> > > On Mon, Jun 17, 2019 at 4:09 PM David Turner 
> > > wrote:
> > > >
> > > > This was a little long to respond with on Twitter, so I thought I'd
> > > share my thoughts here. I love the idea of a 12 month cadence. I like
> > > October because admins aren't upgrading production within the first few
> > > months of a new release. It gives it plenty of time to be stable for the 
> > > OS
> > > distros as well as giving admins something low-key to work on over the
> > > holidays with testing the new releases in stage/QA.
> > >
> > > October sounds ideal, but in reality, we haven't been able to release
> > > right on time as long as I can remember. Realistically, if we set
> > > October, we are probably going to get into November/December.
> > >
> > > For example, Nautilus was set to release in February and we got it out
> > > late in late March (Almost April)
> > >
> > > Would love to see more of a discussion around solving the problem of
> > > releasing when we say we are going to - so that we can then choose
> > > what the cadence is.
> > >
> > > >
> > > > On Mon, Jun 17, 2019 at 12:22 PM Sage Weil  wrote:
> > > >>
> > > >> On Wed, 5 Jun 2019, Sage Weil wrote:
> > > >> > That brings us to an important decision: what time of year should we
> > > >> > release?  Once we pick the timing, we'll be releasing at that time
> > > *every
> > > >> > year* for each release (barring another schedule shift, which we want
> > > to
> > > >> > avoid), so let's choose carefully!
> > > >>
> > > >> I've put up a twitter poll:
> > > >>
> > > >> https://twitter.com/liewegas/status/1140655233430970369
> > > >>
> > > >> Thanks!
> > > >> sage
> > > >> ___
> > > >> ceph-users mailing list
> > > >> ceph-users@lists.ceph.com
> > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > >
> > > > ___
> > > > ceph-users mailing list
> > > > ceph-users@lists.ceph.com
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > 
> > 
> > -- 
> > 
> > Alfonso Martínez
> > 
> > Senior Software Engineer, Ceph Storage
> > 
> > Red Hat 
> > 
> > ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-26 Thread Sage Weil 
On Wed, 26 Jun 2019, Alfonso Martinez Hidalgo wrote:
> I think March is a good idea.

Spring had a slight edge over fall in the twitter poll (for whatever 
that's worth).  I see the appeal for fall when it comes to down time for  
retailers, but as a practical matter for Octopus specifically, a target of
say October means freezing in August, which means we only have 2
more months of development time.  I'm worried that will turn Octopus 
in another weak (aka lightly adopted) release.

March would mean freezing in January again, which would give us July to 
Dec... 6 more months.

sage



> 
> On Tue, Jun 25, 2019 at 4:32 PM Alfredo Deza  wrote:
> 
> > On Mon, Jun 17, 2019 at 4:09 PM David Turner 
> > wrote:
> > >
> > > This was a little long to respond with on Twitter, so I thought I'd
> > share my thoughts here. I love the idea of a 12 month cadence. I like
> > October because admins aren't upgrading production within the first few
> > months of a new release. It gives it plenty of time to be stable for the OS
> > distros as well as giving admins something low-key to work on over the
> > holidays with testing the new releases in stage/QA.
> >
> > October sounds ideal, but in reality, we haven't been able to release
> > right on time as long as I can remember. Realistically, if we set
> > October, we are probably going to get into November/December.
> >
> > For example, Nautilus was set to release in February and we got it out
> > late in late March (Almost April)
> >
> > Would love to see more of a discussion around solving the problem of
> > releasing when we say we are going to - so that we can then choose
> > what the cadence is.
> >
> > >
> > > On Mon, Jun 17, 2019 at 12:22 PM Sage Weil  wrote:
> > >>
> > >> On Wed, 5 Jun 2019, Sage Weil wrote:
> > >> > That brings us to an important decision: what time of year should we
> > >> > release?  Once we pick the timing, we'll be releasing at that time
> > *every
> > >> > year* for each release (barring another schedule shift, which we want
> > to
> > >> > avoid), so let's choose carefully!
> > >>
> > >> I've put up a twitter poll:
> > >>
> > >> https://twitter.com/liewegas/status/1140655233430970369
> > >>
> > >> Thanks!
> > >> sage
> > >> ___
> > >> ceph-users mailing list
> > >> ceph-users@lists.ceph.com
> > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> 
> 
> -- 
> 
> Alfonso Martínez
> 
> Senior Software Engineer, Ceph Storage
> 
> Red Hat 
> 
> ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-26 Thread Sage Weil 
On Tue, 25 Jun 2019, Alfredo Deza wrote:
> On Mon, Jun 17, 2019 at 4:09 PM David Turner  wrote:
> >
> > This was a little long to respond with on Twitter, so I thought I'd share 
> > my thoughts here. I love the idea of a 12 month cadence. I like October 
> > because admins aren't upgrading production within the first few months of a 
> > new release. It gives it plenty of time to be stable for the OS distros as 
> > well as giving admins something low-key to work on over the holidays with 
> > testing the new releases in stage/QA.
> 
> October sounds ideal, but in reality, we haven't been able to release
> right on time as long as I can remember. Realistically, if we set
> October, we are probably going to get into November/December.
> 
> For example, Nautilus was set to release in February and we got it out
> late in late March (Almost April)
> 
> Would love to see more of a discussion around solving the problem of
> releasing when we say we are going to - so that we can then choose
> what the cadence is.

I think the "on time" part is solveable.  We should just use the amount 
of time between take the previous release's freeze date and the 
target release date and go with that.  It is a bit fuzzy because I left it 
up to the leads how they handle the freeze, but I think mid-Januaray is 
about right (in reality we waiting longer than that for lots of RADOS 
stuff).  v14.2.0 was Mar 18, so ~2 months.

The cadence is really separate from that, though: even if every release 
were 2 full months late, if we start with the same target it's still a 1 
year cycle.

sage

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-25 Thread Alfredo Deza 
On Mon, Jun 17, 2019 at 4:09 PM David Turner  wrote:
>
> This was a little long to respond with on Twitter, so I thought I'd share my 
> thoughts here. I love the idea of a 12 month cadence. I like October because 
> admins aren't upgrading production within the first few months of a new 
> release. It gives it plenty of time to be stable for the OS distros as well 
> as giving admins something low-key to work on over the holidays with testing 
> the new releases in stage/QA.

October sounds ideal, but in reality, we haven't been able to release
right on time as long as I can remember. Realistically, if we set
October, we are probably going to get into November/December.

For example, Nautilus was set to release in February and we got it out
late in late March (Almost April)

Would love to see more of a discussion around solving the problem of
releasing when we say we are going to - so that we can then choose
what the cadence is.

>
> On Mon, Jun 17, 2019 at 12:22 PM Sage Weil  wrote:
>>
>> On Wed, 5 Jun 2019, Sage Weil wrote:
>> > That brings us to an important decision: what time of year should we
>> > release?  Once we pick the timing, we'll be releasing at that time *every
>> > year* for each release (barring another schedule shift, which we want to
>> > avoid), so let's choose carefully!
>>
>> I've put up a twitter poll:
>>
>> https://twitter.com/liewegas/status/1140655233430970369
>>
>> Thanks!
>> sage
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-17 Thread Daniel Baumann 
Hi,

I didn't bother to create a twitter account just to be able to
participate in the poll.. so.. please count me in for October.

Regards,
Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-17 Thread David Turner 
This was a little long to respond with on Twitter, so I thought I'd share
my thoughts here. I love the idea of a 12 month cadence. I like October
because admins aren't upgrading production within the first few months of a
new release. It gives it plenty of time to be stable for the OS distros as
well as giving admins something low-key to work on over the holidays with
testing the new releases in stage/QA.

On Mon, Jun 17, 2019 at 12:22 PM Sage Weil  wrote:

> On Wed, 5 Jun 2019, Sage Weil wrote:
> > That brings us to an important decision: what time of year should we
> > release?  Once we pick the timing, we'll be releasing at that time
> *every
> > year* for each release (barring another schedule shift, which we want to
> > avoid), so let's choose carefully!
>
> I've put up a twitter poll:
>
> https://twitter.com/liewegas/status/1140655233430970369
>
> Thanks!
> sage
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-17 Thread Sage Weil 
On Wed, 5 Jun 2019, Sage Weil wrote:
> That brings us to an important decision: what time of year should we 
> release?  Once we pick the timing, we'll be releasing at that time *every 
> year* for each release (barring another schedule shift, which we want to 
> avoid), so let's choose carefully!

I've put up a twitter poll:

https://twitter.com/liewegas/status/1140655233430970369

Thanks!
sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-06 Thread Daniel Baumann 
On 6/6/19 9:26 AM, Xiaoxi Chen wrote:
> I will vote for November for several reasons:

[...]

as an academic institution we're aligned by August to July (school year)
instead of the January to December (calendar year), so all your reasons
(thanks!) are valid for us.. just shifted by 6 months, hence Q1 is ideal
for us.

however, given that academic institutions are the minority, I'm
convinced now that November is the better choice for everyone.

Regards,
Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-06 Thread Xiaoxi Chen 
We go with upstream release and mostly Nautilus now, probably the most
aggressive ones among serious production user (i.e tens of PB+ ),

I will vote for November for several reasons:

 1.   Q4 is holiday season and usually production rollout was blocked
, especially storage related change, which usually give team more time
to prepare/ testing/ LnP the new releases, as well as catch up with
new features.

 2.  Q4/Q1 is usually the planning season,  having the upstream
released and testing to know the readiness of new feature, will
greatly helps when planning the feature/offering of next year.

 3.  Users have whole year to migrate their
provision/monitoring/deployment/remediation system to new version, and
have enough time to fix and stable the surrounding system before next
holiday season.

Release in Feb or March will make the Q4 just in the middle of the
cycle, and lot of changes will land at last minutes(month),   in which
case, few things can be test and forecasted based on the state-of-art
in Q4.

-Xiaoxi

Linh Vu  于2019年6月6日周四 上午8:32写道：
>
> I think 12 months cycle is much better from the cluster operations 
> perspective. I also like March as a release month as well.
> 
> From: ceph-users  on behalf of Sage Weil 
> 
> Sent: Thursday, 6 June 2019 1:57 AM
> To: ceph-us...@ceph.com; ceph-de...@vger.kernel.org; d...@ceph.io
> Subject: [ceph-users] Changing the release cadence
>
> Hi everyone,
>
> Since luminous, we have had the follow release cadence and policy:
>  - release every 9 months
>  - maintain backports for the last two releases
>  - enable upgrades to move either 1 or 2 releases heads
>(e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)
>
> This has mostly worked out well, except that the mimic release received
> less attention that we wanted due to the fact that multiple downstream
> Ceph products (from Red Has and SUSE) decided to based their next release
> on nautilus.  Even though upstream every release is an "LTS" release, as a
> practical matter mimic got less attention than luminous or nautilus.
>
> We've had several requests/proposals to shift to a 12 month cadence. This
> has several advantages:
>
>  - Stable/conservative clusters only have to be upgraded every 2 years
>(instead of every 18 months)
>  - Yearly releases are more likely to intersect with downstream
>distribution release (e.g., Debian).  In the past there have been
>problems where the Ceph releases included in consecutive releases of a
>distro weren't easily upgradeable.
>  - Vendors that make downstream Ceph distributions/products tend to
>release yearly.  Aligning with those vendors means they are more likely
>to productize *every* Ceph release.  This will help make every Ceph
>release an "LTS" release (not just in name but also in terms of
>maintenance attention).
>
> So far the balance of opinion seems to favor a shift to a 12 month
> cycle[1], especially among developers, so it seems pretty likely we'll
> make that shift.  (If you do have strong concerns about such a move, now
> is the time to raise them.)
>
> That brings us to an important decision: what time of year should we
> release?  Once we pick the timing, we'll be releasing at that time *every
> year* for each release (barring another schedule shift, which we want to
> avoid), so let's choose carefully!
>
> A few options:
>
>  - November: If we release Octopus 9 months from the Nautilus release
>(planned for Feb, released in Mar) then we'd target this November.  We
>could shift to a 12 months candence after that.
>  - February: That's 12 months from the Nautilus target.
>  - March: That's 12 months from when Nautilus was *actually* released.
>
> November is nice in the sense that we'd wrap things up before the
> holidays.  It's less good in that users may not be inclined to install the
> new release when many developers will be less available in December.
>
> February kind of sucked in that the scramble to get the last few things
> done happened during the holidays.  OTOH, we should be doing what we can
> to avoid such scrambles, so that might not be something we should factor
> in.  March may be a bit more balanced, with a solid 3 months before when
> people are productive, and 3 months after before they disappear on holiday
> to address any post-release issues.
>
> People tend to be somewhat less available over the summer months due to
> holidays etc, so an early or late summer release might also be less than
> ideal.
>
> Thoughts?  If we can narrow it down to a few options maybe we could do a
> poll to gauge user preferences.
>
> Thanks!
> sage
>
>
> [1] 
> https://protect-au.mimecast.com/s/N1l6CROAEns1RN1Zu9Jwts?domain=twitter.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ___
> ceph-users

Re: [ceph-users] Changing the release cadence

2019-06-06 Thread Dietmar Rieder 
+1
Operators view: 12 months cycle is definitely better than 9. March seem
to be a reasonable compromise.

Best
  Dietmar

On 6/6/19 2:31 AM, Linh Vu wrote:
> I think 12 months cycle is much better from the cluster operations
> perspective. I also like March as a release month as well. 
> 
> *From:* ceph-users  on behalf of Sage
> Weil 
> *Sent:* Thursday, 6 June 2019 1:57 AM
> *To:* ceph-us...@ceph.com; ceph-de...@vger.kernel.org; d...@ceph.io
> *Subject:* [ceph-users] Changing the release cadence
>  
> Hi everyone,
> 
> Since luminous, we have had the follow release cadence and policy:  
>  - release every 9 months
>  - maintain backports for the last two releases
>  - enable upgrades to move either 1 or 2 releases heads
>    (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)
> 
> This has mostly worked out well, except that the mimic release received
> less attention that we wanted due to the fact that multiple downstream
> Ceph products (from Red Has and SUSE) decided to based their next release
> on nautilus.  Even though upstream every release is an "LTS" release, as a
> practical matter mimic got less attention than luminous or nautilus.
> 
> We've had several requests/proposals to shift to a 12 month cadence. This
> has several advantages:
> 
>  - Stable/conservative clusters only have to be upgraded every 2 years
>    (instead of every 18 months)
>  - Yearly releases are more likely to intersect with downstream
>    distribution release (e.g., Debian).  In the past there have been
>    problems where the Ceph releases included in consecutive releases of a
>    distro weren't easily upgradeable.
>  - Vendors that make downstream Ceph distributions/products tend to
>    release yearly.  Aligning with those vendors means they are more likely
>    to productize *every* Ceph release.  This will help make every Ceph
>    release an "LTS" release (not just in name but also in terms of
>    maintenance attention).
> 
> So far the balance of opinion seems to favor a shift to a 12 month
> cycle[1], especially among developers, so it seems pretty likely we'll
> make that shift.  (If you do have strong concerns about such a move, now
> is the time to raise them.)
> 
> That brings us to an important decision: what time of year should we
> release?  Once we pick the timing, we'll be releasing at that time *every
> year* for each release (barring another schedule shift, which we want to
> avoid), so let's choose carefully!
> 
> A few options:
> 
>  - November: If we release Octopus 9 months from the Nautilus release
>    (planned for Feb, released in Mar) then we'd target this November.  We
>    could shift to a 12 months candence after that.
>  - February: That's 12 months from the Nautilus target.
>  - March: That's 12 months from when Nautilus was *actually* released.
> 
> November is nice in the sense that we'd wrap things up before the
> holidays.  It's less good in that users may not be inclined to install the
> new release when many developers will be less available in December.
> 
> February kind of sucked in that the scramble to get the last few things
> done happened during the holidays.  OTOH, we should be doing what we can
> to avoid such scrambles, so that might not be something we should factor
> in.  March may be a bit more balanced, with a solid 3 months before when
> people are productive, and 3 months after before they disappear on holiday
> to address any post-release issues.
> 
> People tend to be somewhat less available over the summer months due to
> holidays etc, so an early or late summer release might also be less than
> ideal.
> 
> Thoughts?  If we can narrow it down to a few options maybe we could do a
> poll to gauge user preferences.
> 
> Thanks!
> sage
> 
> 
> [1]
> https://protect-au.mimecast.com/s/N1l6CROAEns1RN1Zu9Jwts?domain=twitter.com
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-05 Thread Linh Vu 
I think 12 months cycle is much better from the cluster operations perspective. 
I also like March as a release month as well.

From: ceph-users  on behalf of Sage Weil 

Sent: Thursday, 6 June 2019 1:57 AM
To: ceph-us...@ceph.com; ceph-de...@vger.kernel.org; d...@ceph.io
Subject: [ceph-users] Changing the release cadence

Hi everyone,

Since luminous, we have had the follow release cadence and policy:
 - release every 9 months
 - maintain backports for the last two releases
 - enable upgrades to move either 1 or 2 releases heads
   (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)

This has mostly worked out well, except that the mimic release received
less attention that we wanted due to the fact that multiple downstream
Ceph products (from Red Has and SUSE) decided to based their next release
on nautilus.  Even though upstream every release is an "LTS" release, as a
practical matter mimic got less attention than luminous or nautilus.

We've had several requests/proposals to shift to a 12 month cadence. This
has several advantages:

 - Stable/conservative clusters only have to be upgraded every 2 years
   (instead of every 18 months)
 - Yearly releases are more likely to intersect with downstream
   distribution release (e.g., Debian).  In the past there have been
   problems where the Ceph releases included in consecutive releases of a
   distro weren't easily upgradeable.
 - Vendors that make downstream Ceph distributions/products tend to
   release yearly.  Aligning with those vendors means they are more likely
   to productize *every* Ceph release.  This will help make every Ceph
   release an "LTS" release (not just in name but also in terms of
   maintenance attention).

So far the balance of opinion seems to favor a shift to a 12 month
cycle[1], especially among developers, so it seems pretty likely we'll
make that shift.  (If you do have strong concerns about such a move, now
is the time to raise them.)

That brings us to an important decision: what time of year should we
release?  Once we pick the timing, we'll be releasing at that time *every
year* for each release (barring another schedule shift, which we want to
avoid), so let's choose carefully!

A few options:

 - November: If we release Octopus 9 months from the Nautilus release
   (planned for Feb, released in Mar) then we'd target this November.  We
   could shift to a 12 months candence after that.
 - February: That's 12 months from the Nautilus target.
 - March: That's 12 months from when Nautilus was *actually* released.

November is nice in the sense that we'd wrap things up before the
holidays.  It's less good in that users may not be inclined to install the
new release when many developers will be less available in December.

February kind of sucked in that the scramble to get the last few things
done happened during the holidays.  OTOH, we should be doing what we can
to avoid such scrambles, so that might not be something we should factor
in.  March may be a bit more balanced, with a solid 3 months before when
people are productive, and 3 months after before they disappear on holiday
to address any post-release issues.

People tend to be somewhat less available over the summer months due to
holidays etc, so an early or late summer release might also be less than
ideal.

Thoughts?  If we can narrow it down to a few options maybe we could do a
poll to gauge user preferences.

Thanks!
sage


[1] https://twitter.com/larsmb/status/1130010208971952129

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-05 Thread Chris Taylor 


It seems like since the change to the 9 months cadence it has been bumpy 
for the Debian based installs. Changing to a 12 month cadence sounds 
like a good idea. Perhaps some Debian maintainers can suggest a good 
month for them to get the packages in time for their release cycle.



On 2019-06-05 12:16 pm, Alexandre DERUMIER wrote:

Hi,



- November: If we release Octopus 9 months from the Nautilus release
(planned for Feb, released in Mar) then we'd target this November. We
could shift to a 12 months candence after that.


For the 2 last debian releases, the freeze was around january-february,
november seem to be a good time for ceph release.

- Mail original -
De: "Sage Weil" 
À: "ceph-users" , "ceph-devel"
, d...@ceph.io
Envoyé: Mercredi 5 Juin 2019 17:57:52
Objet: Changing the release cadence

Hi everyone,

Since luminous, we have had the follow release cadence and policy:
- release every 9 months
- maintain backports for the last two releases
- enable upgrades to move either 1 or 2 releases heads
(e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; 
...)


This has mostly worked out well, except that the mimic release received
less attention that we wanted due to the fact that multiple downstream
Ceph products (from Red Has and SUSE) decided to based their next 
release
on nautilus. Even though upstream every release is an "LTS" release, as 
a

practical matter mimic got less attention than luminous or nautilus.

We've had several requests/proposals to shift to a 12 month cadence. 
This

has several advantages:

- Stable/conservative clusters only have to be upgraded every 2 years
(instead of every 18 months)
- Yearly releases are more likely to intersect with downstream
distribution release (e.g., Debian). In the past there have been
problems where the Ceph releases included in consecutive releases of a
distro weren't easily upgradeable.
- Vendors that make downstream Ceph distributions/products tend to
release yearly. Aligning with those vendors means they are more likely
to productize *every* Ceph release. This will help make every Ceph
release an "LTS" release (not just in name but also in terms of
maintenance attention).

So far the balance of opinion seems to favor a shift to a 12 month
cycle[1], especially among developers, so it seems pretty likely we'll
make that shift. (If you do have strong concerns about such a move, now
is the time to raise them.)

That brings us to an important decision: what time of year should we
release? Once we pick the timing, we'll be releasing at that time 
*every
year* for each release (barring another schedule shift, which we want 
to

avoid), so let's choose carefully!

A few options:

- November: If we release Octopus 9 months from the Nautilus release
(planned for Feb, released in Mar) then we'd target this November. We
could shift to a 12 months candence after that.
- February: That's 12 months from the Nautilus target.
- March: That's 12 months from when Nautilus was *actually* released.

November is nice in the sense that we'd wrap things up before the
holidays. It's less good in that users may not be inclined to install 
the

new release when many developers will be less available in December.

February kind of sucked in that the scramble to get the last few things
done happened during the holidays. OTOH, we should be doing what we can
to avoid such scrambles, so that might not be something we should 
factor

in. March may be a bit more balanced, with a solid 3 months before when
people are productive, and 3 months after before they disappear on 
holiday

to address any post-release issues.

People tend to be somewhat less available over the summer months due to
holidays etc, so an early or late summer release might also be less 
than

ideal.

Thoughts? If we can narrow it down to a few options maybe we could do a
poll to gauge user preferences.

Thanks!
sage


[1] https://twitter.com/larsmb/status/1130010208971952129

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-05 Thread Alexandre DERUMIER 
Hi,


>>- November: If we release Octopus 9 months from the Nautilus release 
>>(planned for Feb, released in Mar) then we'd target this November. We 
>>could shift to a 12 months candence after that. 

For the 2 last debian releases, the freeze was around january-february,
november seem to be a good time for ceph release.

- Mail original -
De: "Sage Weil" 
À: "ceph-users" , "ceph-devel" 
, d...@ceph.io
Envoyé: Mercredi 5 Juin 2019 17:57:52
Objet: Changing the release cadence

Hi everyone, 

Since luminous, we have had the follow release cadence and policy: 
- release every 9 months 
- maintain backports for the last two releases 
- enable upgrades to move either 1 or 2 releases heads 
(e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...) 

This has mostly worked out well, except that the mimic release received 
less attention that we wanted due to the fact that multiple downstream 
Ceph products (from Red Has and SUSE) decided to based their next release 
on nautilus. Even though upstream every release is an "LTS" release, as a 
practical matter mimic got less attention than luminous or nautilus. 

We've had several requests/proposals to shift to a 12 month cadence. This 
has several advantages: 

- Stable/conservative clusters only have to be upgraded every 2 years 
(instead of every 18 months) 
- Yearly releases are more likely to intersect with downstream 
distribution release (e.g., Debian). In the past there have been 
problems where the Ceph releases included in consecutive releases of a 
distro weren't easily upgradeable. 
- Vendors that make downstream Ceph distributions/products tend to 
release yearly. Aligning with those vendors means they are more likely 
to productize *every* Ceph release. This will help make every Ceph 
release an "LTS" release (not just in name but also in terms of 
maintenance attention). 

So far the balance of opinion seems to favor a shift to a 12 month 
cycle[1], especially among developers, so it seems pretty likely we'll 
make that shift. (If you do have strong concerns about such a move, now 
is the time to raise them.) 

That brings us to an important decision: what time of year should we 
release? Once we pick the timing, we'll be releasing at that time *every 
year* for each release (barring another schedule shift, which we want to 
avoid), so let's choose carefully! 

A few options: 

- November: If we release Octopus 9 months from the Nautilus release 
(planned for Feb, released in Mar) then we'd target this November. We 
could shift to a 12 months candence after that. 
- February: That's 12 months from the Nautilus target. 
- March: That's 12 months from when Nautilus was *actually* released. 

November is nice in the sense that we'd wrap things up before the 
holidays. It's less good in that users may not be inclined to install the 
new release when many developers will be less available in December. 

February kind of sucked in that the scramble to get the last few things 
done happened during the holidays. OTOH, we should be doing what we can 
to avoid such scrambles, so that might not be something we should factor 
in. March may be a bit more balanced, with a solid 3 months before when 
people are productive, and 3 months after before they disappear on holiday 
to address any post-release issues. 

People tend to be somewhat less available over the summer months due to 
holidays etc, so an early or late summer release might also be less than 
ideal. 

Thoughts? If we can narrow it down to a few options maybe we could do a 
poll to gauge user preferences. 

Thanks! 
sage 


[1] https://twitter.com/larsmb/status/1130010208971952129 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

2019-06-05 Thread Daniel Baumann 
On 6/5/19 5:57 PM, Sage Weil wrote:
> So far the balance of opinion seems to favor a shift to a 12 month 
> cycle [...] it seems pretty likely we'll make that shift.

thanks, much appreciated (from an cluster operating point of view).

> Thoughts?

GNOME and a few others are doing April and October releases which seems
balanced and to be good timing for most people; personally I prefer
spring rather than autum for upgrades, hence.. would suggest April.

Regards,
Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com