I start to think that Druid versioning doesn't need to be semantic and doesn't need a distinction between major and minor. A release is just a release, like IntelliJ's 2018.1, 2018.2, 2018.3. Every release could have some extensions API broken. If there is an operations breaking change, we could commit to support rollback strategies e. g. for 1 year of releases after that, that means in the next four releases (if we stick to 3-month cadence).
On Fri, 21 Dec 2018 at 19:10, Charles Allen <cral...@apache.org> wrote: > If I'm greedily honest, I don't want to maintain that many backport > channels. I'd rather have "If you want XYZ backport for version 14, then > you have to take the latest minor version for 14" and have a policy to > where someone can upgrade from 14.x --> 14.latest with (hopefully) no > config changes. > > > > > On Fri, Dec 21, 2018 at 9:03 AM David Glasser <glas...@apollographql.com> > wrote: > > > One nice advantage to moving out of 0.x is that it frees up a digit on > the > > right side to more cleanly differentiate between "minor release (a random > > assortment of bug fixes, small features, etc)" and "patch release > > (literally the minimum delta to give you a security fix or high impact > bug > > fix)". > > > > --dave > > > > On Fri, Dec 21, 2018 at 8:58 AM Gian Merlino <g...@apache.org> wrote: > > > > > I'm not too fussy around whether we do a 1.0 or simply drop the 0. and > > have > > > it be a 14.0 or 15.0 or 16.0 or wherever we are at the time we do it. I > > > also like the quarterly cadence of release-from-master we had before we > > got > > > blocked on the ASF transition, and would like to pick that back up > again > > > (with the next branch cut from master at the end of January, since we > did > > > the 0.13.0 branch cut in late October). > > > > > > Seems to me that good points of discussion are, what should we use as > the > > > rule for incrementing the major version? Do we do what we've been doing > > > (incrementing whenever there's either an incompatible change in > extension > > > APIs, or in query APIs, or when necessary to preserve the ability to > > always > > > be able to roll forward/back one major release). Or do we do something > > else > > > (Roman seems to be suggesting dropping extension APIs from > > consideration). > > > > > > And also, what does 1.0 or 14.0 or 15.0 or what-have-you mean to us? Is > > it > > > something that should be tied to ASF graduation? Completeness of > vision? > > > Stability of APIs or operational characteristics? Something else? You > are > > > right that it is sort of a marketing/mentality thing, so it's an > > > opportunity for us to declare that we feel Druid has reached some > > > milestone. My feeling at this time is probably ASF graduation or > > > completeness of vision (see my earlier mail for thoughts there) are the > > > ones that make most sense to me. > > > > > > On Fri, Dec 21, 2018 at 10:41 AM Charles Allen <cral...@apache.org> > > wrote: > > > > > > > Is there any feeling in the community that the logic behind the > > releases > > > > needs to change? > > > > > > > > If so then I think we should discuss what that release cadence needs > to > > > > look like. > > > > > > > > If not then dropping the 0. prefix is a marketing / mental item. Kind > > of > > > > like the 3.x->4.x Linux kernel upgrade. If this is the case then > would > > we > > > > even want to go with 1.x? I think Roman's proposal would work fine in > > > this > > > > case. Where we just call it Apache Druid 14 (or 15 or whatever it is > > when > > > > we get there) and just keep the same logic for when we release stuff, > > > which > > > > has been something like: > > > > > > > > For a X.Y release, going to a X.? release should be very straight > > forward > > > > for anyone running stock Druid. > > > > For a X.Y release, going to a (X+1).? or from a (X+1).? back to an > X.Y > > > > release should be feasible. It might require running a tool supported > > by > > > > the community. > > > > For a X.Y release, going to an (X+2).? or an (X-2).? is not > supported. > > > Some > > > > things that will not have tools might have warning logs printed that > > the > > > > functionality will change (should we change these to alerts?) > > > > > > > > If this sounds reasonable then jumping straight to Apache Druid 14 on > > the > > > > first official apache release would make a lot of sense. > > > > > > > > Cheers, > > > > Charles Allen > > > > > > > > > > > > On Thu, Dec 20, 2018 at 11:07 PM Gian Merlino <g...@apache.org> > wrote: > > > > > > > > > I think it's a good point. Culturally we have been willing to break > > > > > extension APIs for relatively small benefits. But we have generally > > > been > > > > > unwilling to make breaking changes on the operations side quite so > > > > > liberally. Also, most cluster operators don't have their own custom > > > > > extensions, in my experience. So it does make sense to > differentiate > > > > them. > > > > > I'm not sure how it makes sense to differentiate them, though. It > > could > > > > be > > > > > done through the version number (only increment the major version > for > > > > > operations breaking changes) or it could be done through an > > "upgrading" > > > > > guide in the documentation (increment the major version for > > operations > > > or > > > > > extension breaking changes, but, have a guide that tells people > which > > > > > versions have operations breaking changes to aid in upgrades). > > > > > > > > > > Coming back to the question in the subject of your mail: IMO, for > > > > > "graduation" out of 0.x, we should talk as a community about what > > that > > > > > means to us. It is a milestone that on the one hand, doesn't mean > > much, > > > > but > > > > > on the other hand, can be deeply symbolic. Some things that it has > > > meant > > > > to > > > > > other projects: > > > > > > > > > > 1) Production readiness. Obviously Druid is well past this. If this > > is > > > > what > > > > > dropping the 0. means, then we should do it immediately. > > > > > > > > > > 2) Belief that the APIs have become relatively stable. Like you > said, > > > the > > > > > extension APIs don't seem particularly close to stable, but maybe > > > that's > > > > > okay. However, the pace of breaking changes on the operations and > > query > > > > > side for non-experimental features has been relatively calm for the > > > past > > > > > couple of years, so if we focus on that then we can make a case > here. > > > > > > > > > > 3) Completeness of vision. This one is the most interesting to me. > I > > > > > suspect that different people in the community have different > visions > > > for > > > > > Druid. It is also the kind of project that may never truly be > > complete > > > in > > > > > vision (in principle, the platform could become a competitive data > > > > > warehouse, search engine, etc, …). For what it's worth, my vision > of > > > > Druid > > > > > for the next year at least involves robust stream ingestion being a > > > first > > > > > class ingestion method (Kafka / Kinesis indexing service style) and > > SQL > > > > > being a first class query language. These are both, today, still > > > > > experimental features. So are lookups. All of these 3 features, > from > > > > what I > > > > > can see, are quite popular amongst Druid users despite being > > > > experimental. > > > > > For a 'completeness of vision' based 1.0 I would want to lift all > of > > > > those > > > > > out of experimental status and, for SQL in particular, to have its > > > > > functionality rounded out a bit more (to support the native query > > > > features > > > > > it doesn't currently support, like multi-value dimensions, > > > datasketches, > > > > > etc). > > > > > > > > > > 4) Marketing / timing. Like, doing a 1.0 around the time we > graduate > > > from > > > > > the Incubator. Not sure how much this really matters, but projects > do > > > it > > > > > sometimes. > > > > > > > > > > Another question is, how often do we intend to rev the version? At > > the > > > > rate > > > > > we're going, we rev 2-3 major versions a year. Would we intend to > > keep > > > > that > > > > > up, or slow it down by making more of an effort to avoid breaking > > > > changes? > > > > > > > > > > On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov < > > leventov...@gmail.com> > > > > > wrote: > > > > > > > > > > > It may also make sense to distinguish "operations" breaking > changes > > > > from > > > > > > API breaking changes. Operations breaking changes establish the > > > minimum > > > > > > cadence of Druid cluster upgrades, that allow rolling Druid > > versions > > > > back > > > > > > and forward. I. e. it's related to segment format, the format of > > the > > > > data > > > > > > kept in ZooKeeper and the SQL database, or events such as > stopping > > > > > support > > > > > > of ZooKeeper for certain things (e. g. forcing using of HTTP > > > > > > announcements). So Druid cluster operators cannot update Druid > from > > > > > version > > > > > > X to version Z skipping the version Y, if both Y and Z have some > > > > > operations > > > > > > breaking changes. (Any such changes should support rollback > options > > > at > > > > > > least until the next version with operations breaking changes.) > > > > > > > > > > > > API breaking changes are just changes in Druid extensions APIs. > > Druid > > > > > > cluster operators could skip any number of releases with such > > > breaking > > > > > > changes, as long as their extension's code is updated for the > > latest > > > > > > version of API. > > > > > > > > > > > > On Thu, 20 Dec 2018 at 20:03, Roman Leventov < > leven...@apache.org> > > > > > wrote: > > > > > > > > > > > > > It doesn't seem to me that Druid API is going to stabilize in > the > > > > near > > > > > > > future (if ever), because there are so many extension points > and > > > > > > something > > > > > > > is broken in every release. On the other hand, Druid is not > > Hadoop > > > or > > > > > > > Spark, which have applications API. Druid API for extensions, > not > > > > > > > applications. It is used by people who are closer to Druid > > > > development > > > > > > and > > > > > > > fixing their extensions is routine. > > > > > > > > > > > > > > With that, I think it make sense to drop "0." from the Druid > > > version > > > > > and > > > > > > > call it Druid 14, Druid 15, etc. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
