On Thu, Oct 4, 2018 at 9:39 AM Ahmet Altay <al...@google.com> wrote: > I agree that LTS releases require more thought. Thank you for raising > these questions. What other open questions do we have related LTS releases? > > One way to do this would be to add them to a particular tracking list > (e.g. 2.9.0 blocking list) that way we would be ready for an LTS release > ahead of time. > > Related to dependencies, I agree with Thomas. If we block on waiting for > dependencies, we may end up taking a long time before making any LTS > release. And the reality of Beam releases right now is that there are no > supported releases today that and in the long term that might hurt user > trust. In my opinion, we need to fix that sooner rather than later. >
Agree on the idea of focussing on stability instead of feature set when it comes to LTS releases. Based on the previous discussion on this [1] looks like the intended audience is enterprise customers that would like to maintain a stable environment and that usually have a long testing cycle before deploying a new version. Seems like, rushing in features or dependency upgrades for an LTS release will be counterintuitive for such use cases. On the other hand, I think a part of Ismaël point was that all major features of Beam (Spark Runner, Flink Runner, Kafka IO, etc ..) should be supportable for the duration of the time Beam LTS release is supported (1 year based on [1]). If a major dependency is expected to become unsupported within an year, it makes sense to upgrade that before the LTS release. I suggest we do this at least one release before the planned LTS release (so 2.9.0 if we want to make 2.10.0 a LTS release) so that we have all major dependency/feature updates battle tested before committing to them. Thanks, Cham [1] https://lists.apache.org/thread.html/b2e2e79e0e146ed5264a12831d9e40cabfcb75651a994484c58a6b01@%3Cdev.beam.apache.org%3E > > Ahmet > > > On Thu, Oct 4, 2018 at 9:27 AM, Ahmet Altay <al...@google.com> wrote: > >> Thank you all for the feedback. I will continue with 2.8.0 as a regular >> release and separate the LTS discussion to a new thread. >> >> On Thu, Oct 4, 2018 at 7:58 AM, Thomas Weise <t...@apache.org> wrote: >> >>> Given the feedback so far, we should probably decouple LTS and 2.8.0 >>> discussions. In case both converge before 10/10 then fine, but not >>> necessary. I also agree that we should not jump the gun on LTS and minimum >>> 72 hours feedback window for the topic looks appropriate. >>> >>> The issues raised by Tim look like blockers and unless we are confident >>> that they can be addressed as a patch release may warrant to defer LTS? Can >>> we start to tag such JIRAs with an LTS label? >>> >>> On the other hand, I think we could allow for a bit of experimentation >>> error for the first LTS attempt and feed guidelines/policies from >>> learnings/feedback. >>> >>> Dependency updates for LTS: I don't think we should block LTS because >>> there is a newer version of a dependency out there or we should rush >>> updates. If we prioritize stability, then the latest usually isn't the >>> best. In the case of Flink, 1.5.x is probably what most users have at this >>> time and it has seen 4 patch releases. If Flink community continues to >>> support last two minor (X.Y) versions, then 1.5.x support may drop when 1.7 >>> comes out, but that does not mean we cannot use it if we were to cut a Beam >>> LTS release today. I generally think that LTS needs to focus more on the >>> stability of Beam itself. >>> >>> Thanks, >>> Thomas >>> >>> >>> >>> On Thu, Oct 4, 2018 at 6:59 AM Alexey Romanenko < >>> aromanenko....@gmail.com> wrote: >>> >>>> Regarding LTS release - I agree that we need to have clear view what >>>> kind of support will be provided for such releases. >>>> >>>> Despite of the concerns mentioned before, I have another one about API >>>> labeled as “@Experimental". I think there are most of IOs, SQL, PCollection >>>> with Schema, etc, labeled with this annotation. >>>> According to definition, such API should be considered as unstable in >>>> terms that it can be changed/removed in next releases. >>>> >>>> So, the question is - how “@Experimental” API affects LTS releases (if >>>> it does)? What kind of support should be provided in this case, especially, >>>> in case if API continued evolving after LTS has been issued? Do we need to >>>> provide a guarantee (another annotation, for example) that API won’t be >>>> changed between two LTS releases? >>>> >>>> And one more related question, which probably deserves another >>>> discussion (or was already discussed) - what is criteria to remove >>>> status “@Experimental” from API? How we decide that API is stable and not >>>> changing anymore? >>>> >>>> >>>> On 4 Oct 2018, at 12:35, Robert Bradshaw <rober...@google.com> wrote: >>>> >>>> +1 to cutting the release. >>>> >>>> I agree that the LTS label requires more discussion. I think it boils >>>> down to the question of whether we are comfortable with encouraging people >>>> to not upgrade to the latest Beam. It probably boils down to creating a >>>> list of (potential) blockers and then going from there. Also, on this note, >>>> I think we should be very conservative in updating dependencies for an LTS >>>> release. >>>> >>>> We could also consider for this release doing an "LTS light" where we >>>> prove the process, gain some experience, but don't promise a full 12 months >>>> of support (say, cutting it to 6 months). >>>> >>>> - Robert >>>> >>>> >>>> >>>> >>>> On Thu, Oct 4, 2018 at 11:25 AM Tim Robertson < >>>> timrobertson...@gmail.com> wrote: >>>> >>>>> I was in the middle of writing something similar when Ismaël posted. >>>>> >>>>> Please do bear in mind that this is an international project and 7hrs >>>>> is not long enough to decide upon something that affects us all. >>>>> >>>>> +1 on cutting 2.8.0 on 10/10 and thank you for pushing it forward >>>>> >>>>> -1 on designating it as LTS: >>>>> While LTS is a statement of expectation in maintenance it also carries >>>>> an element of trust. I propose we should have a separate discussion about >>>>> what we might like to collectively achieve before announcing our first LTS >>>>> edition. >>>>> My concern stems from usability and first impressions - for example: >>>>> - Beam has real issues with HDFS today (BEAM-5036) which I propose as >>>>> blocker for announcing LTS >>>>> - DirectRunner and the inability to run basic pipelines on a few GB of >>>>> data is *really* putting people off our project - we might consider >>>>> exploring that as it affects our "brand" >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Oct 4, 2018 at 11:18 AM Ismaël Mejía <ieme...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> Thanks Ahmet for volunteering to do the release, and proposing this >>>>>> as an LTS. >>>>>> >>>>>> I have still some questions on our LTS policies (which may have >>>>>> consequences on the discussed release): >>>>>> >>>>>> What are the expected implications of upgrades in the LTS, e.g. If a >>>>>> connector let’s say Kafka is released using the 1.0 dependency, can it >>>>>> be moved upwards in a LTS to version 2.0 or this will be considered a >>>>>> breaking change and we should only move in minor versions. Will this >>>>>> rule be more relaxed for example for all cloud based dependencies >>>>>> (GCP, AWS) for example if a security issue or correctness/performance >>>>>> improvement? >>>>>> >>>>>> Given that this will last for a year maybe we should raise some of the >>>>>> dependencies to the latest versions. Following the recent discussion >>>>>> on dependencies that cannot be ‘automatically’ updated because of end >>>>>> user consequences, I still think about what we should do with >>>>>> (probably related to the previous paragraph): >>>>>> >>>>>> - Should we move Flink then to 1.6.x considering that 1.5.x won’t be >>>>>> maintained in less than 6 months. >>>>>> - Should we wait and upgrade Spark into version 2.4.0 (which is being >>>>>> voted at this moment but not released but could make sense for a LTS) >>>>>> or just stay in 2.3.x. Spark is less of an issue because it is a >>>>>> provided dep but still worth. >>>>>> - Should we update the IO connectors dependencies to the latest stable >>>>>> versions who aren’t, e.g. Elasticsearch, HBase, >>>>>> >>>>>> Of course the goal is not a last minute rush to do this so it fits in >>>>>> the LTS release, but to see that for LTS we may consider the ‘lasting >>>>>> consequences'. >>>>>> >>>>>> One last comment, next time we discuss a proposal please ensure that >>>>>> we wait at least 24h to reach conclusions or proceed, otherwise this >>>>>> will exclude opinions from people who are not in the right time zone >>>>>> (this is the reason why votes last 72h to ensure that everyone may be >>>>>> aware of what is been voted). This is not a mandatory requirement, but >>>>>> agreeing on a LTS in 7h seems a bit short. >>>>>> On Thu, Oct 4, 2018 at 1:36 AM Ahmet Altay <al...@google.com> wrote: >>>>>> > >>>>>> > Great. I will do the cut on 10/10. >>>>>> > >>>>>> > Let's start by triaging the open issues targeted for 2.8.0 [1]. If >>>>>> you have any issues in this list please resolve them or move to the next >>>>>> release. If you are aware of any critical issues please add to this list. >>>>>> > >>>>>> > Ahmet >>>>>> > >>>>>> > [1] >>>>>> https://issues.apache.org/jira/browse/BEAM-5456?jql=project%20%3D%20BEAM%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%202.8.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC >>>>>> > >>>>>> > > +1 for the 2.7.0 release schedule. Thanks for volunteering. Do we >>>>>> want a standing owner for the LTS branch (like the Linux kernel has) or >>>>>> will we just take volunteers for each LTS release as they arise? >>>>>> > >>>>>> > We have not thought about this before. IMO, it is better to keep >>>>>> things simple and use the same process (i.e. "we just take volunteers for >>>>>> each LTS release as they arise") for patch releases in the future if/when >>>>>> we happen to need those. >>>>>> > >>>>>> > >>>>>> > On Wed, Oct 3, 2018 at 1:21 PM, Thomas Weise <t...@apache.org> >>>>>> wrote: >>>>>> >> >>>>>> >> +1 >>>>>> >> >>>>>> >> On Wed, Oct 3, 2018 at 12:33 PM Ted Yu <yuzhih...@gmail.com> >>>>>> wrote: >>>>>> >>> >>>>>> >>> +1 >>>>>> >>> >>>>>> >>> On Wed, Oct 3, 2018 at 9:52 AM Jean-Baptiste Onofré < >>>>>> j...@nanthrax.net> wrote: >>>>>> >>>> >>>>>> >>>> +1 >>>>>> >>>> >>>>>> >>>> but we have to be fast in release process. 2.7.0 took more than >>>>>> 1 month >>>>>> >>>> to be cut ! >>>>>> >>>> >>>>>> >>>> If no blocker, we have to just move forward. >>>>>> > >>>>>> > >>>>>> > +1 >>>>>> > >>>>>> >>>> >>>>>> >>>> >>>>>> >>>> Regards >>>>>> >>>> JB >>>>>> >>>> >>>>>> >>>> On 03/10/2018 18:25, Ahmet Altay wrote: >>>>>> >>>> > Hi all, >>>>>> >>>> > >>>>>> >>>> > Release cut date for the next release is 10/10 according to >>>>>> Beam release >>>>>> >>>> > calendar [1]. Since the previous release is already mostly >>>>>> wrapped up >>>>>> >>>> > (modulo blog post), I would like to propose starting the next >>>>>> release on >>>>>> >>>> > time (10/10). >>>>>> >>>> > >>>>>> >>>> > Additionally I propose designating this release as the first >>>>>> >>>> > long-term-support (LTS) release [2]. This should have no >>>>>> impact on the >>>>>> >>>> > release process, however it would mean that we commit to patch >>>>>> this >>>>>> >>>> > release for the next 12 months for major issues. >>>>>> >>>> > >>>>>> >>>> > I volunteer to perform this release. >>>>>> >>>> > >>>>>> >>>> > What do you think? >>>>>> >>>> > >>>>>> >>>> > Ahmet >>>>>> >>>> > >>>>>> >>>> > [1] >>>>>> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com&ctz=America%2FLos_Angeles >>>>>> >>>> > [2] https://beam.apache.org/community/policies/#releases >>>>>> >>>> >>>>>> >>>> -- >>>>>> >>>> Jean-Baptiste Onofré >>>>>> >>>> jbono...@apache.org >>>>>> >>>> http://blog.nanthrax.net >>>>>> >>>> Talend - http://www.talend.com >>>>>> > >>>>>> > >>>>>> >>>>> >>>> >> >
