Thanks everybody for all the comments in the doc. Based on the reaction so far, seems like community generally agrees to introducing policies around managing dependencies. I updated the suggested policies based on comments and rewrote them in a more concise form and added room for more human intervention when needed. Updated policies are given below (and mentioned in bold in the document). Please see the document for details.
(1) Human readable reports on status of Beam dependencies are generated weekly and shared with the Beam community through the dev list. (2) Beam components should always have dependencies and their versions defined at the top level. (3) A significantly outdated dependency (identified manually or through tooling) should result in a JIRA that is a blocker for the next release. Release manager may choose to push the blocker to the subsequent release or downgrade from a blocker. (4) Dependency declarations may identify owners that are responsible for upgrading the respective dependencies. (5) Dependencies of Java SDK components that may cause issues to other components if leaked should be shaded. Thanks, Cham On Fri, Jun 1, 2018 at 2:44 PM Chamikara Jayalath <chamik...@google.com> wrote: > Thanks. I left these ideas bit detailed for clarification. I'll rewrite > them in a more concise form when we have enough feedback. > > - Cham > > On Fri, Jun 1, 2018 at 1:58 PM Kenneth Knowles <k...@google.com> wrote: > >> This does seem really useful. I appreciate the detailed explanations. If >> we formalize it into policy, I'd love to make it a bit more concise, and >> with appropriate room for human contestation of the guidelines. >> >> On Fri, Jun 1, 2018 at 1:47 PM Scott Wegner <sweg...@google.com> wrote: >> >>> Thanks Cham. Overall this seems like a useful hygiene improvement for >>> the project. I've left some comments in the doc. >>> >>> On Fri, Jun 1, 2018 at 10:48 AM Chamikara Jayalath <chamik...@google.com> >>> wrote: >>> >>>> Hi All, >>>> >>>> I've copied ideas proposed in the doc below for more visibility. Any >>>> comments are welcome. >>>> >>>> >>>> >>>> * - Human readable per-SDK reports on status of Beam dependencies are >>>> generated weekly and shared with the Beam community through the dev list. >>>> These reports should be concise and should highlight the cases where the >>>> community has to act on. See [4] for more details on this.- Beam Components >>>> (IO connectors, runners, etc) should always try to use versions of >>>> dependencies that are defined at the top level. Per-component dependency >>>> version overrides should only be performed in rare cases and should come >>>> with clear warnings for users.- Upgrading a dependency with an outdated >>>> major version becomes a blocker for next major version release of Beam and >>>> for any minor version releases after next immediate minor version release. >>>> For example, if a dependency is identified to be outdated while the latest >>>> release is x.y.z, upgrading this dependency becomes a blocker for releases >>>> (x+1).0.0 and x.(y+2).0 of Beam. Additionally, upgrading to a major version >>>> of a dependency will only be enforced if the new major version of the >>>> dependency can be adapted without a significant rewrite to any Beam >>>> component. Note that this policy intentionally allows one of the minor >>>> version releases to proceed without upgrading the dependency which I >>>> believe will give Beam community enough breathing room to upgrade >>>> dependencies without significantly affecting the release frequency.- >>>> Upgrading a dependency with a significantly outdated minor version (based >>>> on methodology defined in [4]) becomes a blocker for next major version >>>> release of Beam and for any minor version releases of Beam after next >>>> immediate minor version release. Note that this policy does not force Beam >>>> to adapt every minor version release of a dependency.- When performing a >>>> release, release manager should make sure that blockers identified through >>>> above process are resolved before the release candidate is cut.- >>>> Optionally, dependency declarations may have comments that identify owners >>>> that should be responsible for upgrading the respective dependencies. >>>> Release manager may choose to assign a blocking JIRA for a dependency to >>>> its owner.* >>>> Thanks, >>>> Cham >>>> >>>> On Thu, May 31, 2018 at 7:11 PM Chamikara Jayalath < >>>> chamik...@google.com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> We recently ran into many issues due to Beam dependencies being >>>>> significantly out of date. For example see [1], [2], and [3]. >>>>> >>>>> Yifan Zou recently introduced a proposal [4] that would allow us to >>>>> identify outdated dependencies. But to really make sure that this helps >>>>> the >>>>> Beam project and community I believe we should adapt several small policy >>>>> changes to our development and release process. >>>>> >>>>> To this end, I have created following short document that identifies >>>>> the dependency issue and proposes several policy changes. I greatly >>>>> appreciate if you can take a look and comment. >>>>> >>>>> >>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing >>>>> >>>>> Thanks, >>>>> Cham >>>>> >>>>> [1] https://issues.apache.org/jira/browse/BEAM-3098 >>>>> [2] https://issues.apache.org/jira/browse/BEAM-3991 >>>>> [3] https://issues.apache.org/jira/browse/BEAM-4229 >>>>> [4] >>>>> https://lists.apache.org/thread.html/758625106a6cfe9ba23d7b39625da20e050c6279b138b18b3f0013e7@%3Cdev.beam.apache.org%3E >>>>> >>>>
