This is now live: https://beam.apache.org/contribute/dependencies/
Thanks everybody for comments and ideas. - Cham On Thu, Jun 21, 2018 at 5:18 PM Chamikara Jayalath <[email protected]> wrote: > https://github.com/apache/beam-site/pull/475 adds a dependencies guide to > Website based on the discussions here and in the doc. > > Based on a recent discussion in the doc, I added a section on backwards > compatibility. Basically, when upgrading dependencies, PR authors and > reviewers should take care not to break backwards compatibility guarantee > of Beam (till next major version) and authors of PRs that introduce > dependency upgrades should include a statement regarding this verification > in the form of a PR comment. I think this will be a useful addition since > we are committed to semantic versioning. > > Any comments are welcome. > > Thanks, > Cham > > On Tue, Jun 12, 2018 at 8:03 AM Chamikara Jayalath <[email protected]> > wrote: > >> Sounds good. I think we can add this under Technical Docs here >> <https://beam.apache.org/contribute/>. >> >> Thanks, >> Cham >> >> On Tue, Jun 12, 2018 at 5:35 AM Kenneth Knowles <[email protected]> wrote: >> >>> Makes sense. Then finding a good home on the web site is the way to go. >>> >>> Kenn >>> >>> On Mon, Jun 11, 2018 at 10:35 PM Bashir Sadjad <[email protected]> >>> wrote: >>> >>>> FWIW, I also think that this has relevance for users. I am a user of >>>> Beam not a contributor and only monitor this list at a high level. But I >>>> think the dependency issue is something that many users have to deal with. >>>> It has bitten us at least twice over the last few months due to the fact >>>> that we depend on other libraries too and sometimes we get version >>>> conflicts (which is one of the issues highlighted in the doc >>>> <https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit> >>>> Cham shared). I usually go through file histories on GitHub to try to >>>> figure out why a certain version requirement is there. It would be nice if >>>> the reasons are maintained at a higher level easier to consume by users. >>>> >>>> Cheers >>>> >>>> -B >>>> >>>> On Tue, Jun 12, 2018 at 12:19 AM Ahmet Altay <[email protected]> wrote: >>>> >>>>> I think this is relevant for users. It makes sense for users to know >>>>> about how Beam work with its dependencies and understand how conflicts >>>>> will >>>>> be addressed and when dependencies will be upgraded. >>>>> >>>>> On Mon, Jun 11, 2018 at 9:09 PM, Kenneth Knowles <[email protected]> >>>>> wrote: >>>>> >>>>>> Do you think this has relevance for users? >>>>>> >>>>>> If not, it might be a good use of the new Confluence space. I'm not >>>>>> too familiar with the way permission work, but perhaps we can have a more >>>>>> locked down area that is for policy decisions like this. >>>>>> >>>>>> Kenn >>>>>> >>>>>> On Mon, Jun 11, 2018 at 3:58 PM Chamikara Jayalath < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> Based on the vote (3 PMC +1s and no -1s) and based on the >>>>>>> discussions in the doc (seems to be mostly positive), I think we can go >>>>>>> ahead and implement some of the policies discussed so far. >>>>>>> >>>>>>> I have given some of the potential action items below. >>>>>>> >>>>>>> * Automatically generate human readable reports on status of Beam >>>>>>> dependencies weekly and share in dev list. >>>>>>> * Create JIRAs for significantly outdated dependencies based on >>>>>>> above reports. >>>>>>> * Copy some of the component level dependency version declarations >>>>>>> to top level. >>>>>>> * Try to identify owners for dependencies and specify owners in >>>>>>> comments close to dependency declarations. >>>>>>> * Vendor any dependencies that can cause issues if leaked to other >>>>>>> components. >>>>>>> * Add policies discussed so far to the Web site along with reasoning >>>>>>> (from doc). >>>>>>> >>>>>>> Of course, I'm happy to refine or add to these polices as needed. >>>>>>> >>>>>>> Thanks, >>>>>>> Cham >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 7, 2018 at 9:40 AM Lukasz Cwik <[email protected]> wrote: >>>>>>> >>>>>>>> +1 >>>>>>>> >>>>>>>> On Thu, Jun 7, 2018 at 5:18 AM Kenneth Knowles <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> +1 to these. Thanks for clarifying! >>>>>>>>> >>>>>>>>> Kenn >>>>>>>>> >>>>>>>>> On Wed, Jun 6, 2018 at 10:40 PM Chamikara Jayalath < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Kenn, >>>>>>>>>> >>>>>>>>>> On Wed, Jun 6, 2018 at 8:14 PM Kenneth Knowles <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> +0.5 >>>>>>>>>>> >>>>>>>>>>> I like the spirit of these policies. I think they need a little >>>>>>>>>>> wording work. Comments inline. >>>>>>>>>>> >>>>>>>>>>> On Wed, Jun 6, 2018 at 4:53 PM, Chamikara Jayalath < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> (1) Human readable reports on status of Beam dependencies are >>>>>>>>>>>>> generated weekly and shared with the Beam community through the >>>>>>>>>>>>> dev list. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> Who is responsible for generating these? The mechanism or >>>>>>>>>>> responsibility should be made clear. >>>>>>>>>>> >>>>>>>>>>> I clicked through a doc -> thread -> doc to find even some >>>>>>>>>>> details. It looks like manual run of a gradle command was adopted. >>>>>>>>>>> So the >>>>>>>>>>> responsibility needs an owner, even if it is "unspecified volunteer >>>>>>>>>>> on dev@ >>>>>>>>>>> and feel free to complain or do it yourself if you don't see it" >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This is described in following doc (referenced by my doc). >>>>>>>>>> >>>>>>>>>> https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit# >>>>>>>>>> >>>>>>>>>> Proposal is to run an automated Jenkins job that is run weekly, >>>>>>>>>> so no need for someone to manually generate these reports. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> (2) Beam components should define dependencies and their >>>>>>>>>>>>> versions at the top level. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> I think the big "should" works better with some guidance about >>>>>>>>>>> when something might be an exception, or at least explicit mention >>>>>>>>>>> that >>>>>>>>>>> there can be rare exceptions. Unless you think that is never the >>>>>>>>>>> case. If >>>>>>>>>>> there are no exceptions, then say "must" and if we hit a roadblock >>>>>>>>>>> we can >>>>>>>>>>> revisit the policy. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The idea was to allow exceptions. Added more details to the doc. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> (3) A significantly outdated dependency (identified manually or >>>>>>>>>>>>> through tooling) should result in a JIRA that is a blocker for >>>>>>>>>>>>> the next >>>>>>>>>>>>> release. Release manager may choose to push the blocker to the >>>>>>>>>>>>> subsequent >>>>>>>>>>>>> release or downgrade from a blocker. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> How is "significantly outdated" defined? By dev@ discussion? >>>>>>>>>>> Seems like the right way. Anyhow that's what will happen in >>>>>>>>>>> practice as >>>>>>>>>>> people debate the blocker bug. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> This will be either through the automated Jenkins job (see the >>>>>>>>>> doc above, where the proposal is to flag new major versions and new >>>>>>>>>> minor >>>>>>>>>> versions that are more than six months old) or manually (for any >>>>>>>>>> critical >>>>>>>>>> updates that will not be captured by the Jenkins job) (more details >>>>>>>>>> in the >>>>>>>>>> doc). Manually identified critical dependency updates may involve a >>>>>>>>>> discussion in the dev list. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> (4) Dependency declarations may identify owners that are >>>>>>>>>>>>> responsible for upgrading the respective dependencies. >>>>>>>>>>>>> >>>>>>>>>>>>> (5) Dependencies of Java SDK components that may cause issues >>>>>>>>>>>>> to other components if leaked should be shaded. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> We previously agreed upon our intent to migrate to "pre-shaded" >>>>>>>>>>> aka "vendored" packages: >>>>>>>>>>> https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E >>>>>>>>>>> >>>>>>>>>>> With Maven, this involved a lot of boilerplate so I never did >>>>>>>>>>> it. With Gradle, we can easily build a re-usable rule to create >>>>>>>>>>> such a >>>>>>>>>>> package in a couple of lines. I just opened the first WIP PR here: >>>>>>>>>>> https://github.com/apache/beam/pull/5570 it is blocked by >>>>>>>>>>> deleting the poms anyhow so by then we should have a configuration >>>>>>>>>>> that >>>>>>>>>>> works to vendor our currently shaded artifacts. >>>>>>>>>>> >>>>>>>>>>> So I think this should be rephrased to "should be vendored" so >>>>>>>>>>> we don't have to revise the policy. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks for the pointer. I agree that vendoring is a good approach. >>>>>>>>>> >>>>>>>>>> Here are the updated policies (and more details added to doc). I >>>>>>>>>> agree with Ahmet's point that votes should be converted to web sites >>>>>>>>>> where >>>>>>>>>> we can give more details and examples. >>>>>>>>>> >>>>>>>>>> (1.a) Human readable reports on status of Beam dependencies are >>>>>>>>>> generated weekly by an automated Jenkins job and shared with the Beam >>>>>>>>>> community through the dev list. >>>>>>>>>> >>>>>>>>>> (2.a) Beam components should define dependencies and their >>>>>>>>>> versions at the top level. There can be rare exceptions, but they >>>>>>>>>> should >>>>>>>>>> come with explanations. >>>>>>>>>> >>>>>>>>>> (3.a) A significantly outdated dependency (identified manually >>>>>>>>>> or through the automated Jenkins job) should result in a JIRA that >>>>>>>>>> is a >>>>>>>>>> blocker for the next release. Release manager may choose to push the >>>>>>>>>> blocker to the subsequent release or downgrade from a blocker. >>>>>>>>>> >>>>>>>>>> (4.a) Dependency declarations may identify owners that are >>>>>>>>>> responsible for upgrading the respective dependencies. >>>>>>>>>> >>>>>>>>>> (5.a) Dependencies of Java SDK components that may cause issues >>>>>>>>>> to other components if leaked should be vendored. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Cham >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Kenn >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Please vote: >>>>>>>>>>>>> [ ] +1, Approve that we adapt these policies >>>>>>>>>>>>> [ ] -1, Do not approve (please provide specific comments) >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Cham >>>>>>>>>>>>> >>>>>>>>>>>>> [1] >>>>>>>>>>>>> https://lists.apache.org/thread.html/8738c13ad7e576bc2fef158d2cc6f809e1c238ab8d5164c78484bf54@%3Cdev.beam.apache.org%3E >>>>>>>>>>>>> [2] >>>>>>>>>>>>> https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit?usp=sharing >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>
