Thanks everyone for feedbacks! We will embed some text to explain the details of the report and guide people what to do with it at this point. Cham and I prepare to start updating those dependencies, and have them grouped and find them owners if possible. Also, we will try to automate JIRA filing and assign them to owners to review/upgrade dependencies.
Regards. Yifan Zou On Wed, Jun 13, 2018 at 10:45 AM Chamikara Jayalath <chamik...@google.com> wrote: > > Thanks Yifan. > > On Wed, Jun 13, 2018 at 10:21 AM Ahmet Altay <al...@google.com> wrote: > >> Thanks Yifan, this is great! >> >> My unsolicited feedback: >> - Could it warn against dependencies that did not get updates for a long >> time? For python there were examples of a dependency being abandoned by its >> own developers and it took us a while to figure it out and switch to >> maintained one. (Currently googledatastore is a dependency like that.) >> >> I will second the Scott's question for what should I do with this report? >> Is it possible to add a link to quickly create a JIRA issue for a given >> dependency? Or is it possible to link to already open issues for the >> identified dependencies? >> > > We ultimately want to create JIRAs automatically and assign them to owners > if defined according to the policy document. We can also add a link to the > JIRA to the report. But given the number of outdated dependencies listed > here, looks like we'll have to bootstrap the process by manually updating > many of these dependencies. > > - Cham > > >> Ahmet >> >> On Wed, Jun 13, 2018 at 10:13 AM, Scott Wegner <sweg...@google.com> >> wrote: >> >>> Nifty. Here's some unsolicited feedback: >>> >>> * The report gives a nice view of the data and leaves it as an exercise >>> to the reader to do the math on each row (v0.25.0 to v1.3.0 = 1 major >>> version behind, 2017-06-26 to 2018-06-08 = 1 year behind). I would find the >>> report more digestable if these details were already included. >>> * The next question after reading this report is "What should I do with >>> this?" I recommend embedding details or links to answer that. For example: >>> >>> "High Priority Dependency Updates are defined as XYZ. In Beam, we make >>> a best-effort attempt at keeping all dependencies up-to-date according to >>> ABC. In the future, issues will be filed and tracked for these >>> automatically, but in the meantime you can search for existing issues or >>> open a new one <here>. Read more about our dependency update policy <here>." >>> >>> On Wed, Jun 13, 2018 at 9:02 AM Pablo Estrada <pabl...@google.com> >>> wrote: >>> >>>> Ahh very nice... thanks Yifan & Cham! >>>> Lots of old dependencies eh... very interesting. >>>> Best >>>> -P. >>>> >>>> On Wed, Jun 13, 2018 at 7:45 AM Yifan Zou <yifan...@google.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> >>>>> I want to follow up and explain this email. >>>>> >>>>> >>>>> This is a sample email that reports the results of Beam SDK dependency >>>>> check, which was proposed here >>>>> <https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#heading=h.u75g8bk11ngp>. >>>>> The goal is finding updates for all Beam Python & Java SDKs' dependencies >>>>> and prioritize them. The job will be auto triggered in Jenkins once a week >>>>> and generate a report. The report lists the high priority updates base on >>>>> the following criteria: >>>>> >>>>> >>>>> The dependency update is high priority if: >>>>> >>>>> 1. It has major versions update available; >>>>> >>>>> e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0 >>>>> >>>>> 2. or, it is over 3 minor versions behind the latest version; >>>>> >>>>> e.g. org.tukaani:xz 1.5 -> 1.8 >>>>> >>>>> 3. or, the current version is behind the later version for over 180 >>>>> days. >>>>> >>>>> e.g. com.google.auto.service:auto-service 2014-10-24 -> >>>>> 2017-12-11 >>>>> >>>>> >>>>> This job helps Beam contributors to determine the dependency which is >>>>> far behind the latest released version. The next step would be automating >>>>> filing JIRA bugs for dep updates, group dependencies and identify owners >>>>> to >>>>> take care of the upgrades follow Chamikara's proposal >>>>> <https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit> >>>>> . >>>>> >>>>> >>>>> For more readings: >>>>> >>>>> [Proposal] Beam dependency check automation >>>>> <https://docs.google.com/document/d/1rqr_8a9NYZCgeiXpTIwWLCL7X8amPAVfRXsO72BpBwA/edit#heading=h.u75g8bk11ngp> >>>>> by Yifan Zou >>>>> >>>>> [Proposal] Beam dependency update policy >>>>> <https://docs.google.com/document/d/15m1MziZ5TNd9rh_XN0YYBJfYkt0Oj-Ou9g0KFDPL2aA/edit> >>>>> by *Chamikara Jayalath* >>>>> >>>>> Thank you. >>>>> >>>>> Yifan Zou >>>>> >>>>> On Wed, Jun 13, 2018 at 7:41 AM Apache Jenkins Server < >>>>> jenk...@builds.apache.org> wrote: >>>>> >>>>>> High Priority Dependency Updates Of Beam Python SDK: >>>>>> *Dependency Name* *Current Version* *Later Version* *Current Version >>>>>> Release Date* *Later Version Release Date* >>>>>> google-cloud-bigquery 0.25.0 1.3.0 2017-06-26 2018-06-08 >>>>>> httplib2 0.9.2 0.11.3 2015-09-28 2018-03-30 High Priority Dependency >>>>>> Updates Of Beam Java SDK: >>>>>> *Dependency Name* *Current Version* *Later Version* *Current Version >>>>>> Release Date* *Later Version Release Date* >>>>>> org.assertj:assertj-core 2.5.0 3.10.0 2016-07-03 2018-05-11 >>>>>> com.google.auto.service:auto-service 1.0-rc2 1.0-rc4 2014-10-24 >>>>>> 2017-12-11 >>>>>> biz.aQute:bndlib 1.43.0 2.0.0.20130123-133441 2011-04-01 2013-02-27 >>>>>> org.apache.cassandra:cassandra-all 3.9 3.11.2 2016-09-26 2018-02-14 >>>>>> commons-cli:commons-cli 1.2 1.4 2009-03-19 2017-03-09 >>>>>> commons-codec:commons-codec 1.9 1.11 2013-12-20 2017-10-17 >>>>>> org.apache.commons:commons-dbcp2 2.1.1 2.3.0 2015-08-02 2018-05-08 >>>>>> com.typesafe:config 1.3.0 1.3.3 2015-05-08 2018-02-21 >>>>>> de.flapdoodle.embed:de.flapdoodle.embed.mongo 1.50.1 2.0.3 2015-12-11 >>>>>> 2018-02-14 >>>>>> de.flapdoodle.embed:de.flapdoodle.embed.process 1.50.1 2.0.3 >>>>>> 2015-12-11 2018-02-14 >>>>>> org.apache.derby:derby 10.12.1.1 10.14.2.0 2015-10-10 2018-05-03 >>>>>> org.apache.derby:derbyclient 10.12.1.1 10.14.2.0 2015-10-10 >>>>>> 2018-05-03 >>>>>> org.apache.derby:derbynet 10.12.1.1 10.14.2.0 2015-10-10 2018-05-03 >>>>>> org.elasticsearch:elasticsearch 5.6.3 6.2.4 2017-10-06 2018-04-12 >>>>>> org.elasticsearch:elasticsearch-hadoop 5.0.0 6.2.4 2016-10-26 >>>>>> 2018-04-12 >>>>>> org.elasticsearch.client:elasticsearch-rest-client 5.6.3 6.2.4 >>>>>> 2017-10-06 2018-04-12 >>>>>> com.alibaba:fastjson 1.2.12 1.2.47 2016-05-21 2018-03-15 >>>>>> org.elasticsearch.test:framework 5.6.3 6.2.4 2017-10-06 2018-04-12 >>>>>> org.freemarker:freemarker 2.3.25-incubating 2.3.28 2016-06-14 >>>>>> 2018-03-30 >>>>>> org.codehaus.groovy:groovy-all 2.4.13 3.0.0-alpha-2 2017-11-22 >>>>>> 2018-04-16 >>>>>> org.apache.hbase:hbase-common 1.2.6 2.0.0.3.0.0.3-2 2017-05-29 >>>>>> 2018-05-31 >>>>>> org.apache.hbase:hbase-hadoop-compat 1.2.6 2.0.0.3.0.0.3-2 2017-05-29 >>>>>> 2018-05-31 >>>>>> org.apache.hbase:hbase-hadoop2-compat 1.2.6 2.0.0.3.0.0.3-2 >>>>>> 2017-05-29 2018-05-31 >>>>>> org.apache.hbase:hbase-server 1.2.6 2.0.0.3.0.0.3-2 2017-05-29 >>>>>> 2018-05-31 >>>>>> org.apache.hbase:hbase-shaded-client 1.2.6 2.0.0.3.0.0.3-2 2017-05-29 >>>>>> 2018-05-31 >>>>>> org.apache.hbase:hbase-shaded-server 1.2.6 2.0.0-alpha2 2017-05-29 >>>>>> 2018-05-31 >>>>>> org.apache.hive:hive-cli 2.1.0 3.0.0.3.0.0.3-2 2016-06-16 2018-05-21 >>>>>> org.apache.hive:hive-common 2.1.0 3.0.0.3.0.0.3-2 2016-06-16 >>>>>> 2018-05-21 >>>>>> org.apache.hive:hive-exec 2.1.0 3.0.0.3.0.0.3-2 2016-06-16 2018-05-21 >>>>>> org.apache.hive.hcatalog:hive-hcatalog-core 2.1.0 3.0.0.3.0.0.3-2 >>>>>> 2016-06-16 2018-05-21 >>>>>> org.apache.httpcomponents:httpasyncclient 4.1.2 4.1.3 2016-06-18 >>>>>> 2017-02-05 >>>>>> org.apache.httpcomponents:httpclient 4.5.2 4.5.5 2016-02-21 >>>>>> 2018-01-18 >>>>>> org.apache.httpcomponents:httpcore 4.4.5 4.4.9 2016-06-08 2018-01-11 >>>>>> net.java.dev.javacc:javacc 4.0 7.0.3 2018-06-08 2017-11-06 >>>>>> jline:jline 2.14.6 3.0.0.M1 2018-03-26 2018-06-08 >>>>>> net.java.dev.jna:jna 4.1.0 4.5.1 2014-03-06 2017-12-27 >>>>>> com.esotericsoftware.kryo:kryo 2.21 2.24.0 2013-02-27 2014-05-04 >>>>>> io.dropwizard.metrics:metrics-core 3.1.2 4.1.0-rc2 2015-04-25 >>>>>> 2018-05-03 >>>>>> org.mongodb:mongo-java-driver 3.2.2 3.8.0-beta3 2016-02-15 2018-05-29 >>>>>> io.netty:netty-all 4.1.17.Final 5.0.0.Alpha2 2017-11-08 2018-06-06 >>>>>> io.grpc:protoc-gen-grpc-java 1.2.0 1.12.0 2017-03-15 2018-05-07 >>>>>> org.apache.qpid:proton-j 0.13.1 0.27.1 2016-07-01 2018-04-25 >>>>>> com.carrotsearch.randomizedtesting:randomizedtesting-runner 2.5.0 >>>>>> 2.6.3 2017-01-23 2018-06-11 >>>>>> org.scala-lang:scala-library 2.11.8 2.13.0-M4 2017-03-08 2018-05-14 >>>>>> org.slf4j:slf4j-api 1.7.25 1.8.0-beta2 2017-03-16 2018-03-21 >>>>>> org.slf4j:slf4j-jdk14 1.7.25 1.8.0-beta2 2017-03-16 2018-03-21 >>>>>> org.apache.solr:solr-core 5.5.4 7.3.1 2017-10-20 2018-05-17 >>>>>> org.apache.solr:solr-solrj 5.5.4 7.3.1 2017-10-20 2018-05-17 >>>>>> org.apache.solr:solr-test-framework 5.5.4 7.3.1 2017-10-20 2018-05-17 >>>>>> org.springframework:spring-expression 4.3.5.RELEASE 5.0.7.RELEASE >>>>>> 2017-01-25 2018-06-12 >>>>>> sqlline:sqlline 1.3.0 1.4.0 2017-05-30 2018-05-30 >>>>>> com.clearspring.analytics:stream 2.9.5 2.9.6 2016-08-10 2018-01-10 >>>>>> org.elasticsearch.client:transport 5.0.0 6.2.4 2016-10-25 2018-04-12 >>>>>> org.elasticsearch.plugin:transport-netty4-client 5.6.3 6.2.4 >>>>>> 2017-11-06 2018-04-12 >>>>>> org.tukaani:xz 1.5 1.8 2014-03-08 2018-01-04 >>>>>> >>>>> -- >>>> Got feedback? go/pabloem-feedback >>>> <https://goto.google.com/pabloem-feedback> >>>> >>> >>
