On Tue, Feb 6, 2018 at 5:20 AM, Robert Varga <n...@hq.sk> wrote: > On 06/02/18 06:53, Thanh Ha wrote: > > The old-style mdsal-merge job deployed and I kicked off a build: > > > > https://jenkins.opendaylight.org/releng/view/Merge-Jobs/ > job/mdsal-merge-oxygen/1/console > > > > It takes about 50 minutes for mdsal to build so we'll know in ~50 > > minutes if that fixed the issue. > > Hello Thanh, > > the one thing I found is a difference in file upload strategy in the two > jobs: > > old: > > > Deploying the main artifact iana-afn-safi-2013.07.04.12.0-SNAPSHOT.jar > > Downloading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml > > Downloaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml (2 KB at 20.8 KB/sec) > > Uploading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013. > 07.04.12.0-20180206.071107-112.jar > > Uploaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013. > 07.04.12.0-20180206.071107-112.jar (23 KB at 2.4 KB/sec) > > Uploading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013. > 07.04.12.0-20180206.071107-112.pom > > Uploaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013. > 07.04.12.0-20180206.071107-112.pom (2 KB at 7.5 KB/sec) > > Downloading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/maven-metadata.xml > > Downloaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/maven-metadata.xml (481 B at 31.3 KB/sec) > > Uploading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml > > Uploaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml (2 KB at 4.3 KB/sec) > > Uploading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/maven-metadata.xml > > Uploaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/maven-metadata.xml (481 B at 0.6 KB/sec) > > Deploying the main artifact iana-afn-safi-2013.07.04.12.0- > SNAPSHOT-javadoc.jar > > Uploading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013. > 07.04.12.0-20180206.071107-112-javadoc.jar > > Uploaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013. > 07.04.12.0-20180206.071107-112-javadoc.jar (50 KB at 58.1 KB/sec) > > Uploading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml > > Uploaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml (2 KB at 2.7 KB/sec) > > Deploying the main artifact iana-afn-safi-2013.07.04.12.0- > SNAPSHOT-sources.jar > > Uploading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013. > 07.04.12.0-20180206.071107-112-sources.jar > > Uploaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/iana-afn-safi-2013. > 07.04.12.0-20180206.071107-112-sources.jar (13 KB at 13.0 KB/sec) > > Uploading: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml > > Uploaded: https://nexus.opendaylight.org/content/repositories/ > opendaylight.snapshot/org/opendaylight/mdsal/model/iana- > afn-safi/2013.07.04.12.0-SNAPSHOT/maven-metadata.xml (2 KB at 1.9 KB/sec) > > new: > > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/maven-metadata.xml > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.jar.sha1 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.jar > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.jar.md5 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.pom > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.pom.sha1 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111.pom.md5 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/maven-metadata.xml.sha1 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/maven-metadata.xml.md5 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111-javadoc.jar > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156- > 111-javadoc.jar.sha1 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111-javadoc.jar.md5 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111-sources.jar > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156- > 111-sources.jar.sha1 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/2013.07.04.12.0- > SNAPSHOT/iana-afn-safi-2013.07.04.12.0-20180206.034156-111-sources.jar.md5 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/maven-metadata.xml > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/maven-metadata. > xml.sha1 > > Uploading org/opendaylight/mdsal/model/iana-afn-safi/maven-metadata. > xml.md5 > > It seems we are doing things in different order, old: > - download artifact-$VERSION metadata > - upload artifact-$VERSION content (jar, pom) > - download artifact metadata > - upload artifact-$VERSION metadata > - upload artifact metadata > > Whereas new does not seem to do any sort of ordering -- and uploads also > checksums, etc. > > Now I don't know if nexus is doing anything the the metadata it has or > how it reacts to parallel deployment of related artifacts. I would > suggest creating a smarter strategy: > > - group files by their artifact name (i.e. directory) > > for each artifact file group: > - upload content (jars, pom, javadoc, sources, etc. etc.) for an > artifact in a serial manner first > - upload versioned metadata > - upload metadata > > Never use parallelism inside a group, use parallelism across groups -- > i.e. deploy multiple artifacts concurrently while retaining file upload > ordering matching how the old job does things within each artifact. > > Regards, > Robert >
Hi Robert, So how the new maven merge job works is as follows: 1. Download maven-metadata.xml using wget recursive and copies it to $WORKSPACE/m2repo and $WORKSPACE/m2repo-backup in the same structure as a maven repo. 2. Perform mvn clean deploy -DaltDeploymentRepository=staging::default::file:$WORKSPACE/m2repo in this case Maven considers this directory a m2repo and updates maven-metadata.xml as if the deployment repo was on a webserver 3. Diff the 2 repos m2repo and m2repo-backup and delete all files that were not modified 4. Use cURL + gnu-parallel to upload the artifacts to Nexus The issue is not with step 4 (Although uploading metadata last is a good idea) but with step 2 (and I realize now why the validation tool in 3 didn't catch some errors) when `mvn clean deploy` is running, for some mvn occasionally produces artifacts and maven-metadata.xml that have the different timestamps. This is is a known issue for sure with mvn 3.5.0 and fixed in 3.5.2 I'm not sure if 3.3.9 had the issue or not. Last night Luis found a job that consistently always generated the wrong metadata, when I switched that job to mvn35 it passed, would need more data to confirm if 3.5.2 really solves the issue or we got lucky but it seems like perhaps Maven 3.3.9 has issues with deploying to a local maven repo. As for why the validation tool failed to catch the issue was, we were only checking for consistency inside of maven-metadata.xml so did not catch when the maven-metadata.xml was consistent inside the file but still had different filenames. This unfortunately puts us back on the old merge job which causes Jenkins instability but considering our crunch time now, a little instability with Jenkins is better than having bad metadata. Thanh
_______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
