[
https://issues.apache.org/jira/browse/IMPALA-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133834#comment-17133834
]
ASF subversion and git services commented on IMPALA-9107:
---------------------------------------------------------
Commit fb282852ef52d72079a86c55a90982ffac567cc7 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fb28285 ]
IMPALA-9107 (part 2): Add script to use the m2 archive tarball
This adds a script to find an appropriate m2 archive
tarball, download it, and use it to prepopulate the
~/.m2 directory.
The script uses the JSON interface for Jenkins to search through
the all-build-options-ub1604 builds on jenkins.impala.io to
find one that:
1. Is building the "master" branch
2. Has the m2_archive.tar.gz
Then, it downloads the m2 archive and uses it to populate ~/.m2.
It does not overwrite or remove any files already in ~/.m2.
The build scripts that call populate_m2_directory.py do not
rely on the script succeeding. They will continue even if
the script fails.
This also modifies the build-all-flag-combinations.sh script
to only build the m2 archive if the GENERATE_M2_ARCHIVE
environment variable is true. GENERATE_M2_ARCHIVE=true will
clear out the ~/.m2 directory to build an accurate m2 archive.
Precommit jobs will use GENERATE_M2_ARCHIVE=false, which
will allow them to use the m2 archive to speed up the build.
Testing:
- Ran gerrify-verify-dryrun
- Tested locally
Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Reviewed-on: http://gerrit.cloudera.org:8080/15735
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Reduce time spent downloading maven artifacts for precommit tests
> -----------------------------------------------------------------
>
> Key: IMPALA-9107
> URL: https://issues.apache.org/jira/browse/IMPALA-9107
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend, Infrastructure
> Affects Versions: Impala 3.4.0
> Reporter: Joe McDonnell
> Priority: Major
>
> When building Impala from scratch with an empty .m2 directory, maven needs to
> download a large number of jars and pom files. This is taking a long time and
> adds about 15-20 minutes to the build. For example, here is some output from
> a recent ubuntu-16.04-from-scratch, there is an 18 minute delay between the
> end of building the backend tests and the end of building the frontend:
> {code:java}
> 00:58:33 [100%] Built target hash-ring-util
> ...
> 01:16:37 [100%] Built target fe{code}
> Almost all of that time is spent downloading maven artifacts. Quite a few
> come from the maven central repository.
> This is taking way too much time, and we need to reduce it. The total size of
> artifacts being downloaded is not large. One approach would be to produce a
> tarball with the jars/poms that don't come from the CDP or CDH GBN repos. We
> can download that tarball and use it to either populate the .m2 directory or
> to stash it in IMPALA_TOOLCHAIN and use it as a maven repository with a
> file:// URI.
> This impacts all our jobs: ubuntu-16.04-from-scratch,
> ubuntu-16.04-dockerised-tests, all-build-options-ub1604,
> ubuntu-16.04-build-only, clang-tidy-ub1604
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
