Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16320
to look at the new patch set (#6).
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still symlinks all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This allows each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 120 MB,
and the statestored image by 700 MB.
impalad_coordinator and impalad_coordinator images are now 771 MB, and
impalad_executor images are 651MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
---
M CMakeLists.txt
M docker/CMakeLists.txt
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
M docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/hive-exec/CMakeLists.txt
R mvn-deps/shaded-deps/hive-exec/pom.xml
R mvn-deps/shaded-deps/s3a-aws-sdk/CMakeLists.txt
R mvn-deps/shaded-deps/s3a-aws-sdk/pom.xml
17 files changed, 398 insertions(+), 251 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/6
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar <stak...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
