Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20549
Change subject: Add secondary S3 upload location for toolchain packages ...................................................................... Add secondary S3 upload location for toolchain packages All Apache Impala precommit activities happen in AWS EC2, in their us-west-2 region, while the native-toolchain bucket hosting the binary toolchain packages is located in us-west-1 for historical reasons. This adds a constant cost to every single build launched on the upstream infrastucture. This patch adds a secondary upload location to the package upload logic to allow a second, "mirroring" bucket to be configured, which is expected to be closer to the build location network-wise. The mirror bucket is currently expected to be located in EC2's us-west-2 region, but this is not specified in the upload code. The secondary upload location is expected in a new environment variable, S3_MIRROR_BUCKET. The upload logic adds aggressive cache-control directives as well, relying on the fact that each toolchain build generates binary packages with a unique name, so the packages can be considered immutable after upload. Note: One important feature of the canonical toolchain bucket, s3://native-toolchain, is to store the source packages for the toolchain components. This function is not duplicated in the mirror buckets, so toolchain builds will still download their source packages from s3://native-toolchain, regardless of any upload location (primary or mirror) specified. The patch fixes a random typo as well. Tested by running a full toolchain build, uploading the binary packages to two temporary test buckets in S3, then comparing the result across the two buckets. When this passed, a core-mode test pass was executed on the Impala upstream master branch. Change-Id: I3aabaaf699dde865efe11e600f4e78aeca550378 --- M functions.sh M in-docker.py 2 files changed, 19 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/49/20549/1 -- To view, visit http://gerrit.cloudera.org:8080/20549 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3aabaaf699dde865efe11e600f4e78aeca550378 Gerrit-Change-Number: 20549 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal <laszlo.g...@cloudera.com>
