[native-toolchain-CR] Enable reusing ccache directories.
Laszlo Gaal has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Enable reusing ccache directories. Commit 2d6d6ba119e2bf303f82dbc3ce07e8120e7e3914 enabled building with ccache, but there was no mechanism to save and restore caches. This commit adds two functions, download_ccache and upload_ccache, which download and upload a tarred ccache directory. The cache is uploaded to s3 at the end of a full build if UPLOAD_CCACHE=1. The 3 variables that affect the handling of reusing ccache are: DOWNLOAD_CCACHE = Downloads ccache from s3. USE_CCACHE = Modifies environment variables so that ccache is called. UPLOAD_CCACHE = Uploads a ccache tarball to s3. DOWNLOAD_CCACHE and USE_CCACHE now default to 1. Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Reviewed-on: http://gerrit.cloudera.org:8080/12996 Reviewed-by: Thomas Marshall Reviewed-by: Laszlo Gaal Tested-by: Laszlo Gaal --- M Makefile M buildall.sh M functions.sh M in-docker.py M init.sh 5 files changed, 79 insertions(+), 13 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Laszlo Gaal: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 4 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[native-toolchain-CR] Enable reusing ccache directories.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 30 Apr 2019 15:26:37 + Gerrit-HasComments: No
[native-toolchain-CR] Enable reusing ccache directories.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Patch Set 3: Verified+1 Verified using private Jenkins infrastructure. -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 30 Apr 2019 15:32:17 + Gerrit-HasComments: No
[native-toolchain-CR] Enable reusing ccache directories.
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 24 Apr 2019 18:02:06 + Gerrit-HasComments: No
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/12996/1/functions.sh File functions.sh: http://gerrit.cloudera.org:8080/#/c/12996/1/functions.sh@543 PS1, Line 543: local EXPIRES="$(date -d '+3 months' --utc +'%Y-%m-%dT%H:%M:%SZ')" > wrap at 90 chars, if possible Done http://gerrit.cloudera.org:8080/#/c/12996/1/functions.sh@545 PS1, Line 545: ON=" > nit: whitespace Done http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh File functions.sh: http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh@519 PS3, Line 519: local TAR=ccache.tar > Handling of the tarball filename is asymmetric between download_ccache() an This was done following the same logic for the download_dependency function and the publishing logic in the build_dist_package function. It looks like $S3_BUCKET is only used for uploads, which makes sense in some scenarios: This code expects a very specific layout in some urls which would need to be replicated even if we simply want to upload packages to an alternative bucket. The same can be said about the ccache tarball. Tying the bucket that gets used to upload artifacts to the bucket that's used to download dependencies makes things harder in some, admittedly specific use cases. http://gerrit.cloudera.org:8080/#/c/12996/1/init.sh File init.sh: http://gerrit.cloudera.org:8080/#/c/12996/1/init.sh@34 PS1, Line 34: # - DOWNLOAD_CCACHE > DOWNLOAD_CCACHE Done http://gerrit.cloudera.org:8080/#/c/12996/1/init.sh@133 PS1, Line 133: > Could you add a (brief) comment about this, including mentioning UPLOAD_CCA Done -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 18 Apr 2019 17:42:26 + Gerrit-HasComments: Yes
[native-toolchain-CR] Enable reusing ccache directories.
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Patch Set 3: Code-Review+1 (2 comments) Just a few points on readability http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh File functions.sh: http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh@519 PS3, Line 519: local TAR=ccache.tar Handling of the tarball filename is asymmetric between download_ccache() and upload_ccache(): it is abstracted here (maybe suggesting that it can be changed), but hardcoded in upload_ccache(). Would be easier to read if it followed the same convention. OTOH if there's a reason for the asymmetry, then a comment could clarify it. http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh@520 PS3, Line 520: S3_URL="https://native-toolchain Bucket name is hardcoded here, but configurable in upload_ccache(). Would be nice if it were symetrical. I realize that other places hardcode the bucket name too, but these two functions are obviously related. -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 16 Apr 2019 21:56:45 + Gerrit-HasComments: Yes
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Enable reusing ccache directories. Commit 2d6d6ba119e2bf303f82dbc3ce07e8120e7e3914 enabled building with ccache, but there was no mechanism to save and restore caches. This commit adds two functions, download_ccache and upload_ccache, which download and upload a tarred ccache directory. The cache is uploaded to s3 at the end of a full build if UPLOAD_CCACHE=1. The 3 variables that affect the handling of reusing ccache are: DOWNLOAD_CCACHE = Downloads ccache from s3. USE_CCACHE = Modifies environment variables so that ccache is called. UPLOAD_CCACHE = Uploads a ccache tarball to s3. DOWNLOAD_CCACHE and USE_CCACHE now default to 1. Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 --- M Makefile M buildall.sh M functions.sh M in-docker.py M init.sh 5 files changed, 79 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/96/12996/3 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Enable reusing ccache directories. Commit 2d6d6ba119e2bf303f82dbc3ce07e8120e7e3914 enabled building with ccache, but there was no mechanism to save and restore caches. This commit adds two functions, download_ccache and upload_ccache, which download and upload a tarred ccache directory. The cache is uploaded to s3 at the end of a full build if UPLOAD_CCACHE=1. The 3 variables that affect the handling of reusing ccache are: DOWNLOAD_CCACHE = Downloads ccache from s3. USE_CCACHE = Modifies environment variables so that ccache is called. UPLOAD_CCACHE = Uploads a ccache tarball to s3. DOWNLOAD_CCACHE and USE_CCACHE now default to 1. Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 --- M Makefile M buildall.sh M functions.sh M in-docker.py M init.sh 5 files changed, 79 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/96/12996/2 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[native-toolchain-CR] Enable reusing ccache directories.
Thomas Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/12996 )
Change subject: Enable reusing ccache directories.
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/12996/1/functions.sh
File functions.sh:
http://gerrit.cloudera.org:8080/#/c/12996/1/functions.sh@543
PS1, Line 543: tar --exclude="ccache.done" --exclude="ccache.conf" -C
$(dirname $CCACHE_DIR) -cf - $(basename $CCACHE_DIR) |\
wrap at 90 chars, if possible
http://gerrit.cloudera.org:8080/#/c/12996/1/functions.sh@545
PS1, Line 545: |aws
nit: whitespace
http://gerrit.cloudera.org:8080/#/c/12996/1/init.sh
File init.sh:
http://gerrit.cloudera.org:8080/#/c/12996/1/init.sh@34
PS1, Line 34: # - FAIL_ON_PUBLISH
DOWNLOAD_CCACHE
http://gerrit.cloudera.org:8080/#/c/12996/1/init.sh@133
PS1, Line 133: : ${DOWNLOAD_CCACHE=1}
Could you add a (brief) comment about this, including mentioning UPLOAD_CCACHE
--
To view, visit http://gerrit.cloudera.org:8080/12996
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0
Gerrit-Change-Number: 12996
Gerrit-PatchSet: 1
Gerrit-Owner: Hector Acosta
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Thomas Marshall
Gerrit-Comment-Date: Mon, 15 Apr 2019 21:00:31 +
Gerrit-HasComments: Yes
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12996 Change subject: Enable reusing ccache directories. .. Enable reusing ccache directories. Commit 2d6d6ba119e2bf303f82dbc3ce07e8120e7e3914 enabled building with ccache, but there was no mechanism to save and restore caches. This commit adds two functions, download_ccache and upload_ccache, which download and upload a tarred ccache directory. The cache is uploaded to s3 at the end of a full build if UPLOAD_CCACHE=1. The 3 variables that affect the handling of reusing ccache are: DOWNLOAD_CCACHE = Downloads ccache from s3. USE_CCACHE = Modifies environment variables so that ccache is called. UPLOAD_CCACHE = Uploads a ccache tarball to s3. DOWNLOAD_CCACHE and USE_CCACHE now default to 1. Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 --- M Makefile M buildall.sh M functions.sh M in-docker.py M init.sh 5 files changed, 62 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/96/12996/1 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
