[native-toolchain-CR] Compile thrift using the python in our toolchain.
Hello Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15468 to look at the new patch set (#3). Change subject: Compile thrift using the python in our toolchain. .. Compile thrift using the python in our toolchain. Change-Id: Iec332462ba7f9eaa699247f546d2b6ba1faabd60 --- M buildall.sh M source/python/build.sh M source/thrift/build.sh 3 files changed, 10 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/68/15468/3 -- To view, visit http://gerrit.cloudera.org:8080/15468 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iec332462ba7f9eaa699247f546d2b6ba1faabd60 Gerrit-Change-Number: 15468 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Add support for centos8
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/15465 ) Change subject: Add support for centos8 .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/15465/1/docker/all/postinstall.sh File docker/all/postinstall.sh: http://gerrit.cloudera.org:8080/#/c/15465/1/docker/all/postinstall.sh@31 PS1, Line 31: alternatives --set python /usr/bin/python2 > Just a question: does this alias pip2 to pip as well? Kinda. pip doesn't get installed through the OS packages. It gets installed on line 39 of this script. Since it gets called using python2, we end up with pip running under python2.7 (os dependent). So basically: dnf install python2 python3 # which python which: no python in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin) # which pip which: no pip in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin) # alternatives --set python /usr/bin/python2 # which python /usr/bin/python # which pip which: no pip in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin) # python get-pip.py # which pip /usr/bin/pip http://gerrit.cloudera.org:8080/#/c/15465/1/docker/redhat8.df File docker/redhat8.df: http://gerrit.cloudera.org:8080/#/c/15465/1/docker/redhat8.df@48 PS1, Line 48: RUN postinstall.sh : : COPY ./all > nit: I guess this comment is stale now Done -- To view, visit http://gerrit.cloudera.org:8080/15465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc15c202f61e251761fd0b1dc9aa0b15c27b3363 Gerrit-Change-Number: 15465 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 17 Mar 2020 22:49:44 + Gerrit-HasComments: Yes
[native-toolchain-CR] WIP: Add platform tag for Ubuntu 14.04
Hector Acosta has abandoned this change. ( http://gerrit.cloudera.org:8080/15469 ) Change subject: WIP: Add platform tag for Ubuntu 14.04 .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/15469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I32dac434ddf12ec3a2367f6a0974b3504bd137c5 Gerrit-Change-Number: 15469 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal
[native-toolchain-CR] Add support for centos8
Hello Laszlo Gaal, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15465 to look at the new patch set (#2). Change subject: Add support for centos8 .. Add support for centos8 This commit adds a new centos8 docker image. Most of this is pretty straightforward with the exception of having to explicitly set our default python. This needs to happen early in the postinstall process since other tools (aws cli) depend on it. Change-Id: Idc15c202f61e251761fd0b1dc9aa0b15c27b3363 --- M docker/all/assert-dependencies-present.py M docker/all/postinstall.sh A docker/redhat8.df M in-docker.py 4 files changed, 66 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/65/15465/2 -- To view, visit http://gerrit.cloudera.org:8080/15465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idc15c202f61e251761fd0b1dc9aa0b15c27b3363 Gerrit-Change-Number: 15465 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal
[native-toolchain-CR] Remove centos6, ubuntu12, ubuntu14 platforms
Hello Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15464 to look at the new patch set (#2). Change subject: Remove centos6, ubuntu12, ubuntu14 platforms .. Remove centos6, ubuntu12, ubuntu14 platforms This commit removes unused platform redhat6, and EOL platforms ubuntu12 and ubuntu14. Change-Id: Icef9293fc528bce3d60956cf3b879cf71e933403 --- M Makefile D docker/redhat6.df D docker/ubuntu1404.df M in-docker.py 4 files changed, 0 insertions(+), 111 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/64/15464/2 -- To view, visit http://gerrit.cloudera.org:8080/15464 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icef9293fc528bce3d60956cf3b879cf71e933403 Gerrit-Change-Number: 15464 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] WIP: Add platform tag for Ubuntu 14.04
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/15469 ) Change subject: WIP: Add platform tag for Ubuntu 14.04 .. Patch Set 1: > I must be missing something here: I was convinced that the earlier > change, https://gerrit.cloudera.org/c/15464/1 removed the > Ubuntu-14.04 toolchain build (so artifacts wouldn't be prodiced > either). How come that the build is still producing artifacts then? > Wouldn't the right fix be to make sure that Ubuntu-14.04 bits are > just not built? I think I found the problem. The list of supported platforms is listed in the Makefile. I'll remove them from there and will abandon this RR. -- To view, visit http://gerrit.cloudera.org:8080/15469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32dac434ddf12ec3a2367f6a0974b3504bd137c5 Gerrit-Change-Number: 15469 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Tue, 17 Mar 2020 22:34:36 + Gerrit-HasComments: No
[native-toolchain-CR] Remove CYRUS SASL
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15466 Change subject: Remove CYRUS_SASL .. Remove CYRUS_SASL CYRUS_SASL requires libdb4 which is not available in rhel8. Change-Id: I7f043c0c32dd26f3b4b7d7b16749ce310860d9c2 --- M buildall.sh 1 file changed, 0 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/66/15466/1 -- To view, visit http://gerrit.cloudera.org:8080/15466 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7f043c0c32dd26f3b4b7d7b16749ce310860d9c2 Gerrit-Change-Number: 15466 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Remove centos6, ubuntu12, ubuntu14 platforms
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15464 Change subject: Remove centos6, ubuntu12, ubuntu14 platforms .. Remove centos6, ubuntu12, ubuntu14 platforms This commit removes unused platform redhat6, and EOL platforms ubuntu12 and ubuntu14. Change-Id: Icef9293fc528bce3d60956cf3b879cf71e933403 --- D docker/redhat6.df D docker/ubuntu1404.df M in-docker.py 3 files changed, 0 insertions(+), 108 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/64/15464/1 -- To view, visit http://gerrit.cloudera.org:8080/15464 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icef9293fc528bce3d60956cf3b879cf71e933403 Gerrit-Change-Number: 15464 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Fix bison compilation with glibc 2.28
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15467 Change subject: Fix bison compilation with glibc 2.28 .. Fix bison compilation with glibc 2.28 Change-Id: Ie07da9fcebde4ae5003885f442d8856537f96f3a --- M .gitignore M buildall.sh A source/bison/bison-3.0.4-patches/110-glibc-change-work-around.patch 3 files changed, 35 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/67/15467/1 -- To view, visit http://gerrit.cloudera.org:8080/15467 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie07da9fcebde4ae5003885f442d8856537f96f3a Gerrit-Change-Number: 15467 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] WIP: Add platform tag for Ubuntu 14.04
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15469 Change subject: WIP: Add platform tag for Ubuntu 14.04 .. WIP: Add platform tag for Ubuntu 14.04 Without this the packages for Ubuntu 14.04 are uploaded with tarball names ending in 'generic', which makes bootstrap_toolchain.py unable to find them later, during an Impala build. Change-Id: I32dac434ddf12ec3a2367f6a0974b3504bd137c5 --- M in-docker.py 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/69/15469/1 -- To view, visit http://gerrit.cloudera.org:8080/15469 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I32dac434ddf12ec3a2367f6a0974b3504bd137c5 Gerrit-Change-Number: 15469 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal
[native-toolchain-CR] Add support for centos8
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15465 Change subject: Add support for centos8 .. Add support for centos8 This commit adds a new docker image.. Change-Id: Idc15c202f61e251761fd0b1dc9aa0b15c27b3363 --- M docker/all/assert-dependencies-present.py M docker/all/postinstall.sh A docker/redhat8.df M in-docker.py 4 files changed, 72 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/65/15465/1 -- To view, visit http://gerrit.cloudera.org:8080/15465 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idc15c202f61e251761fd0b1dc9aa0b15c27b3363 Gerrit-Change-Number: 15465 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Compile thrift using the python in our toolchain.
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15468 Change subject: Compile thrift using the python in our toolchain. .. Compile thrift using the python in our toolchain. Change-Id: Iec332462ba7f9eaa699247f546d2b6ba1faabd60 --- M buildall.sh M source/python/build.sh M source/thrift/build.sh 3 files changed, 8 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/68/15468/1 -- To view, visit http://gerrit.cloudera.org:8080/15468 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iec332462ba7f9eaa699247f546d2b6ba1faabd60 Gerrit-Change-Number: 15468 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Fix issues with toolchain Python and bump version
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/13249 ) Change subject: Fix issues with toolchain Python and bump version .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh File source/python/build.sh: http://gerrit.cloudera.org:8080/#/c/13249/1/source/python/build.sh@37 PS1, Line 37: # a different directory from the one used during our toolchain build. > That makes sense. I think that's the default way that ccache installs, anyw The problem here (besides what Tim mentioned) was that then we'd have to insert paths ($PATH) after $CCACHE_PATH.. more here: https://gerrit.cloudera.org/#/c/12574/3//COMMIT_MSG -- To view, visit http://gerrit.cloudera.org:8080/13249 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e6c9c4371d3d6c1193c2cc02d45c22b04137672 Gerrit-Change-Number: 13249 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 07 May 2019 19:59:46 + Gerrit-HasComments: Yes
[native-toolchain-CR] Create CCACHE DIR if not present
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/13232 ) Change subject: Create CCACHE_DIR if not present .. Patch Set 1: > LGTM, I should have caught it the first time around though... > Apparently download_ccache is called before init-compiler.sh has a > chance to create the CCACHE dir. > Letting Hector take a look as well FWIW, this works fine in the docker path (https://github.com/cloudera/native-toolchain/blob/master/in-docker.py#L112), which is why the jenkins job works fine. -- To view, visit http://gerrit.cloudera.org:8080/13232 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2bbc7753ce31df94e68a4aff20940deec9b66d64 Gerrit-Change-Number: 13232 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 07 May 2019 19:15:38 + Gerrit-HasComments: No
[native-toolchain-CR] Create CCACHE DIR if not present
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/13232 ) Change subject: Create CCACHE_DIR if not present .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13232 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2bbc7753ce31df94e68a4aff20940deec9b66d64 Gerrit-Change-Number: 13232 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 07 May 2019 18:49:12 + Gerrit-HasComments: No
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/12996/1/functions.sh File functions.sh: http://gerrit.cloudera.org:8080/#/c/12996/1/functions.sh@543 PS1, Line 543: local EXPIRES="$(date -d '+3 months' --utc +'%Y-%m-%dT%H:%M:%SZ')" > wrap at 90 chars, if possible Done http://gerrit.cloudera.org:8080/#/c/12996/1/functions.sh@545 PS1, Line 545: ON=" > nit: whitespace Done http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh File functions.sh: http://gerrit.cloudera.org:8080/#/c/12996/3/functions.sh@519 PS3, Line 519: local TAR=ccache.tar > Handling of the tarball filename is asymmetric between download_ccache() an This was done following the same logic for the download_dependency function and the publishing logic in the build_dist_package function. It looks like $S3_BUCKET is only used for uploads, which makes sense in some scenarios: This code expects a very specific layout in some urls which would need to be replicated even if we simply want to upload packages to an alternative bucket. The same can be said about the ccache tarball. Tying the bucket that gets used to upload artifacts to the bucket that's used to download dependencies makes things harder in some, admittedly specific use cases. http://gerrit.cloudera.org:8080/#/c/12996/1/init.sh File init.sh: http://gerrit.cloudera.org:8080/#/c/12996/1/init.sh@34 PS1, Line 34: # - DOWNLOAD_CCACHE > DOWNLOAD_CCACHE Done http://gerrit.cloudera.org:8080/#/c/12996/1/init.sh@133 PS1, Line 133: > Could you add a (brief) comment about this, including mentioning UPLOAD_CCA Done -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 18 Apr 2019 17:42:26 + Gerrit-HasComments: Yes
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Enable reusing ccache directories. Commit 2d6d6ba119e2bf303f82dbc3ce07e8120e7e3914 enabled building with ccache, but there was no mechanism to save and restore caches. This commit adds two functions, download_ccache and upload_ccache, which download and upload a tarred ccache directory. The cache is uploaded to s3 at the end of a full build if UPLOAD_CCACHE=1. The 3 variables that affect the handling of reusing ccache are: DOWNLOAD_CCACHE = Downloads ccache from s3. USE_CCACHE = Modifies environment variables so that ccache is called. UPLOAD_CCACHE = Uploads a ccache tarball to s3. DOWNLOAD_CCACHE and USE_CCACHE now default to 1. Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 --- M Makefile M buildall.sh M functions.sh M in-docker.py M init.sh 5 files changed, 79 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/96/12996/3 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1013201
Hector Acosta has posted comments on this change. (
http://gerrit.cloudera.org:8080/13002 )
Change subject: Bump CDP_BUILD_NUMBER to 1013201
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13002/3/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:
http://gerrit.cloudera.org:8080/#/c/13002/3/bin/bootstrap_toolchain.py@508
PS3, Line 508: toolchain_host = os.environ.get("IMPALA_TOOLCHAIN_HOST")
: cdh_build_number = os.environ.get("CDH_BUILD_NUMBER")
nit: looks like it's safe to just use os.environ['IMPALA_TOOLCHAIN_HOST'] and
os.environ['CDH_BUILD_NUMBER']
http://gerrit.cloudera.org:8080/#/c/13002/3/bin/bootstrap_toolchain.py@538
PS3, Line 538: use_cdp_hive = os.getenv("USE_CDP_HIVE") == "true"
: if use_cdp_hive:
Since use_cdp_hive is not used again:
if os.environ.get('USE_CDP_HIVE') == true:
http://gerrit.cloudera.org:8080/#/c/13002/3/buildall.sh
File buildall.sh:
http://gerrit.cloudera.org:8080/#/c/13002/3/buildall.sh@540
PS3, Line 540: #
extraneous #.. There's two more in this parragraph
--
To view, visit http://gerrit.cloudera.org:8080/13002
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I18c7274085be4f87ecdaf0cd29a601715f594ada
Gerrit-Change-Number: 13002
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Hector Acosta
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 16 Apr 2019 16:58:31 +
Gerrit-HasComments: Yes
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12996 ) Change subject: Enable reusing ccache directories. .. Enable reusing ccache directories. Commit 2d6d6ba119e2bf303f82dbc3ce07e8120e7e3914 enabled building with ccache, but there was no mechanism to save and restore caches. This commit adds two functions, download_ccache and upload_ccache, which download and upload a tarred ccache directory. The cache is uploaded to s3 at the end of a full build if UPLOAD_CCACHE=1. The 3 variables that affect the handling of reusing ccache are: DOWNLOAD_CCACHE = Downloads ccache from s3. USE_CCACHE = Modifies environment variables so that ccache is called. UPLOAD_CCACHE = Uploads a ccache tarball to s3. DOWNLOAD_CCACHE and USE_CCACHE now default to 1. Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 --- M Makefile M buildall.sh M functions.sh M in-docker.py M init.sh 5 files changed, 79 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/96/12996/2 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[native-toolchain-CR] Enable reusing ccache directories.
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12996 Change subject: Enable reusing ccache directories. .. Enable reusing ccache directories. Commit 2d6d6ba119e2bf303f82dbc3ce07e8120e7e3914 enabled building with ccache, but there was no mechanism to save and restore caches. This commit adds two functions, download_ccache and upload_ccache, which download and upload a tarred ccache directory. The cache is uploaded to s3 at the end of a full build if UPLOAD_CCACHE=1. The 3 variables that affect the handling of reusing ccache are: DOWNLOAD_CCACHE = Downloads ccache from s3. USE_CCACHE = Modifies environment variables so that ccache is called. UPLOAD_CCACHE = Uploads a ccache tarball to s3. DOWNLOAD_CCACHE and USE_CCACHE now default to 1. Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 --- M Makefile M buildall.sh M functions.sh M in-docker.py M init.sh 5 files changed, 62 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/96/12996/1 -- To view, visit http://gerrit.cloudera.org:8080/12996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I482aa13e833d4680efe7cab98aad7f4fb998bfc0 Gerrit-Change-Number: 12996 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] IMPALA-8382 Add support for SLES 12 SP3
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12922 ) Change subject: IMPALA-8382 Add support for SLES 12 SP3 .. Patch Set 3: > Patch Set 2: Commit message was updated. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/12922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia3cb1311b15226f1130be7e1d79110d16e3287ef Gerrit-Change-Number: 12922 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 06 Apr 2019 17:56:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8383 Bump toolchain version
Hector Acosta has abandoned this change. ( http://gerrit.cloudera.org:8080/12923 ) Change subject: IMPALA-8383 Bump toolchain version .. Abandoned Included in another review -- To view, visit http://gerrit.cloudera.org:8080/12923 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: If221f001bc106530c5faf08b38385028c056983f Gerrit-Change-Number: 12923 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8383 Bump toolchain version
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12923 Change subject: IMPALA-8383 Bump toolchain version .. IMPALA-8383 Bump toolchain version Thrift's python bindings in the current toolchain don't include the fastbinary shared object due to a bug in the native-toolchain code. A fix for that has been submitted to the native-toolchain and the newly generated toolchain builds should include fastbinary.so Testing: Compared thrift's tarballs using: diff <(curl -s $URL1| tar tzf - |sort) \ <(curl -s $URL2 | tar tzf -|sort) 147d146 < thrift-0.9.3-p5/python/lib/python2.7/site-packages/thrift/protocol/fastbinary.so Change-Id: If221f001bc106530c5faf08b38385028c056983f --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/12923/1 -- To view, visit http://gerrit.cloudera.org:8080/12923 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If221f001bc106530c5faf08b38385028c056983f Gerrit-Change-Number: 12923 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] IMPALA-8382 Add support for SLES 12 SP3
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12922 Change subject: IMPALA-8382 Add support for SLES 12 SP3 .. IMPALA-8382 Add support for SLES 12 SP3 Change-Id: Ia3cb1311b15226f1130be7e1d79110d16e3287ef Testing: Ran a build, reployed a cluster on sles 12 sp3. --- M bin/bootstrap_toolchain.py 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/12922/1 -- To view, visit http://gerrit.cloudera.org:8080/12922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia3cb1311b15226f1130be7e1d79110d16e3287ef Gerrit-Change-Number: 12922 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Check for thirft.protocol.fastbinary in both lib and lib64
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12915 Change subject: Check for thirft.protocol.fastbinary in both lib and lib64 .. Check for thirft.protocol.fastbinary in both lib and lib64 db301f6b992d36ea315c037fde954e01340be1e2 introduced an assertion on the thrift build script which ensures that thrift.protocol.fastbinary gets compiled in thrift's python bindings, however in some distros, the shared object is placed on $PY_PREFIX/lib64 (instead of $PY_PREFIX/lib) so we check in both locations. Change-Id: Ib2294eff2f945af94de0345322f5496777fb1b08 --- M source/thrift/build.sh 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/15/12915/1 -- To view, visit http://gerrit.cloudera.org:8080/12915 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib2294eff2f945af94de0345322f5496777fb1b08 Gerrit-Change-Number: 12915 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Thomas Marshall
[native-toolchain-CR] Install newer nss version in centos 6.6
Hello Laszlo Gaal, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12862 to look at the new patch set (#4). Change subject: Install newer nss version in centos 6.6 .. Install newer nss version in centos 6.6 Centos 6.6 ships with an nss version that can't clone from github. To fix it we include a new Centos-Vault repo file with disabled repositories and explicitly install nss from centos 6.9, which works. Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb --- M docker/all/assert-dependencies-present.py A docker/redhat/Centos6-Vault.repo M docker/redhat6.df 3 files changed, 23 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/62/12862/4 -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 4 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal
[native-toolchain-CR] Work around flatbuffers flaky build
Hector Acosta has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/12886 )
Change subject: Work around flatbuffers flaky build
..
Work around flatbuffers flaky build
The flatbuffers build ocasionally fails with the following error:
/flatbuffers/flatbuffers-1.6.0/src/idl_parser.cpp
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:19:17:
error: 'MyGame' has not been declared
using namespace MyGame::Sample;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:19:25:
error: 'Sample' is not a namespace-name
using namespace MyGame::Sample;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:19:31:
error: expected namespace-name before ';' token
using namespace MyGame::Sample;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp: In
function 'int main(int, const char**)':
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:25:3:
error: 'flatbuffers' has not been declared
flatbuffers::FlatBufferBuilder builder;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:28:26:
error: 'builder' was not declared in this scope
auto weapon_one_name = builder.CreateString("Sword");
^
...
Disabling the BUILD_TESTS target removes the flakiness and makes the
compilation faster.
Change-Id: I57e95617ccfc2d7d63cc277ab2b858f28ccad44c
---
M source/flatbuffers/build.sh
1 file changed, 6 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain
refs/changes/86/12886/3
--
To view, visit http://gerrit.cloudera.org:8080/12886
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57e95617ccfc2d7d63cc277ab2b858f28ccad44c
Gerrit-Change-Number: 12886
Gerrit-PatchSet: 3
Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Parallelize image building process
Hector Acosta has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12864 ) Change subject: Parallelize image building process .. Parallelize image building process Images can now be built in parallel when using the new buildall.py script. Other than the parallelism and logging changes, the script behaves the same way. Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 --- M docker/README.md A docker/buildall.py D docker/buildall.sh 3 files changed, 57 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/64/12864/3 -- To view, visit http://gerrit.cloudera.org:8080/12864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 Gerrit-Change-Number: 12864 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Install newer nss version in centos 6.6
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer nss version in centos 6.6 .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12862/2/docker/redhat6.df File docker/redhat6.df: http://gerrit.cloudera.org:8080/#/c/12862/2/docker/redhat6.df@53 PS2, Line 53: --enablerepo=C6.9-base nss > This worries me: my concern is that this creates a Frankenbuild with some p We've agreed that keeping the 6.6 base is the least invasive path. -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 29 Mar 2019 03:18:26 + Gerrit-HasComments: Yes
[native-toolchain-CR] Install newer nss version in centos 6.6
Hector Acosta has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer nss version in centos 6.6 .. Install newer nss version in centos 6.6 Centos 6.6 ships with an nss version that can't clone from github. To fix it we include a new Centos-Vault repo file with disabled repositories and explicitly install nss from centos 6.9, which works. Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb --- M docker/all/assert-dependencies-present.py A docker/redhat/Centos6-Vault.repo M docker/redhat6.df 3 files changed, 23 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/62/12862/3 -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal
[native-toolchain-CR] Fix debian:7 and debian:8 image builds
Hector Acosta has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12863 ) Change subject: Fix debian:7 and debian:8 image builds .. Fix debian:7 and debian:8 image builds Wheezy and Jessie were moved out of the main debian mirrors: https://lists.debian.org/debian-devel-announce/2019/03/msg6.html So we're fetching from archive now. Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 --- A docker/debian/7/sources.list A docker/debian/8/sources.list M docker/debian7.df M docker/debian8.df 4 files changed, 14 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/63/12863/3 -- To view, visit http://gerrit.cloudera.org:8080/12863 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 Gerrit-Change-Number: 12863 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Include python-dev in all build images
Hector Acosta has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12882 ) Change subject: Include python-dev in all build images .. Include python-dev in all build images Thrift doesn't fail the build if it fails to find the python development headers, it instead silently builds without support for the TBinaryProtocolAccelerated class. This commit adds python-dev to the missing docker images and adds a couple of assertions to ensure we fail the build whenever this happens. Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 --- M docker/all/assert-dependencies-present.py M docker/debian7.df M docker/debian8.df M docker/ubuntu1204.df M docker/ubuntu1404.df M docker/ubuntu1604.df M docker/ubuntu1804.df M source/thrift/build.sh 8 files changed, 22 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/82/12882/3 -- To view, visit http://gerrit.cloudera.org:8080/12882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 Gerrit-Change-Number: 12882 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Workaround flatbuffers flaky build
Hector Acosta has uploaded this change for review. (
http://gerrit.cloudera.org:8080/12886
Change subject: Workaround flatbuffers flaky build
..
Workaround flatbuffers flaky build
The flatbuffers build ocasionally fails with the following error:
/flatbuffers/flatbuffers-1.6.0/src/idl_parser.cpp
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:19:17:
error: 'MyGame' has not been declared
using namespace MyGame::Sample;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:19:25:
error: 'Sample' is not a namespace-name
using namespace MyGame::Sample;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:19:31:
error: expected namespace-name before ';' token
using namespace MyGame::Sample;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp: In
function 'int main(int, const char**)':
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:25:3:
error: 'flatbuffers' has not been declared
flatbuffers::FlatBufferBuilder builder;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:28:26:
error: 'builder' was not declared in this scope
auto weapon_one_name = builder.CreateString("Sword");
^
...
Disabling the BUILD_TESTS target removes the flakiness and makes the
compilation faster.
Change-Id: I57e95617ccfc2d7d63cc277ab2b858f28ccad44c
---
M source/flatbuffers/build.sh
1 file changed, 6 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/native-toolchain
refs/changes/86/12886/2
--
To view, visit http://gerrit.cloudera.org:8080/12886
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I57e95617ccfc2d7d63cc277ab2b858f28ccad44c
Gerrit-Change-Number: 12886
Gerrit-PatchSet: 2
Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Include python-dev in all build images
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12882 ) Change subject: Include python-dev in all build images .. Include python-dev in all build images Thrift doesn't fail the build if it fails to find the python development headers, it instead silently builds without support for the TBinaryProtocolAccelerated class. This commit adds python-dev to the missing docker images and adds a couple of assertions to ensure we fail the build whenever this happens. Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 --- M docker/all/assert-dependencies-present.py M docker/debian7.df M docker/debian8.df M docker/ubuntu1204.df M docker/ubuntu1404.df M docker/ubuntu1604.df M docker/ubuntu1804.df M source/thrift/build.sh 8 files changed, 22 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/82/12882/2 -- To view, visit http://gerrit.cloudera.org:8080/12882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 Gerrit-Change-Number: 12882 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Include python-dev in all build images
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12882 ) Change subject: Include python-dev in all build images .. Patch Set 2: Code-Review+1 Patchset one ended up failing for thrift-0.11. This approach works for all thrift versions. -- To view, visit http://gerrit.cloudera.org:8080/12882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 Gerrit-Change-Number: 12882 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 28 Mar 2019 23:54:20 + Gerrit-HasComments: No
[native-toolchain-CR] Let buildall.sh run without setting PUBLISH DEPENDENCIES*
Hector Acosta has posted comments on this change. (
http://gerrit.cloudera.org:8080/12883 )
Change subject: Let buildall.sh run without setting PUBLISH_DEPENDENCIES*
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12883/2/init.sh
File init.sh:
http://gerrit.cloudera.org:8080/#/c/12883/2/init.sh@62
PS2, Line 62: # If PUBLISH_DEPENDENCIES_S3 and/or PUBLISH_DEPENDENCIES_S3 is
set to 1, the script will
: # upload the artifacts to S3 and/or the internal artifactory
respectively.
: # PUBLISH_DEPENDENCIES=1 results in PUBLISH_DEPENDENCIES_S3=1 and
: # PUBLISH_DEPENDENCIES_ARTIFACTORY=1
: : ${PUBLISH_DEPENDENCIES=0}
: if [[ "$PUBLISH_DEPENDENCIES" -eq 1 ]]; then
: PUBLISH_DEPENDENCIES_S3=1
: PUBLISH_DEPENDENCIES_ARTIFACTORY=1
: fi
: : ${PUBLISH_DEPENDENCIES_S3=0}
: : ${PUBLISH_DEPENDENCIES_ARTIFACTORY=0}
: export PUBLISH_DEPENDENCIES
: export PUBLISH_DEPENDENCIES_S3
: export PUBLISH_DEPENDENCIES_ARTIFACTORY
I did something similar here:
https://gerrit.cloudera.org/#/c/12810/3/functions.sh
Perhaps that should be removed from functions.sh?
--
To view, visit http://gerrit.cloudera.org:8080/12883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4c88f44f597398c66881a7b4f5ebdee06a6f6b0b
Gerrit-Change-Number: 12883
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Hector Acosta
Gerrit-Reviewer: Thomas Marshall
Gerrit-Comment-Date: Thu, 28 Mar 2019 23:50:43 +
Gerrit-HasComments: Yes
[native-toolchain-CR] Include python-dev in all build images
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12882 Change subject: Include python-dev in all build images .. Include python-dev in all build images Thrift doesn't fail the build if it fails to find the python development headers, it instead silently builds without support for the TBinaryProtocolAccelerated class. This commit adds python-dev to the missing docker images and adds a couple of assertions to ensure we fail the build whenever this happens. Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 --- M docker/all/assert-dependencies-present.py M docker/debian7.df M docker/debian8.df M docker/ubuntu1204.df M docker/ubuntu1404.df M docker/ubuntu1604.df M docker/ubuntu1804.df M source/thrift/build.sh 8 files changed, 22 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/82/12882/1 -- To view, visit http://gerrit.cloudera.org:8080/12882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 Gerrit-Change-Number: 12882 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Install newer nss version in centos 6.6
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer nss version in centos 6.6 .. Install newer nss version in centos 6.6 Centos 6.6 ships with an nss version that can't clone from github. To fix it we include a new Centos-Vault repo file with disabled repositories and explicitly install nss from centos 6.9, which works. Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb --- M docker/all/assert-dependencies-present.py A docker/redhat/Centos6-Vault.repo M docker/redhat6.df 3 files changed, 390 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/62/12862/2 -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Prepend docker image to both stderr and stdout
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12861 ) Change subject: Prepend docker image to both stderr and stdout .. Prepend docker image to both stderr and stdout Sometimes, it may be hard to debug which distro produced an error, this commit ensures that both stderr and stdout get piped through sed. Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 --- M Makefile 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/61/12861/2 -- To view, visit http://gerrit.cloudera.org:8080/12861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 Gerrit-Change-Number: 12861 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Install newer git version in centos 6.6
Hector Acosta has restored this change. ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer git version in centos 6.6 .. Restored -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: restore Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Install newer git version in centos 6.6
Hector Acosta has abandoned this change. ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer git version in centos 6.6 .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Fix debian:7 and debian:8 image builds
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12863 Change subject: Fix debian:7 and debian:8 image builds .. Fix debian:7 and debian:8 image builds Wheezy and Jessie were moved out of the main debian mirrors: https://lists.debian.org/debian-devel-announce/2019/03/msg6.html So we're fetching from archive now. Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 --- A docker/debian/7/sources.list A docker/debian/8/sources.list M docker/debian7.df M docker/debian8.df 4 files changed, 14 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/63/12863/1 -- To view, visit http://gerrit.cloudera.org:8080/12863 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 Gerrit-Change-Number: 12863 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Prepend docker image to both stderr and stdout
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12861 Change subject: Prepend docker image to both stderr and stdout .. Prepend docker image to both stderr and stdout Sometimes, it may be hard to debug which distro produced an error, this commit ensures that both stderr and stdout get piped through sed. Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 --- M Makefile 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/61/12861/1 -- To view, visit http://gerrit.cloudera.org:8080/12861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 Gerrit-Change-Number: 12861 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Install newer git version in centos 6.6
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12862 Change subject: Install newer git version in centos 6.6 .. Install newer git version in centos 6.6 Centos 6.6 contains a git version that doesn't work over dumb-http. To fix it we include a new Centos-Vault repo file with disabled repositories and explicitly install git from centos 6.9, which works. Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb --- M docker/all/assert-dependencies-present.py A docker/redhat/Centos6-Vault.repo M docker/redhat6.df 3 files changed, 389 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/62/12862/1 -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Parallelize image building process
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12864 Change subject: Parallelize image building process .. Parallelize image building process Images can now be built in parallel when using the new buildall.py script. Other than the parallelism and logging changes, the script behaves the same way. Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 --- M docker/README.md A docker/buildall.py D docker/buildall.sh 3 files changed, 57 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/64/12864/1 -- To view, visit http://gerrit.cloudera.org:8080/12864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 Gerrit-Change-Number: 12864 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] Use 'wget http://169.254.169.254/' to determine if we're running in aws
Hello Lars Volker, Laszlo Gaal, Jim Apple, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12727 to look at the new patch set (#6). Change subject: Use 'wget http://169.254.169.254/' to determine if we're running in aws .. Use 'wget http://169.254.169.254/' to determine if we're running in aws https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html lists the above endpoint as the first choice. In my running instance 'dmidecode -s bios-version' prints out: 1.0 dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative here. I removed the 'grep' assertions since it's possible that images may have already configured ntp here. Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 --- M bin/bootstrap_system.sh 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/12727/6 -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 6 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] Use 'wget http://169.254.169.254/' to determine if we're running in aws
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use 'wget http://169.254.169.254/' to determine if we're running in aws .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12727/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12727/4//COMMIT_MSG@7 PS4, Line 7: `wget http://169.254.169.254/` > Please excuse the drive-by comment. Can we change the quotes to normale quo Done -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 4 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 22 Mar 2019 19:39:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Use 'wget http://169.254.169.254/' to determine if we're running in aws
Hello Lars Volker, Laszlo Gaal, Jim Apple, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12727 to look at the new patch set (#5). Change subject: Use 'wget http://169.254.169.254/' to determine if we're running in aws .. Use 'wget http://169.254.169.254/' to determine if we're running in aws https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html lists the above endpoint as the first choice. In my running instance `dmidecode -s bios-version` prints out: 1.0 dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative here. I removed the 'grep' assertions since it's possible that images may have already configured ntp here. Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 --- M bin/bootstrap_system.sh 1 file changed, 3 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/12727/5 -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 5 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] Use `wget http://169.254.169.254/` to determine if we're running in aws
Hello Laszlo Gaal, Jim Apple, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12727 to look at the new patch set (#4). Change subject: Use `wget http://169.254.169.254/` to determine if we're running in aws .. Use `wget http://169.254.169.254/` to determine if we're running in aws https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html lists the above endpoint as the first choice. In my running instance `dmidecode -s bios-version` prints out: 1.0 dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative here. I removed the `grep` assertions since it's possible that images may have already configured ntp here. Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 --- M bin/bootstrap_system.sh 1 file changed, 3 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/12727/4 -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 4 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] Use `wget http://169.254.169.254/` to determine if we're running in aws
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use `wget http://169.254.169.254/` to determine if we're running in aws .. Patch Set 3: Code-Review+1 Carry +1 -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Thu, 21 Mar 2019 16:18:19 + Gerrit-HasComments: No
[Impala-ASF-CR] Use `wget http://169.254.169.254/` to determine if we're running in aws
Hello Laszlo Gaal, Jim Apple, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12727 to look at the new patch set (#3). Change subject: Use `wget http://169.254.169.254/` to determine if we're running in aws .. Use `wget http://169.254.169.254/` to determine if we're running in aws https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html lists the above endpoint as the first choice. In my running instance `dmidecode -s bios-version` prints out: 1.0 dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative here. I removed the `grep` assertions since it's possible that images may have already configured ntp here. Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 --- M bin/bootstrap_system.sh 1 file changed, 2 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/12727/3 -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Laszlo Gaal
[native-toolchain-CR] Add documentation for docker-based toolchain builds
Hello Thomas Marshall, Laszlo Gaal, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12810 to look at the new patch set (#2). Change subject: Add documentation for docker-based toolchain builds .. Add documentation for docker-based toolchain builds This commit adds basic documentation with regard to docker-based toolchain builds. It briefly outlines how to build images and how to use them to build the toolchain. I also added the ability to fetch images from a docker registry. And fixed an 'undefined' variable problem with docker-based builds. Change-Id: Id971bda58d537fa15ec63004f744d3f730bad170 --- M Makefile A docker/README.md M functions.sh 3 files changed, 52 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/10/12810/2 -- To view, visit http://gerrit.cloudera.org:8080/12810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id971bda58d537fa15ec63004f744d3f730bad170 Gerrit-Change-Number: 12810 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[native-toolchain-CR] Add documentation for docker-related toolchain builds
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12810 Change subject: Add documentation for docker-related toolchain builds .. Add documentation for docker-related toolchain builds This commit adds basic documentation with regard to docker-based toolchain builds. It breifly outlines how to build images and how to use them to build the toolchain. I also added the ability to fetch images from a docker registry. And fixed an 'undefined' variable problem with docker-based builds. Change-Id: Id971bda58d537fa15ec63004f744d3f730bad170 --- M Makefile A docker/README.md M functions.sh 3 files changed, 52 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/10/12810/1 -- To view, visit http://gerrit.cloudera.org:8080/12810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id971bda58d537fa15ec63004f744d3f730bad170 Gerrit-Change-Number: 12810 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[native-toolchain-CR] Fix error when PUBLISH DEPENDENCIES=0
Hector Acosta has posted comments on this change. (
http://gerrit.cloudera.org:8080/12793 )
Change subject: Fix error when PUBLISH_DEPENDENCIES=0
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12793/1/functions.sh
File functions.sh:
http://gerrit.cloudera.org:8080/#/c/12793/1/functions.sh@391
PS1, Line 391: if [[ "PUBLISH_DEPENDENCIES" -eq "1" ]]; then
: : ${PUBLISH_DEPENDENCIES_S3=1}
: : ${PUBLISH_DEPENDENCIES_ARTIFACTORY=1}
: else
: : ${PUBLISH_DEPENDENCIES_S3=0}
: : ${PUBLISH_DEPENDENCIES_ARTIFACTORY=0}
: fi
I'm about to basically submit the same thing. I think this is equivalent to:
: ${PUBLISH_DEPENDENCIES_S3=${PUBLISH_DEPENDENCIES}}
--
To view, visit http://gerrit.cloudera.org:8080/12793
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ief2d6d9daf1b06091f3773b5abae0aebe0a3d15a
Gerrit-Change-Number: 12793
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Hector Acosta
Gerrit-Comment-Date: Tue, 19 Mar 2019 23:33:20 +
Gerrit-HasComments: Yes
[native-toolchain-CR] Add ccache support
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 ) Change subject: Add ccache support .. Patch Set 2: > Hector, are you blocked on something? Or just waiting for someone > to merge? Just waiting on someone to merge. -- To view, visit http://gerrit.cloudera.org:8080/12574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144 Gerrit-Change-Number: 12574 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 17:14:42 + Gerrit-HasComments: No
[Impala-ASF-CR] Use `wget http://169.254.169.254/` to determine if we're running in aws
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use `wget http://169.254.169.254/` to determine if we're running in aws .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12727/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12727/2//COMMIT_MSG@9 PS2, Line 9: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html > That says "check the instance identity document, including its signature". IIUC wget is not doing that. In order to verify the instance metadata returned by http://169.254.169.254/ we'd need to perform the steps outlined here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html Verifying metadata signatures here seems like an overkill to me. This patch makes the assumption that if http://169.254.169.254/ is reachable then it's reasonable to assume that we're in an ec2 instance. we're just using /latest/dynamic/instance-identity/ as a documented url endpoint. -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 18 Mar 2019 04:30:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Add missing sudo call to `service postgresql start`
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12724 ) Change subject: Add missing sudo call to `service postgresql start` .. Add missing sudo call to `service postgresql start` Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75 --- M bin/bootstrap_system.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/12724/2 -- To view, visit http://gerrit.cloudera.org:8080/12724 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75 Gerrit-Change-Number: 12724 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-8307 Use test -x to check for ntp-wait
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12726 ) Change subject: IMPALA-8307 Use test -x to check for ntp-wait .. IMPALA-8307 Use test -x to check for ntp-wait Running ntp-wait --help can return 0 or 1 depending on the output of ntpq. For example: $ sudo ntp-wait --help 2> /dev/null; echo $? 0 $ sudo killall ntpd $ sudo ntp-wait --help 2> /dev/null; echo $? 1 This commit instead tests whether ntp-wait exists and is executable to determine if it's installed. Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c --- M testdata/cluster/admin 1 file changed, 10 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/12726/2 -- To view, visit http://gerrit.cloudera.org:8080/12726 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c Gerrit-Change-Number: 12726 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] Move DEBIAN FRONTEND=noninteractive variable to the ubuntu section of the script
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12723 ) Change subject: Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script .. Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script This also changes the behavior to always use DEBIAN_FRONTEND=noninteractive even for interactive terminals. Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac --- M bin/bootstrap_system.sh 1 file changed, 1 insertion(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/12723/2 -- To view, visit http://gerrit.cloudera.org:8080/12723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac Gerrit-Change-Number: 12723 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] Use `wget http://169.254.169.254/` to determine if we're running in aws
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use `wget http://169.254.169.254/` to determine if we're running in aws .. Use `wget http://169.254.169.254/` to determine if we're running in aws https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html lists the above endpoint as the first choice. In my running instance `dmidecode -s bios-version` prints out: 1.0 dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative here. I removed the `grep` assertions since it's possible that images may have already configured ntp here. Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 --- M bin/bootstrap_system.sh 1 file changed, 1 insertion(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/12727/2 -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-8314 Set correct permissions for ~/.ssh files
Hector Acosta has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/12725 ) Change subject: IMPALA-8314 Set correct permissions for ~/.ssh files .. IMPALA-8314 Set correct permissions for ~/.ssh files Without this running bootstrap_system.sh resulted in: ++ chmod go-rwx /home/hacosta/.ssh ++ [[ -f /home/hacosta/.ssh/id_rsa ]] ++ ssh-keygen -t rsa -N '' -q -f /home/hacosta/.ssh/id_rsa ++ cat /home/hacosta/.ssh/id_rsa.pub ++ echo 'NoHostAuthenticationForLocalhost yes' ++ ssh localhost whoami Bad owner or permissions on /home/hacosta/.ssh/config Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087 --- M bin/bootstrap_system.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/12725/2 -- To view, visit http://gerrit.cloudera.org:8080/12725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087 Gerrit-Change-Number: 12725 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] Use `curl http://169.254.169.254/` to determine if we're running in aws
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use `curl http://169.254.169.254/` to determine if we're running in aws .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12727/1/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/12727/1/bin/bootstrap_system.sh@240 PS1, Line 240: http://169.254.169.254 > On second thought: we may need a stricter test for AWS if the goal is to de I'm not sure I would be able to properly test other non-ec2 cases. But that's outside the scope of what I'm trying to accomplish here. I'm happy to either: a) Re-implement using wget and timeout values. b) Abandon this for now and create a more robust version based on testing whether ntp endpoints are reachable I'm leaning towards a, lmk if you agree. -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Wed, 13 Mar 2019 23:26:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Move DEBIAN FRONTEND=noninteractive variable to the ubuntu section of the script
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12723 ) Change subject: Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script .. Patch Set 1: > (1 comment) > > Let me know what you think. If we go with always using > DEBIAN_FRONTEND=noninteractive, then please note that in the commit > message. I can confirm that with the current package listing, on an ubuntu 16.04 docker image the script behaves the same way before and after this patch (no interactive prompts on apt-get). Personally, I prefer to export DEBIAN_FRONTEND=noninteractive in the interactive terminal case since a) we're already assuming that this script can be used by automated scripts (ex. docker/entrypoint.sh) b) we already perform system-wide configuration changes (postgres, ntp, etc.) so presumably if in the future we add a dependency that does require user interaction we can modify the configuration files as we do now (or revisit this patch altogether :-). -- To view, visit http://gerrit.cloudera.org:8080/12723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac Gerrit-Change-Number: 12723 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 13 Mar 2019 22:42:18 + Gerrit-HasComments: No
[Impala-ASF-CR] Use test -x to check for ntp-wait
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12726 ) Change subject: Use test -x to check for ntp-wait .. Patch Set 1: > This looks good to me. Quick question: Is this a cosmetic issue or > is there an actual failure case? I'm wondering if we want a JIRA > for this. If it is cosmetic, then I think we can skip the JIRA. There's a failure case: ntp-wait runs `/usr/sbin/ntpq -c "rv 0"` regardless of whether --help is used or not.. Its exit status depends on the output of ntpq. So for example: [root@8280aa22322d /]# ntp-wait --help ; echo $? /usr/sbin/ntp-wait version [unknown] calling Getopt::Std::getopts (version 1.07 [paranoid]), running under Perl version 5.16.3. Usage: ntp-wait [-OPTIONS [-MORE_OPTIONS]] [--] [PROGRAM_ARG1 ...] The following single-character options are accepted: With arguments: -n -s Boolean (without arguments): -v Options may be merged together. -- stops processing of options. Space is not required between options and their arguments. [Now continuing due to backward compatibility and excessive paranoia. See 'perldoc Getopt::Std' about $Getopt::Std::STANDARD_HELP_VERSION.] 1 [root@8280aa22322d /]# ntpd [root@8280aa22322d /]# ntp-wait --help ; echo $? /usr/sbin/ntp-wait version [unknown] calling Getopt::Std::getopts (version 1.07 [paranoid]), running under Perl version 5.16.3. Usage: ntp-wait [-OPTIONS [-MORE_OPTIONS]] [--] [PROGRAM_ARG1 ...] The following single-character options are accepted: With arguments: -n -s Boolean (without arguments): -v Options may be merged together. -- stops processing of options. Space is not required between options and their arguments. [Now continuing due to backward compatibility and excessive paranoia. See 'perldoc Getopt::Std' about $Getopt::Std::STANDARD_HELP_VERSION.] 0 It also looks like --help is simply ignored. -- To view, visit http://gerrit.cloudera.org:8080/12726 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c Gerrit-Change-Number: 12726 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 13 Mar 2019 20:05:36 + Gerrit-HasComments: No
[Impala-ASF-CR] Use `curl http://169.254.169.254/` to determine if we're running in aws
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12727 Change subject: Use `curl http://169.254.169.254/` to determine if we're running in aws .. Use `curl http://169.254.169.254/` to determine if we're running in aws https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html lists the above endpoint as the first choice. In my running instance `dmidecode -s bios-version` prints out: 1.0 dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative here. I removed the `grep` assertions since it's possible that images may have already configured ntp here. Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 --- M bin/bootstrap_system.sh 1 file changed, 1 insertion(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/12727/1 -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] Use test -x to check for ntp-wait
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12726 Change subject: Use test -x to check for ntp-wait .. Use test -x to check for ntp-wait In centos 7, running ntp-wait --help results in: /usr/sbin/ntp-wait version [unknown] calling Getopt::Std::getopts (version 1.07 [paranoid]), running under Perl version 5.16.3. Usage: ntp-wait [-OPTIONS [-MORE_OPTIONS]] [--] [PROGRAM_ARG1 ...] The following single-character options are accepted: With arguments: -n -s Boolean (without arguments): -v Options may be merged together. -- stops processing of options. Space is not required between options and their arguments. [Now continuing due to backward compatibility and excessive paranoia. See 'perldoc Getopt::Std' about $Getopt::Std::STANDARD_HELP_VERSION.] So instead of running it we get the path using command -v. Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c --- M testdata/cluster/admin 1 file changed, 10 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/12726/1 -- To view, visit http://gerrit.cloudera.org:8080/12726 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I53c63dfa651ac242050171da70540d24c4caf32c Gerrit-Change-Number: 12726 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] Add missing sudo call to `service postgresql start`
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12724 Change subject: Add missing sudo call to `service postgresql start` .. Add missing sudo call to `service postgresql start` Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75 --- M bin/bootstrap_system.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/12724/1 -- To view, visit http://gerrit.cloudera.org:8080/12724 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie3859ad1308b7cd1d483d043b9f71d9f95abdf75 Gerrit-Change-Number: 12724 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] Move DEBIAN FRONTEND=noninteractive variable to the ubuntu section of the script
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12723 Change subject: Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script .. Move DEBIAN_FRONTEND=noninteractive variable to the ubuntu section of the script Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac --- M bin/bootstrap_system.sh 1 file changed, 1 insertion(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/12723/1 -- To view, visit http://gerrit.cloudera.org:8080/12723 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifab2f56d3e119e4e73685c3cbd5249760ceb33ac Gerrit-Change-Number: 12723 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[Impala-ASF-CR] Set correct permissions for ~/.ssh files
Hector Acosta has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12725 Change subject: Set correct permissions for ~/.ssh files .. Set correct permissions for ~/.ssh files Without this I got the following error in my ssdh log: Authentication refused: bad ownership or modes for file $HOME/.ssh/authorized_keys Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087 --- M bin/bootstrap_system.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/12725/1 -- To view, visit http://gerrit.cloudera.org:8080/12725 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3dd6b12318dd99c67127bff10a89adb931f44087 Gerrit-Change-Number: 12725 Gerrit-PatchSet: 1 Gerrit-Owner: Hector Acosta
[native-toolchain-CR] Add ccache support
Hello Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12574 to look at the new patch set (#2). Change subject: Add ccache support .. Add ccache support Here we add ccache support, which speeds up building the entire toolchain. This almost works out of the box, except for the following complications: * We bootstrap our own gcc, so CCACHE_COMPILERCHECK must be set to 'content', this is because by default ccache includes the modification time of the compiler, which would prevent us to reuse the cache between runs. * We specifically set CC and CXX variables, the initial approach of setting them to 'ccache $CC' breaks boost (and possibly others) because they try to run `ccache $CC`. A second approach that i tried was relying on PATH ordering to make ccache execute the right thing. When using the system compiler, we want: PATH=$CCACHE_PATH:/usr/bin:... CC=$CCACHE_PATH/gcc After bootstrapping we want: PATH=$CCACHE_PATH:build/gcc-4.9.2:$PATH CC=$CCACHE_PATH/gcc I think complex modification of PATH is too britle and would cause ccache to execute the system compiler when $CCACHE_PATH == /usr/bin. Instead we create a wrapper around ccache that executes CC and place it at the beginning of the path, we set CC to this wrapper script. I ran `make -j DISTROS="debian8 redhat6 redhat7 sles12 ubuntu1604 ubuntu1804"` on the following configs: disabled ccache: 3.77user 1.80system 2:08:37elapsed 0%CPU (0avgtext+0avgdata 52680maxresident)k empty ccache: 3.97user 1.83system 2:15:51elapsed 0%CPU (0avgtext+0avgdata 49956maxresident)k empty ccache (compressed): 3.86user 2.05system 2:15:12elapsed 0%CPU (0avgtext+0avgdata 53824maxresident)k full ccache: 2.25user 1.79system 49:14.07elapsed 0%CPU (0avgtext+0avgdata 47520maxresident)k full ccache: (compressed): 2.40user 1.80system 49:45.01elapsed 0%CPU (0avgtext+0avgdata 51484maxresident)k Here's ccache -s after the full ccache build (statistics were cleared before strting the build). + ccache -s cache directory build_docker/ccache primary config build_docker/ccache/ccache.conf secondary config (readonly)/etc/ccache.conf cache hit (direct)138821 cache hit (preprocessed) 17437 cache miss 225 cache hit rate 99.86 % called for link15551 called for preprocessing9567 multiple source files 42 compile failed 7705 preprocessor error 2647 bad compiler arguments 6047 unsupported source language 42 autoconf compile/link 18651 unsupported compiler option 342 unsupported code directive24 no input file 4323 cleanups performed 0 files in cache289759 cache size 30.5 GB max cache size 50.0 GB Statistics using CCACHE_COMPRESS=1 are similar but cache size is ~7GB. Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144 --- M .gitignore M docker/all/assert-dependencies-present.py M docker/all/postinstall.sh M docker/debian7.df M docker/debian8.df M docker/redhat6.df M docker/redhat7.df M docker/sles12.df M docker/ubuntu1204.df M docker/ubuntu1404.df M docker/ubuntu1604.df M docker/ubuntu1804.df M functions.sh M in-docker.py M init-compiler.sh M init.sh 16 files changed, 154 insertions(+), 113 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/74/12574/2 -- To view, visit http://gerrit.cloudera.org:8080/12574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144 Gerrit-Change-Number: 12574 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Philip Zeyliger
[native-toolchain-CR] Add ccache support
Hector Acosta has posted comments on this change. (
http://gerrit.cloudera.org:8080/12574 )
Change subject: Add ccache support
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12574/1/docker/debian7.df
File docker/debian7.df:
http://gerrit.cloudera.org:8080/#/c/12574/1/docker/debian7.df@43
PS1, Line 43: wget "https://www.samba.org/ftp/ccache/ccache-3.3.3.tar.gz; && \
> I recommend moving this into a shell script.
Done
http://gerrit.cloudera.org:8080/#/c/12574/1/init.sh
File init.sh:
http://gerrit.cloudera.org:8080/#/c/12574/1/init.sh@138
PS1, Line 138: : ${USE_CCACHE=0}
: export USE_CCACHE
:
: : ${CCACHE_MAXSIZE=50G}
: export CCACHE_MAXSIZE
:
: : ${CCACHE_DIR=$SOURCE_DIR/ccache}
: export CCACHE_DIR
:
: # Default ccache_compilercheck is mtime, which considers CC's
mtime + size
: # to determine if there's a hit. Setting CCACHE_COMPILERCHECK to
'content'
: # uses the hash of the compiler instead.
: export CCACHE_COMPILERCHECK=${CCACHE_COMPILERCHECK:-content}
:
: export CCACHE_COMPRESS=1
> Do you think these would be better encapsulated in a .ccache.conf file that
I think there's a trade-off here.. If we create and mount a ccache.conf, ccache
would only work inside docker.
If we create the ccache.conf somewhere in functions.sh it would work both
inside and outside of docker, but since someone could have already defined
CCACHE_DIR, we'd need to dance around it to prevent overwriting that conf (not
sure if this matters in practice).
I know being able to run this in the host helped me debugging some problems so
i lean towards being able to enable ccache independent of docker.
--
To view, visit http://gerrit.cloudera.org:8080/12574
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144
Gerrit-Change-Number: 12574
Gerrit-PatchSet: 1
Gerrit-Owner: Hector Acosta
Gerrit-Reviewer: Hector Acosta
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Mon, 04 Mar 2019 17:58:54 +
Gerrit-HasComments: Yes
