[Impala-ASF-CR] Download toolchain in parallel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8237 ) Change subject: Download toolchain in parallel. .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8237 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46a6088bb002402c7653dbc8257dff869afb26ec Gerrit-Change-Number: 8237 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 10 Oct 2017 01:25:25 + Gerrit-HasComments: No
[Impala-ASF-CR] Download toolchain in parallel.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8237 ) Change subject: Download toolchain in parallel. .. Download toolchain in parallel. By downloading from the toolchain S3 buckets in parallel with extracting them, this improves bootstrap_toolchain on my machine from about 1m5s to about 30s. $rm -rf toolchain; time bin/bootstrap_toolchain.py > /dev/null real0m29.226s user0m46.516s sys 0m33.820s On a large EC2 machine, closer to the S3 buckets, the new time is 21s. Because multiprocessing hasn't always been available (python2.4 on RHEL5 won't have it), I fall back to a simpler implementation Change-Id: I46a6088bb002402c7653dbc8257dff869afb26ec Reviewed-on: http://gerrit.cloudera.org:8080/8237 Reviewed-by: Tim Armstrong Reviewed-by: Alex Behm Reviewed-by: Michael Brown Tested-by: Impala Public Jenkins --- M bin/bootstrap_toolchain.py 1 file changed, 30 insertions(+), 7 deletions(-) Approvals: Tim Armstrong: Looks good to me, but someone else must approve Alex Behm: Looks good to me, approved Michael Brown: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8237 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I46a6088bb002402c7653dbc8257dff869afb26ec Gerrit-Change-Number: 8237 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Download toolchain in parallel.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8237 ) Change subject: Download toolchain in parallel. .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1317/ -- To view, visit http://gerrit.cloudera.org:8080/8237 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46a6088bb002402c7653dbc8257dff869afb26ec Gerrit-Change-Number: 8237 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 09 Oct 2017 21:21:36 + Gerrit-HasComments: No
[Impala-ASF-CR] Download toolchain in parallel.
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/8237 ) Change subject: Download toolchain in parallel. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8237 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46a6088bb002402c7653dbc8257dff869afb26ec Gerrit-Change-Number: 8237 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 09 Oct 2017 21:21:12 + Gerrit-HasComments: No
[Impala-ASF-CR] Download toolchain in parallel.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8237 ) Change subject: Download toolchain in parallel. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8237 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46a6088bb002402c7653dbc8257dff869afb26ec Gerrit-Change-Number: 8237 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 09 Oct 2017 21:17:14 + Gerrit-HasComments: No
[Impala-ASF-CR] Download toolchain in parallel.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8237 ) Change subject: Download toolchain in parallel. .. Patch Set 1: Code-Review+1 Makes sense to me. Will give others a chance to look. -- To view, visit http://gerrit.cloudera.org:8080/8237 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46a6088bb002402c7653dbc8257dff869afb26ec Gerrit-Change-Number: 8237 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 09 Oct 2017 21:11:33 + Gerrit-HasComments: No
[Impala-ASF-CR] Download toolchain in parallel.
Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8237 Change subject: Download toolchain in parallel. .. Download toolchain in parallel. By downloading from the toolchain S3 buckets in parallel with extracting them, this improves bootstrap_toolchain on my machine from about 1m5s to about 30s. $rm -rf toolchain; time bin/bootstrap_toolchain.py > /dev/null real0m29.226s user0m46.516s sys 0m33.820s On a large EC2 machine, closer to the S3 buckets, the new time is 21s. Because multiprocessing hasn't always been available (python2.4 on RHEL5 won't have it), I fall back to a simpler implementation Change-Id: I46a6088bb002402c7653dbc8257dff869afb26ec --- M bin/bootstrap_toolchain.py 1 file changed, 30 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/8237/1 -- To view, visit http://gerrit.cloudera.org:8080/8237 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I46a6088bb002402c7653dbc8257dff869afb26ec Gerrit-Change-Number: 8237 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger
