[Impala-ASF-CR] WIP: IMPALA-3343 (part 4), IMPALA-9489: Impala-shell on python 3.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15524 ) Change subject: WIP: IMPALA-3343 (part 4), IMPALA-9489: Impala-shell on python 3. .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5561/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb004d352fe230a890a6b6356496ba76c2fab615 Gerrit-Change-Number: 15524 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Mar 2020 08:01:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Adam Tamas has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. Patch Set 15: (3 comments) http://gerrit.cloudera.org:8080/#/c/15288/15/tests/custom_cluster/test_web_pages.py File tests/custom_cluster/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/15288/15/tests/custom_cluster/test_web_pages.py@114 PS15, Line 114: imput > nit: typo Done http://gerrit.cloudera.org:8080/#/c/15288/15/tests/custom_cluster/test_web_pages.py@130 PS15, Line 130: part > nit: partial Done http://gerrit.cloudera.org:8080/#/c/15288/15/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/15288/15/tests/webserver/test_web_pages.py@422 PS15, Line 422: x > nit: I would explicitly write it as "x " (notice the space after the x), li Done -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 15 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 23 Mar 2020 09:59:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15527 ) Change subject: IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5562/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie55b89bb689375cbb55c22f6d5543a5422b29a82 Gerrit-Change-Number: 15527 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Mar 2020 09:42:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Adam Tamas has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. IMPALA-6360: Don't show full query statement on Impala WebUI by default In the WebUI's query list the query statements are trimmed, but the full query statement can be seen in the details page. The default statement length is 250 chars and it can be adjusted by the query_stmt_size flag that can be set when the cluster starts. Example: bin/start-impala-cluster.py -s1 --impalad_args --query_stmt_size=10 Testing: -manual testing in the WebUI. -added 'test_query_stmt()' to test_web_pages.py -added test to the custom cluster webserver tests to check without truncate and with custom length truncate Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 --- M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M tests/custom_cluster/test_web_pages.py M tests/webserver/test_web_pages.py M www/queries.tmpl 5 files changed, 62 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/15288/16 -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 16 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15214 ) Change subject: IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5560/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9690a926953a8d3c3872277680b4be0551546c68 Gerrit-Change-Number: 15214 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 06:43:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt
zhaoren...@hotmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15527 Change subject: IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt .. IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt Reduce duplicate code in thrift CMakeLists.txt. And in order to support aarch64, we need to use hive 4.0.0 version. This commit can deal situations when hive version is 2.x.x, 3.x.x and 4.x.x. Change-Id: Ie55b89bb689375cbb55c22f6d5543a5422b29a82 --- M common/thrift/CMakeLists.txt 1 file changed, 2 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/15527/1 -- To view, visit http://gerrit.cloudera.org:8080/15527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie55b89bb689375cbb55c22f6d5543a5422b29a82 Gerrit-Change-Number: 15527 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward
[Impala-ASF-CR] IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15527 ) Change subject: IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5517/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie55b89bb689375cbb55c22f6d5543a5422b29a82 Gerrit-Change-Number: 15527 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Mar 2020 09:28:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15363 ) Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5515/ -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 11:01:16 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15528 Change subject: Fix test_fuzz_nested_types .. Fix test_fuzz_nested_types test_fuzz_nested_types had queries that didn't parse successfully because they had duplicated names in an inline view: select count(*) from ( select ... a.pos, ... b.pos from ... ); We have 'a.pos' and 'b.pos' here but Impala still considers those as duplicated names, hence I added aliases for the duplicated fields, e.g.: a.pos as apos Now that the queries pase successfully they start the scanners on corrupted files which is the goal of this test. Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f --- M tests/query_test/test_scanners_fuzz.py 1 file changed, 6 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/15528/1 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 17 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 23 Mar 2020 12:54:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 16 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 23 Mar 2020 12:54:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5519/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 17 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 23 Mar 2020 12:54:59 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 1: Code-Review+2 Oops, thanks for catching up this! -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 13:03:49 + Gerrit-HasComments: No
[Impala-ASF-CR](asf-site) IMPALA-9462: Point to downloads.apache.org for current release
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/15443 ) Change subject: IMPALA-9462: Point to downloads.apache.org for current release .. Patch Set 1: > Patch Set 1: > > I was thinking I would update this when I upload the Impala 3.4 artifacts. > Impala 3.3 will move to having an archive.apache.org URL and Impala 3.4 will > go directly to downloads.apache.org. SGTM -- To view, visit http://gerrit.cloudera.org:8080/15443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: Iccc15ef785529d477958012a3f5c83715580a26d Gerrit-Change-Number: 15443 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 12:38:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9451: Fix test hive text codec interop.py failure in CDP build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15520 ) Change subject: IMPALA-9451: Fix test_hive_text_codec_interop.py failure in CDP build .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5518/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief8e583aae82f548754f41e07efac5d7bca4b930 Gerrit-Change-Number: 15520 Gerrit-PatchSet: 6 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 11:17:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9451: Fix test hive text codec interop.py failure in CDP build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15520 ) Change subject: IMPALA-9451: Fix test_hive_text_codec_interop.py failure in CDP build .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief8e583aae82f548754f41e07efac5d7bca4b930 Gerrit-Change-Number: 15520 Gerrit-PatchSet: 6 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 11:17:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. Patch Set 16: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5563/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 16 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 23 Mar 2020 13:01:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5521/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 17 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 23 Mar 2020 13:14:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9451: Fix test hive text codec interop.py failure in CDP build
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15520 ) Change subject: IMPALA-9451: Fix test_hive_text_codec_interop.py failure in CDP build .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief8e583aae82f548754f41e07efac5d7bca4b930 Gerrit-Change-Number: 15520 Gerrit-PatchSet: 5 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 11:07:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15527 ) Change subject: IMPALA-9543 Reduce duplicate code in thrift CMakeLists.txt .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5517/ -- To view, visit http://gerrit.cloudera.org:8080/15527 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie55b89bb689375cbb55c22f6d5543a5422b29a82 Gerrit-Change-Number: 15527 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Mar 2020 12:20:21 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5520/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 13:08:24 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 13:08:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8980: Remove functional*.alltypesinsert from EE tests
Adam Tamas has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15529 Change subject: IMPALA-8980: Remove functional*.alltypesinsert from EE tests .. IMPALA-8980: Remove functional*.alltypesinsert from EE tests -Modified the ‘test_insert.py’ so the tests can run parallel. -Every test will create its own temporary tables for insert testing. -Swapped out the Reset table and Drop partition SETUP tags to Truncate table QUERY statement. -Deleted a test query in ‘insert.test’. The test incorrectly tried to clear out a table which was not working, and should not work either. Testing: -tests/run-tests.py query_test/test_insert.py -impala-py.test tests/query_test/test_insert.py Change-Id: I257e936868917a2fcc6c030f6c855b247e8a0eea --- M testdata/workloads/functional-query/queries/QueryTest/insert-mem-limit.test M testdata/workloads/functional-query/queries/QueryTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/insert_bad_expr.test M testdata/workloads/functional-query/queries/QueryTest/insert_null.test M testdata/workloads/functional-query/queries/QueryTest/insert_overwrite.test M tests/query_test/test_insert.py 6 files changed, 106 insertions(+), 162 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/15529/1 -- To view, visit http://gerrit.cloudera.org:8080/15529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I257e936868917a2fcc6c030f6c855b247e8a0eea Gerrit-Change-Number: 15529 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 4 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 15:29:48 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5564/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 13:49:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9042: Milestone 1: properly scan files that has full ACID schema
Hello Quanlong Huang, Norbert Luksa, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15395
to look at the new patch set (#5).
Change subject: IMPALA-9042: Milestone 1: properly scan files that has full
ACID schema
..
IMPALA-9042: Milestone 1: properly scan files that has full ACID schema
Full ACID row format looks like this:
{
"operation": 0,
"originalTransaction": 1,
"bucket": 536870912,
"rowId": 0,
"currentTransaction": 1,
"row": {"i": 1}
}
User columns are nested under "row". In the frontend we need to create
slot descriptors that correspond to the file schema. In the catalog we
could mimic the file schema but that would introduce several
complexities and corner cases in column resolution. Also in query
results the heading of the above user column would be "row.i". Star
expansion should also be modified, etc.
Because of that in the Catalog I create the exact opposite of the above
schema:
{
"row__id":
{
"operation": 0,
"originalTransaction": 1,
"bucket": 536870912,
"rowId": 0,
"currentTransaction": 1
}
"i": 1
}
This way very little modification is needed in the frontend. And the
hidden columns can be easily retrieved via 'SELECT row__id.*' when we
need those for debugging/testing.
We only need to change Path.getAbsolutePath() to return a schema path
that corresponds to the file schema. Also in the backend we need some
extra juggling in OrcSchemaResolver::ResolveColumn() to retrieve the
table schema path from the file schema path.
Testing:
I changed data loading to load ORC files in full ACID format by default.
With this change we should be able to scan full ACID tables that are
not minor-compacted, don't have deleted rows, and don't have original
files.
Newly added Tests:
* specific queries about hidden columns (full-acid-rowid.test)
* SHOW CREATE TABLE (show-create-table-full-acid.test)
* DESCRIBE [FORMATTED] TABLE (describe-path.test)
* INSERT should be forbidden (acid-negative.test)
* added tests for column masking (
ranger_column_masking_complex_types.test)
Change-Id: Ic2e2afec00c9a5cf87f1d61b5fe52b0085844bcb
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/orc-metadata-utils.cc
M be/src/exec/orc-metadata-utils.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M common/thrift/CatalogObjects.thrift
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/Path.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java
M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java
M testdata/bin/generate-schema-statements.py
M testdata/datasets/README
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M
testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
M
testdata/workloads/functional-query/queries/DataErrorsTest/orc-type-checks.test
M testdata/workloads/functional-query/queries/QueryTest/acid-negative.test
M testdata/workloads/functional-query/queries/QueryTest/acid.test
M
testdata/workloads/functional-query/queries/QueryTest/create-table-like-file-orc.test
M testdata/workloads/functional-query/queries/QueryTest/describe-path.test
A testdata/workloads/functional-query/queries/QueryTest/full-acid-rowid.test
M
testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking.test
A
testdata/workloads/functional-query/queries/QueryTest/ranger_column_masking_complex_types.test
A
[Impala-ASF-CR] IMPALA-9042: Milestone 1: properly scan files that has full ACID schema
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15395 ) Change subject: IMPALA-9042: Milestone 1: properly scan files that has full ACID schema .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/15395/5/testdata/bin/generate-schema-statements.py File testdata/bin/generate-schema-statements.py: http://gerrit.cloudera.org:8080/#/c/15395/5/testdata/bin/generate-schema-statements.py@319 PS5, Line 319: ' flake8: E129 visually indented line with same indent as next logical line http://gerrit.cloudera.org:8080/#/c/15395/5/tests/query_test/test_scanners_fuzz.py File tests/query_test/test_scanners_fuzz.py: http://gerrit.cloudera.org:8080/#/c/15395/5/tests/query_test/test_scanners_fuzz.py@197 PS5, Line 197: . flake8: E501 line too long (91 > 90 characters) http://gerrit.cloudera.org:8080/#/c/15395/5/tests/query_test/test_scanners_fuzz.py@283 PS5, Line 283: flake8: E261 at least two spaces before inline comment http://gerrit.cloudera.org:8080/#/c/15395/5/tests/query_test/test_scanners_fuzz.py@301 PS5, Line 301: n flake8: E129 visually indented line with same indent as next logical line -- To view, visit http://gerrit.cloudera.org:8080/15395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2e2afec00c9a5cf87f1d61b5fe52b0085844bcb Gerrit-Change-Number: 15395 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 14:58:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Sahil Takiar has removed a vote on this change. Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] WIP: IMPALA-3343 (part 4), IMPALA-9489: Impala-shell on python 3.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15524 ) Change subject: WIP: IMPALA-3343 (part 4), IMPALA-9489: Impala-shell on python 3. .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5516/ -- To view, visit http://gerrit.cloudera.org:8080/15524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb004d352fe230a890a6b6356496ba76c2fab615 Gerrit-Change-Number: 15524 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Mar 2020 13:31:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5522/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 15:30:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 15:30:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8980: Remove functional*.alltypesinsert from EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15529 ) Change subject: IMPALA-8980: Remove functional*.alltypesinsert from EE tests .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5565/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I257e936868917a2fcc6c030f6c855b247e8a0eea Gerrit-Change-Number: 15529 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Mar 2020 13:52:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/15300/7/be/src/gutil/atomicops-internals-arm64.h File be/src/gutil/atomicops-internals-arm64.h: http://gerrit.cloudera.org:8080/#/c/15300/7/be/src/gutil/atomicops-internals-arm64.h@183 PS7, Line 183: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/15300/7/be/src/gutil/atomicops-internals-arm64.h@198 PS7, Line 198: ); line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/15300/7/be/src/gutil/atomicops-internals-arm64.h@198 PS7, Line 198: ); tab used for whitespace -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 15:58:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5569/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 16:28:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15454 ) Change subject: IMPALA-3766: optionally compress spilled data .. Patch Set 9: This is read for review. I based it on top of my IWYU patch because it drastically improved compile times when modifying tmp-file-mgr.h -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 16:28:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9042: Milestone 1: properly scan files that has full ACID schema
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15395 ) Change subject: IMPALA-9042: Milestone 1: properly scan files that has full ACID schema .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5566/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15395 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2e2afec00c9a5cf87f1d61b5fe52b0085844bcb Gerrit-Change-Number: 15395 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 15:43:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5523/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 16:09:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
zhaoren...@hotmail.com has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. IMPALA-9428 Add arm64 atomic ops Atomic ops are implemented by asm. Different arc have diffent implementation. Here add arm64 atomic ops implementation. Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd --- A be/src/gutil/atomicops-internals-arm64.h M be/src/gutil/atomicops.h M be/src/gutil/cpu.cc 3 files changed, 479 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/15300/9 -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15363 ) Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5525/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 16:16:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Hello Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15454 to look at the new patch set (#9). Change subject: IMPALA-3766: optionally compress spilled data .. IMPALA-3766: optionally compress spilled data Enabled via --disk_spill_compression_codec, which uses the same syntax as the compression_codec query option. Recommended codecs are LZ4 and ZSTD. ZSTD supports specifying a compression level. The compression is done in TmpFileMgr using a temporary buffer. Allocation of disk space is reworked slightly so that the allocation can happen after compression. The default power-of-two disk block sizes would lead to a lot of internal fragmentation, so a new strategy for free space management, similar to that used in the data cache, can be used with --disk_spill_punch_holes=true. TmpFileMgr will allocate a range of the actual compressed size and punch holes in the file for each range that is no longer needed. UncompressedWriteIoBytes is added to the buffer pool profiles, so that you can see what degree of compression is achieved. Typically I saw ratios of 2-3x for LZ4 and ZSTD (with LZ4 toward the lower end and ZSTD toward the higher end). Limitations: The management of the compression buffer memory could be improved. Ideally it would be integrated with the buffer pool and use the buffer pool allocator instead of being done "on the side". We would probably want to do this before making this the default, for resource management and performance reasons (doing a malloc() directly does not use the caching supported by the buffer pool). Testing: * Run buffer pool spilling tests with different combinations of the new options. * Extend existing TmpFileMgr tests for file space allocation to run with hole punching enabled. * Switch a couple of spilling tests to use the new option. * Add a metrics test to check for scratch leaks. * Enable the new options by default for end-to-end dockerized tests to get additional coverage. * Add a unit test where allocating compression memory fails, both on the read and write path. * Ran a single-node stress test on TPC-DS SF 1 and TPC-H SF 10 The peak compression buffer usage was ~40MB. Perf: I ran this spilling query using an SSD as the scratch disk: set mem_limit=200m; select count(distinct l_partkey) from tpch30_parquet.lineitem; The time taken for the second run of each query was: No compression: 19.59s LZ4: 18.56s ZSTD: 20.59s Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 --- M be/src/runtime/bufferpool/buffer-pool-counters.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/service/query-options.cc M be/src/util/parse-util.cc M be/src/util/parse-util.h M bin/jenkins/dockerized-impala-run-tests.sh M tests/custom_cluster/test_scratch_disk.py M tests/verifiers/metric_verifier.py 15 files changed, 752 insertions(+), 211 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15454/9 -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15214 ) Change subject: IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9690a926953a8d3c3872277680b4be0551546c68 Gerrit-Change-Number: 15214 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 17:11:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/15378 )
Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol
..
Patch Set 11:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@656
PS11, Line 656: if self.max_tries == 1:
: return 0
: ratio = float(num_tries) / self.max_tries
: if ratio < 0.3:
: return 0.1
: elif ratio < 0.6:
: return 0.3
: return 2
if i'm reading this correctly, the first retry will have num_tries = 1, so the
ratio will be 0., so the method will return 0.3. when num_tries
= 2 it will return 2, and when num_tries = 3 it will return 2. not sure if that
is your intention.
i think a better retry policy would be:
* first retry: don't sleep at all
* second retry: sleep 1 second
* third retry: sleep 2 seconds
furthermore, I think this method should be robust enough so that it still
returns a reasonable retry policy if the value of max_retries is changed.
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@697
PS11, Line 697: self.close_query(set_all_handle)
isn't this already retried?
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@701
PS11, Line 701: except Exception, e:
wont this retry TApplicationException still?
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@935
PS11, Line 935: rpc
you might want to document that this should be a python function and not a
lambda because the error message include the rpc.__name__
http://gerrit.cloudera.org:8080/#/c/15378/8/tests/custom_cluster/test_hs2_fault_injection.py
File tests/custom_cluster/test_hs2_fault_injection.py:
http://gerrit.cloudera.org:8080/#/c/15378/8/tests/custom_cluster/test_hs2_fault_injection.py@128
PS8, Line 128: OpenSession and CloseImpalaOperation rpcs fail.
> I think the point of having a frequency is so that we have some faults and
oh yeah duh
http://gerrit.cloudera.org:8080/#/c/15378/11/tests/custom_cluster/test_hs2_fault_injection.py
File tests/custom_cluster/test_hs2_fault_injection.py:
http://gerrit.cloudera.org:8080/#/c/15378/11/tests/custom_cluster/test_hs2_fault_injection.py@302
PS11, Line 302: output = capsys.readouterr()[0].splitlines()
: assert output[0] == ("Caught exception HTTP code 502:
Injected Fault, "
: "type= in GetLog. Num
remaining tries: 2")
since this pattern is duplicated in several places, i think it would make sense
to add a dedicated method for it and just pass directly pass in the strings
that should be matched to the output
--
To view, visit http://gerrit.cloudera.org:8080/15378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5
Gerrit-Change-Number: 15378
Gerrit-PatchSet: 11
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Mon, 23 Mar 2020 17:43:19 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
zhaoren...@hotmail.com has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. IMPALA-9428 Add arm64 atomic ops Atomic ops are implemented by asm. Different arc have diffent implementation. Here add arm64 atomic ops implementation. Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd --- A be/src/gutil/atomicops-internals-arm64.h M be/src/gutil/atomicops.h M be/src/gutil/cpu.cc 3 files changed, 480 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/15300/7 -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15214 ) Change subject: IMPALA-8870: Bump up guava version from 14.0.1 to 28.1-jre .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5567/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15214 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9690a926953a8d3c3872277680b4be0551546c68 Gerrit-Change-Number: 15214 Gerrit-PatchSet: 7 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 16:10:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5568/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 7 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 16:38:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9451: Fix test hive text codec interop.py failure in CDP build
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15520 ) Change subject: IMPALA-9451: Fix test_hive_text_codec_interop.py failure in CDP build .. IMPALA-9451: Fix test_hive_text_codec_interop.py failure in CDP build In CDP build we use Hive3 which has a bug HIVE-22371 (CTAS puts files in the wrong place). It causes failure of newly added test as CTAS creates empty table. Workaround by explicitly creating an external table when hive version >= 3. Tested: Run this test in newest CDP build using job impala-private-basic-parameterized. Change-Id: Ief8e583aae82f548754f41e07efac5d7bca4b930 Reviewed-on: http://gerrit.cloudera.org:8080/15520 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_hive_text_codec_interop.py 1 file changed, 17 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ief8e583aae82f548754f41e07efac5d7bca4b930 Gerrit-Change-Number: 15520 Gerrit-PatchSet: 7 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9451: Fix test hive text codec interop.py failure in CDP build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15520 ) Change subject: IMPALA-9451: Fix test_hive_text_codec_interop.py failure in CDP build .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ief8e583aae82f548754f41e07efac5d7bca4b930 Gerrit-Change-Number: 15520 Gerrit-PatchSet: 6 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 17:55:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. Patch Set 17: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 17 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 23 Mar 2020 17:59:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15533/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15533/1//COMMIT_MSG@16 PS1, Line 16: $USE_CDP_HIVE : is false. how about when USE_CDP_HIVE=true? do the ranger tests pass now? I expect there are a bunch of other failures as well, but we should validate that at least data-load passes and the ranger tests have now been fixed. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 18:45:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15533 Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 This patch bumps up CDP_BUILD_NUMBER to 2244454. Due to a recent change in RANGER-2688, we added to ranger-admin-site.xml.template a cookie-related configuration so that Ranger server could be properly started. Testing: Except for a currently known issue of IMPALA-9513, verified that this patch passes the exhaustive tests in the DEBUG build when $USE_CDP_HIVE is false. Change-Id: I7750f73834368c7109965e78b147238fc6316f49 --- M bin/impala-config.sh M testdata/cluster/ranger/ranger-admin-site.xml.template 2 files changed, 13 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/15533/1 -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. IMPALA-6360: Don't show full query statement on Impala WebUI by default In the WebUI's query list the query statements are trimmed, but the full query statement can be seen in the details page. The default statement length is 250 chars and it can be adjusted by the query_stmt_size flag that can be set when the cluster starts. Example: bin/start-impala-cluster.py -s1 --impalad_args --query_stmt_size=10 Testing: -manual testing in the WebUI. -added 'test_query_stmt()' to test_web_pages.py -added test to the custom cluster webserver tests to check without truncate and with custom length truncate Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Reviewed-on: http://gerrit.cloudera.org:8080/15288 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M tests/custom_cluster/test_web_pages.py M tests/webserver/test_web_pages.py M www/queries.tmpl 5 files changed, 62 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 18 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5570/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 18:26:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5571/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 19:30:16 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5520/ -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 18:08:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Abhishek Rawat has posted comments on this change. (
http://gerrit.cloudera.org:8080/15378 )
Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol
..
Patch Set 11:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@656
PS11, Line 656: if self.max_tries == 1:
: return 0
: ratio = float(num_tries) / self.max_tries
: if ratio < 0.3:
: return 0.1
: elif ratio < 0.6:
: return 0.3
: return 2
> if i'm reading this correctly, the first retry will have num_tries = 1, so
The current logic basically has following (We have 3 tries total including the
first one):
1s try
if fail
2nd try
if fail
3rd try
if fail return error
It's probably not that robust if someone uses 10 tries. But, we do need to be
able to cap off the sleep time to a reasonable duration. I will think about a
more robust function.
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@697
PS11, Line 697: self.close_query(set_all_handle)
> isn't this already retried?
Also, this is redundant, since we already close the query in 'finally' block. I
will remove this statement.
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@701
PS11, Line 701: except Exception, e:
> wont this retry TApplicationException still?
It does, but we seem to be overusing RPCException for a variety of exceptions.
If we get a TApplicationException, or a HTTP error code, or even if the impala
server returns an error response, we throw RPCException.
We could raise a different type of exception from _do_hs2_rpc, if we get a
TApplicationException? But, that will also require changing handling the new
type of exception in the impala_shell.py.
http://gerrit.cloudera.org:8080/#/c/15378/11/shell/impala_client.py@935
PS11, Line 935: rpc
> you might want to document that this should be a python function and not a
Done
http://gerrit.cloudera.org:8080/#/c/15378/10/tests/custom_cluster/test_hs2_fault_injection.py
File tests/custom_cluster/test_hs2_fault_injection.py:
http://gerrit.cloudera.org:8080/#/c/15378/10/tests/custom_cluster/test_hs2_fault_injection.py@136
PS10, Line 136:
> flake8: E501 line too long (91 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/15378/10/tests/custom_cluster/test_hs2_fault_injection.py@237
PS10, Line 237:
> flake8: E501 line too long (91 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/15378/11/tests/custom_cluster/test_hs2_fault_injection.py
File tests/custom_cluster/test_hs2_fault_injection.py:
http://gerrit.cloudera.org:8080/#/c/15378/11/tests/custom_cluster/test_hs2_fault_injection.py@302
PS11, Line 302: output = capsys.readouterr()[0].splitlines()
: assert output[0] == ("Caught exception HTTP code 502:
Injected Fault, "
: "type= in GetLog. Num
remaining tries: 2")
> since this pattern is duplicated in several places, i think it would make s
Done
--
To view, visit http://gerrit.cloudera.org:8080/15378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5
Gerrit-Change-Number: 15378
Gerrit-PatchSet: 11
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Mon, 23 Mar 2020 18:16:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8690: Add LIRS cache eviction algorithm
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15306 ) Change subject: IMPALA-8690: Add LIRS cache eviction algorithm .. Patch Set 18: (5 comments) http://gerrit.cloudera.org:8080/#/c/15306/15/be/src/util/cache/lirs-cache.cc File be/src/util/cache/lirs-cache.cc: http://gerrit.cloudera.org:8080/#/c/15306/15/be/src/util/cache/lirs-cache.cc@83 PS15, Line 83: erreference recency), and it has three different types of e > Yeah, this needed more clarity. I reworked this paragraph. makes a lot more sense, thanks! http://gerrit.cloudera.org:8080/#/c/15306/15/be/src/util/cache/lirs-cache.cc@122 PS15, Line 122: > Added a description here along with the lifecycle. makes sense, thanks! http://gerrit.cloudera.org:8080/#/c/15306/18/be/src/util/cache/lirs-cache.cc File be/src/util/cache/lirs-cache.cc: http://gerrit.cloudera.org:8080/#/c/15306/18/be/src/util/cache/lirs-cache.cc@77 PS18, Line 77: // If the key has only been accessed once, its reuse distance is considered infinite. if i'm reading the hdfs-file-reader.cc and data-cache.cc code correctly, it looks like it calls Lookup twice whenever there is a cache miss. one call occurs when trying to read the data: HdfsFileReader::ReadDataCache --> DataCache::Partition::Lookup --> Cache::Lookup if that results in a cache miss, it calls tries to insert the data into the cache resulting in another call to Lookup: HdfsFileReader::WriteDataCache --> DataCache::Partition::Store --> Cache::Lookup so pretty much every entry in the cache will have a non-infinite reuse distance will this cause problems for LIRS? http://gerrit.cloudera.org:8080/#/c/15306/18/be/src/util/cache/lirs-cache.cc@150 PS18, Line 150: ref_count nit: could you add some docs for 'ref_count' its not clear to me when it needs to be incremented http://gerrit.cloudera.org:8080/#/c/15306/18/be/src/util/cache/lirs-cache.cc@411 PS18, Line 411: HandleTable table_; nit: add docs -- To view, visit http://gerrit.cloudera.org:8080/15306 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I670fa4b2b7c93998130dc4e8b2546bb93e9a84f8 Gerrit-Change-Number: 15306 Gerrit-PatchSet: 18 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 23 Mar 2020 18:42:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8980: Remove functional*.alltypesinsert from EE tests
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15529 ) Change subject: IMPALA-8980: Remove functional*.alltypesinsert from EE tests .. Patch Set 1: (3 comments) Thanks for taking care about this! http://gerrit.cloudera.org:8080/#/c/15529/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15529/1//COMMIT_MSG@11 PS1, Line 11: -Swapped out the Reset table and Drop partition SETUP tags to Truncate table QUERY statement. Do we still use SETUP anywhere in the tests? If not, then it would be great to remove it from https://github.com/apache/impala/blob/master/tests/common/impala_test_suite.py#L770 http://gerrit.cloudera.org:8080/#/c/15529/1//COMMIT_MSG@12 PS1, Line 12: -Deleted a test query in ‘insert.test’. The test incorrectly tried to clear out a table which was not working, and should not work either. nit: please wrap commit message lines at 72 chars + I added more comments to the deleted test http://gerrit.cloudera.org:8080/#/c/15529/1/testdata/workloads/functional-query/queries/QueryTest/insert.test File testdata/workloads/functional-query/queries/QueryTest/insert.test: http://gerrit.cloudera.org:8080/#/c/15529/1/testdata/workloads/functional-query/queries/QueryTest/insert.test@a675 PS1, Line 675: : : : : : : : : : : : : : : : Instead of deleting we could also for test the current behavior. Note that it is actually not self evident what to do in this case (INSERT OVERWRITE with dynamic partitioning). Hive doesn't delete all partitions, while Spark does it by default (it has an option to decide: spark.sql.sources.partitionOverwriteMode). -- To view, visit http://gerrit.cloudera.org:8080/15529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I257e936868917a2fcc6c030f6c855b247e8a0eea Gerrit-Change-Number: 15529 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 23 Mar 2020 20:39:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 9: I'm not planning to take a look unless you think the extra eyes are needed. -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 20:48:31 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5526/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 19:38:14 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 23 Mar 2020 19:38:13 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 3: Last verification job failed due to IMPALA-9491. -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 20:05:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8533: Impala daemon crash on sort
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15473 ) Change subject: IMPALA-8533: Impala daemon crash on sort .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5522/ -- To view, visit http://gerrit.cloudera.org:8080/15473 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If19303fbf55927c1e1b76b9b22ab354322b21c54 Gerrit-Change-Number: 15473 Gerrit-PatchSet: 5 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 23 Mar 2020 20:28:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Add timestamp to maven logging options.
David Knupp has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15537 Change subject: Add timestamp to maven logging options. .. Add timestamp to maven logging options. We found that using awk to add a timestamp to the maven log can fail if gawk is not installed. It seems better to configure maven to add the timestamp itself. Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e --- M bin/impala-config.sh M bin/mvn-quiet.sh 2 files changed, 5 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/15537/1 -- To view, visit http://gerrit.cloudera.org:8080/15537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e Gerrit-Change-Number: 15537 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Aman Sinha has posted comments on this change. (
http://gerrit.cloudera.org:8080/15462 )
Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive
normal form
..
Patch Set 7:
> Patch Set 7: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5511/
Looking at the archive.zip for this build failure, the failure I see is the
following in TestImpalaShell which seems unrelated to the code changes. However
it doesn't look like PlannerTest was run. Was it supposed to ?
[gw15] FAILED
shell/test_shell_commandline.py::TestImpalaShell::test_socket_opening[table_format_and_file_extension:
('parquet', '.parq') | protocol: beeswax]
--
To view, visit http://gerrit.cloudera.org:8080/15462
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072
Gerrit-Change-Number: 15462
Gerrit-PatchSet: 7
Gerrit-Owner: Aman Sinha
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 23 Mar 2020 20:59:42 +
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15454 ) Change subject: IMPALA-3766: optionally compress spilled data .. Patch Set 9: (5 comments) went through this briefly, few comments http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/bufferpool/buffer-pool-counters.h File be/src/runtime/bufferpool/buffer-pool-counters.h: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/bufferpool/buffer-pool-counters.h@59 PS9, Line 59: RuntimeProfile::Counter* uncompressed_bytes_written; why add this here vs the TmpFileMgr? http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr-internal.h File be/src/runtime/tmp-file-mgr-internal.h: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr-internal.h@96 PS9, Line 96: int64_t bytes_allocated_ = 0; would it be clearer to rename this to 'file_write_offset_' http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.h File be/src/runtime/tmp-file-mgr.h: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.h@157 PS9, Line 157: static constexpr int64_t HOLE_PUNCH_BLOCK_SIZE = 4096; nit: docs on what this is used for and why it is set to 4K http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.cc File be/src/runtime/tmp-file-mgr.cc: http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.cc@61 PS9, Line 61: "most cases this should be used in conjunction with --disk_spill_punch_holes=true " : "to maximize the space saved."); should we just enforce this? e.g. compression can only be used if hole punching is support on the underlying fs. if users complain and start asking for spill compression support on a fs without hole punching, we can always remove the restriction later. i think the benefit is that is simplifies the combination of configurations Impala needs to support http://gerrit.cloudera.org:8080/#/c/15454/9/be/src/runtime/tmp-file-mgr.cc@63 PS9, Line 63: disk_spill_compression_buffer_limit_bytes what happens if compressing a buffer requires more than this many bytes? -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:01:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9428 Add arm64 atomic ops
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15300 ) Change subject: IMPALA-9428 Add arm64 atomic ops .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5523/ -- To view, visit http://gerrit.cloudera.org:8080/15300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I469e0169193ad6ad8acca2a800c8b3f043083ddd Gerrit-Change-Number: 15300 Gerrit-PatchSet: 9 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 21:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15363 ) Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:13:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Bump up CDP BUILD NUMBER to 2244454
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Bump up CDP_BUILD_NUMBER to 2244454 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5572/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 21:30:57 + Gerrit-HasComments: No
[Impala-ASF-CR] Add timestamp to maven logging options.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15537 ) Change subject: Add timestamp to maven logging options. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5573/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e Gerrit-Change-Number: 15537 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 23 Mar 2020 21:38:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688
Fang-Yu Rao has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 .. IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 This patch bumps up CDP_BUILD_NUMBER to 2244454 which contains a change introduced by RANGER-2688. Due to this change, we added to ranger-admin-site.xml.template a cookie-related configuration so that the Ranger server could be properly started. Testing: Verified that the data loading passes and that all the Ranger-related FE and E2E tests are successful - when $USE_CDP_HIVE is false, and - when $USE_CDP_HIVE is true. Change-Id: I7750f73834368c7109965e78b147238fc6316f49 --- M bin/impala-config.sh M testdata/cluster/ranger/ranger-admin-site.xml.template 2 files changed, 13 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/15533/2 -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 .. Patch Set 2: Code-Review+2 LGTM For anyone else watching, yes the CDP GBN bump will break other tests, but I've triaged all the failures already. -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 21:47:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 8 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:49:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5528/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 23 Mar 2020 21:49:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5527/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 8 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:49:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 7: Agree this looks like a flaky test. I filed IMPALA-9547 to track. -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 7 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 21:49:30 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Fix test_fuzz_nested_types test_fuzz_nested_types had queries that didn't parse successfully because they had duplicated names in an inline view: select count(*) from ( select ... a.pos, ... b.pos from ... ); We have 'a.pos' and 'b.pos' here but Impala still considers those as duplicated names, hence I added aliases for the duplicated fields, e.g.: a.pos as apos Now that the queries pase successfully they start the scanners on corrupted files which is the goal of this test. Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Reviewed-on: http://gerrit.cloudera.org:8080/15528 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_scanners_fuzz.py 1 file changed, 6 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Mar 2020 04:44:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Hello Sahil Takiar, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15454 to look at the new patch set (#10). Change subject: IMPALA-3766: optionally compress spilled data .. IMPALA-3766: optionally compress spilled data Enabled via --disk_spill_compression_codec, which uses the same syntax as the compression_codec query option. Recommended codecs are LZ4 and ZSTD. ZSTD supports specifying a compression level. The compression is done in TmpFileMgr using a temporary buffer. Allocation of disk space is reworked slightly so that the allocation can happen after compression. The default power-of-two disk block sizes would lead to a lot of internal fragmentation, so a new strategy for free space management, similar to that used in the data cache, can be used with --disk_spill_punch_holes=true. TmpFileMgr will allocate a range of the actual compressed size and punch holes in the file for each range that is no longer needed. UncompressedWriteIoBytes is added to the buffer pool profiles, so that you can see what degree of compression is achieved. Typically I saw ratios of 2-3x for LZ4 and ZSTD (with LZ4 toward the lower end and ZSTD toward the higher end). TODO: * finalize the observability Limitations: The management of the compression buffer memory could be improved. Ideally it would be integrated with the buffer pool and use the buffer pool allocator instead of being done "on the side". We would probably want to do this before making this the default, for resource management and performance reasons (doing a malloc() directly does not use the caching supported by the buffer pool). Testing: * Run buffer pool spilling tests with different combinations of the new options. * Extend existing TmpFileMgr tests for file space allocation to run with hole punching enabled. * Switch a couple of spilling tests to use the new option. * Add a metrics test to check for scratch leaks. * Enable the new options by default for end-to-end dockerized tests to get additional coverage. * Add a unit test where allocating compression memory fails, both on the read and write path. * Ran a single-node stress test on TPC-DS SF 1 and TPC-H SF 10 The peak compression buffer usage was ~40MB. Perf: I ran this spilling query using an SSD as the scratch disk: set mem_limit=200m; select count(distinct l_partkey) from tpch30_parquet.lineitem; The time taken for the second run of each query was: No compression: 19.59s LZ4: 18.56s ZSTD: 20.59s Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 --- M be/src/runtime/bufferpool/buffer-pool-counters.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/service/query-options.cc M be/src/util/parse-util.cc M be/src/util/parse-util.h M bin/jenkins/dockerized-impala-run-tests.sh M tests/custom_cluster/test_scratch_disk.py M tests/verifiers/metric_verifier.py 15 files changed, 770 insertions(+), 220 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15454/10 -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/15454 )
Change subject: IMPALA-3766: optionally compress spilled data
..
Patch Set 10:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/bufferpool/buffer-pool-test.cc
File be/src/runtime/bufferpool/buffer-pool-test.cc:
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/bufferpool/buffer-pool-test.cc@1628
PS10, Line 1628: void BufferPoolTest::TestTmpFileAllocateError(const string&
compression, bool punch_holes) {
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr-test.cc
File be/src/runtime/tmp-file-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr-test.cc@531
PS10, Line 531: int64_t expected_bytes_allocated = punch_holes ? 0 :
expected_scratch_bytes_allocated;
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr-test.cc@750
PS10, Line 750: ASSERT_OK(tmp_file_mgr.InitCustom(tmp_dir_specs, false, "",
punch_holes, metrics_.get()));
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr-test.cc@998
PS10, Line 998: file_group.Read(uncompressed_handle.get(),
MemRange(big_tmp.data(), big_tmp.size(;
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr.cc
File be/src/runtime/tmp-file-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/15454/10/be/src/runtime/tmp-file-mgr.cc@788
PS10, Line 788: VLOG(3) << "Write " << tmp_file->path() << " " << file_offset
<< " " << buffer_to_write.len();
line too long (96 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/15454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659
Gerrit-Change-Number: 15454
Gerrit-PatchSet: 10
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 23 Mar 2020 23:50:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9107: Add timestamp to maven logging options.
David Knupp has posted comments on this change. (
http://gerrit.cloudera.org:8080/15537 )
Change subject: IMPALA-9107: Add timestamp to maven logging options.
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/15537/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15537/1//COMMIT_MSG@7
PS1, Line 7: IMPALA-9107: Add timestamp to maven log
> I think we can re-use IMPALA-9107 as the jira.
Done
http://gerrit.cloudera.org:8080/#/c/15537/1//COMMIT_MSG@9
PS1, Line 9: We found that using awk to add a timestamp to the maven log can
fail
: if gawk is not installed. It seems better to configure maven to add
: the timestamp itself.
> Please add a small snippet of what the maven output looks like with a times
Done
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh@328
PS1, Line 328: export IMPALA_MAVEN_OPTIONS=${IMPALA_MAVEN_OPTIONS-}
:
: # If enabled, debug symbols are added to cross-comp
> We set IMPALA_MAVEN_OPTIONS in the environment to allow different settings
Maybe it's better to just to this at the same place in mvn-quiet.sh?
LOGGING_OPTIONS = -Dorg.slf4j.simpleLogger.showDateTime
DATETIME_FORMAT = -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss
if ! mvn -B $IMPALA_MAVEN_OPTIONS $LOGGING_OPTIONS $LOGGING_OPTIONS "$@" | \
tee -a "$LOG_FILE" | grep -E -e WARNING -e ERROR -e SUCCESS -e FAILURE -e
Test; then
echo "mvn $IMPALA_MAVEN_OPTIONS $@ exited with code $?"
exit 1
fi
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/mvn-quiet.sh
File bin/mvn-quiet.sh:
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/mvn-quiet.sh@35
PS1, Line 35: DATETIME_FORMAT = -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss
> Keep the -B
Doh. Sorry. Fixed.
--
To view, visit http://gerrit.cloudera.org:8080/15537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e
Gerrit-Change-Number: 15537
Gerrit-PatchSet: 2
Gerrit-Owner: David Knupp
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Comment-Date: Tue, 24 Mar 2020 00:27:23 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15297 ) Change subject: IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5533/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 Gerrit-Change-Number: 15297 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 02:48:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15297 ) Change subject: IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain .. Patch Set 6: Code-Review+2 Thank you for addressing that! -- To view, visit http://gerrit.cloudera.org:8080/15297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 Gerrit-Change-Number: 15297 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 02:48:21 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-9538 Bump up linux-syscall-support.h
zhaoren...@hotmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15510 Change subject: [WIP]IMPALA-9538 Bump up linux-syscall-support.h .. [WIP]IMPALA-9538 Bump up linux-syscall-support.h Bump up linux-syscall-support.h to newest version which support aarch64 Change-Id: I6c46acb17f048890a3f93fc6b910b2df3c1a7058 --- M be/src/gutil/linux_syscall_support.h M be/src/gutil/spinlock_linux-inl.h M be/src/kudu/util/debug-util.cc 3 files changed, 1,746 insertions(+), 891 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/15510/8 -- To view, visit http://gerrit.cloudera.org:8080/15510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6c46acb17f048890a3f93fc6b910b2df3c1a7058 Gerrit-Change-Number: 15510 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15454 ) Change subject: IMPALA-3766: optionally compress spilled data .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5580/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 03:25:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9547: retry accept in test shell commandline
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15541 ) Change subject: IMPALA-9547: retry accept in test_shell_commandline .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5583/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icc9cab98b059042855ca9149427d079951471be0 Gerrit-Change-Number: 15541 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Mar 2020 04:52:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 9: I have a fix for the flakiness thankfully, so we'll get there eventually :) -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 9 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 05:57:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9547: retry accept in test shell commandline
Hello Abhishek Rawat, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15541 to look at the new patch set (#2). Change subject: IMPALA-9547: retry accept in test_shell_commandline .. IMPALA-9547: retry accept in test_shell_commandline This is a point solution to this particular socket.accept() call failing. The more general problem is described in https://www.python.org/dev/peps/pep-0475/ and fixed in Python 3.5. Change-Id: Icc9cab98b059042855ca9149427d079951471be0 --- M tests/shell/test_shell_commandline.py 1 file changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/15541/2 -- To view, visit http://gerrit.cloudera.org:8080/15541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icc9cab98b059042855ca9149427d079951471be0 Gerrit-Change-Number: 15541 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9183: Convert disjunctive predicates to conjunctive normal form
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/15462 ) Change subject: IMPALA-9183: Convert disjunctive predicates to conjunctive normal form .. Patch Set 9: > Patch Set 8: > > Same issue again, I'm not sure what suddenly is triggering it as it seems > unrelated to your patch. Sigh. let's see if third time is a charm. -- To view, visit http://gerrit.cloudera.org:8080/15462 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5a03cd7239333aaf375416ef5f2b7608fcd4a072 Gerrit-Change-Number: 15462 Gerrit-PatchSet: 9 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 05:33:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5904: (part 3) Fix more TSAN bugs
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15363 ) Change subject: IMPALA-5904: (part 3) Fix more TSAN bugs .. Patch Set 4: Code-Review+2 There was a change several years ago to *not* send non-materialized slots to the backend. I wonder if this logic was actually a holdover from before that change. -- To view, visit http://gerrit.cloudera.org:8080/15363 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I99b7b119e256085d1ba6977e1161fc658273b242 Gerrit-Change-Number: 15363 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 23 Mar 2020 22:48:27 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5530/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 23:45:46 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 23 Mar 2020 23:45:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15533 ) Change subject: IMPALA-9546: Update ranger-admin-site.xml.template after RANGER-2688 .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5528/ -- To view, visit http://gerrit.cloudera.org:8080/15533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7750f73834368c7109965e78b147238fc6316f49 Gerrit-Change-Number: 15533 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Tue, 24 Mar 2020 02:44:47 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP]IMPALA-9538 Bump up linux-syscall-support.h
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15510 ) Change subject: [WIP]IMPALA-9538 Bump up linux-syscall-support.h .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5534/ -- To view, visit http://gerrit.cloudera.org:8080/15510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c46acb17f048890a3f93fc6b910b2df3c1a7058 Gerrit-Change-Number: 15510 Gerrit-PatchSet: 8 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 24 Mar 2020 03:32:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9107: Add timestamp to maven logging options.
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/15537 )
Change subject: IMPALA-9107: Add timestamp to maven logging options.
..
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/15537/1/bin/impala-config.sh@328
PS1, Line 328: export IMPALA_MAVEN_OPTIONS=${IMPALA_MAVEN_OPTIONS-}
:
: # If enabled, debug symbols are added to cross-comp
> Maybe it's better to just to this at the same place in mvn-quiet.sh?
That makes sense to me. I think that is the right place for this setting.
http://gerrit.cloudera.org:8080/#/c/15537/2/bin/mvn-quiet.sh
File bin/mvn-quiet.sh:
http://gerrit.cloudera.org:8080/#/c/15537/2/bin/mvn-quiet.sh@34
PS2, Line 34: LOGGING_OPTIONS = -Dorg.slf4j.simpleLogger.showDateTime
: DATETIME_FORMAT = -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss
You'll need to get rid of the space between the equals and the value. Might as
well add quotes. (See LOG_FILE above)
As a small style nit, I think I would prefer not to have two separate variables
in the command below. I think of these as a single setting to enable
timestamps, so it'd be great if it was one variable. This is a time when going
beyond 90 characters is not so terrible. You can also wrap the line like this:
LOGGING_OPTIONS="-Dorg.slf4j.simpleLogger.showDateTime \
-Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss"
--
To view, visit http://gerrit.cloudera.org:8080/15537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e
Gerrit-Change-Number: 15537
Gerrit-PatchSet: 2
Gerrit-Owner: David Knupp
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Comment-Date: Tue, 24 Mar 2020 04:22:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9107: Add timestamp to maven logging options.
David Knupp has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/15537 ) Change subject: IMPALA-9107: Add timestamp to maven logging options. .. IMPALA-9107: Add timestamp to maven logging options. We found that using awk to add a timestamp to the maven log can fail if gawk is not installed. It seems better to configure maven to add the timestamp itself. Running mvn -U -Dorg.slf4j.simpleLogger.showDateTime=true -Dorg.slf4j.simpleLogger.dateTimeFormat=HH:mm:ss -B install -DskipTests Directory /home/dknupp/Impala/ext-data-source 16:37:16 [INFO] Scanning for projects... 16:37:16 [INFO] 16:37:16 [INFO] Reactor Build Order: 16:37:16 [INFO] 16:37:16 [INFO] Apache Impala External Data Source [pom] 16:37:16 [INFO] Apache Impala External Data Source API [jar] 16:37:16 [INFO] Apache Impala External Data Source Sample [jar] 16:37:16 [INFO] Apache Impala External Data Source Test Library [jar] 16:37:17 [INFO] 16:37:17 [INFO] < org.apache.impala:impala-data-source > 16:37:17 [INFO] Building Apache Impala External Data Source 1.0-SNAPSHOT [1/4] 16:37:17 [INFO] [ pom ]- [etc...] Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e --- M bin/mvn-quiet.sh 1 file changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/15537/2 -- To view, visit http://gerrit.cloudera.org:8080/15537 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I10fbe9eb76b66e6ba00db9f95c91063410dd1b4e Gerrit-Change-Number: 15537 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] Fix test fuzz nested types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15528 ) Change subject: Fix test_fuzz_nested_types .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15528 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8e4cc231f38fe84ebea982b82021458621fe992f Gerrit-Change-Number: 15528 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 24 Mar 2020 00:34:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9373: more tactical IWYU fixes
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15248 ) Change subject: IMPALA-9373: more tactical IWYU fixes .. Patch Set 8: (3 comments) This makes sense to me. Thanks for putting this together! I have a couple minor nits, but it would be good to get this in. http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/benchmarks/overflow-benchmark.cc File be/src/benchmarks/overflow-benchmark.cc: http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/benchmarks/overflow-benchmark.cc@27 PS8, Line 27: #include "util/decimal-util.h" Nit: Not just this file: If we are going true IWYU, then we would also want util/decimal-constants.h wherever we use these constants. It doesn't matter from a compilation perspective, but if we end up doing IWYU as a precommit, then we may end up needing it. http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/runtime/sorter.h File be/src/runtime/sorter.h: http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/runtime/sorter.h@30 PS8, Line 30: class RuntimeProfile; Nit: I think this is no longer needed. http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/util/bit-util-test.cc File be/src/util/bit-util-test.cc: http://gerrit.cloudera.org:8080/#/c/15248/8/be/src/util/bit-util-test.cc@39 PS8, Line 39: ArithmeticUtil Nit: From an IWYU perspective, this would imply we need util/arithmetic-util.h. -- To view, visit http://gerrit.cloudera.org:8080/15248 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8de71866bdf3211e53560d9bfe930e7657c4d7f1 Gerrit-Change-Number: 15248 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 24 Mar 2020 01:16:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain
zhaoren...@hotmail.com has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/15297 ) Change subject: IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain .. IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain Download Python dependencies even skipping bootstrap toolchain. Because when you set SKIP_TOOLCHAIN_BOOTSTRAP=true, the python dependencies still need to be downloaded. The toolchain building process will not download the python dependencies autometically Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 --- M bin/impala-config.sh M buildall.sh 2 files changed, 10 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/15297/6 -- To view, visit http://gerrit.cloudera.org:8080/15297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 Gerrit-Change-Number: 15297 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3766: optionally compress spilled data
Hello Sahil Takiar, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15454 to look at the new patch set (#11). Change subject: IMPALA-3766: optionally compress spilled data .. IMPALA-3766: optionally compress spilled data Enabled via --disk_spill_compression_codec, which uses the same syntax as the compression_codec query option. Recommended codecs are LZ4 and ZSTD. ZSTD supports specifying a compression level. The compression is done in TmpFileMgr using a temporary buffer. Allocation of disk space is reworked slightly so that the allocation can happen after compression. The default power-of-two disk block sizes would lead to a lot of internal fragmentation, so a new strategy for free space management, similar to that used in the data cache, can be used with --disk_spill_punch_holes=true. TmpFileMgr will allocate a range of the actual compressed size and punch holes in the file for each range that is no longer needed. UncompressedWriteIoBytes is added to the buffer pool profiles, so that you can see what degree of compression is achieved. Typically I saw ratios of 2-3x for LZ4 and ZSTD (with LZ4 toward the lower end and ZSTD toward the higher end). TODO: * finalize the observability Limitations: The management of the compression buffer memory could be improved. Ideally it would be integrated with the buffer pool and use the buffer pool allocator instead of being done "on the side". We would probably want to do this before making this the default, for resource management and performance reasons (doing a malloc() directly does not use the caching supported by the buffer pool). Testing: * Run buffer pool spilling tests with different combinations of the new options. * Extend existing TmpFileMgr tests for file space allocation to run with hole punching enabled. * Switch a couple of spilling tests to use the new option. * Add a metrics test to check for scratch leaks. * Enable the new options by default for end-to-end dockerized tests to get additional coverage. * Add a unit test where allocating compression memory fails, both on the read and write path. * Ran a single-node stress test on TPC-DS SF 1 and TPC-H SF 10 The peak compression buffer usage was ~40MB. Perf: I ran this spilling query using an SSD as the scratch disk: set mem_limit=200m; select count(distinct l_partkey) from tpch30_parquet.lineitem; The time taken for the second run of each query was: No compression: 19.59s LZ4: 18.56s ZSTD: 20.59s Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 --- M be/src/runtime/bufferpool/buffer-pool-counters.h M be/src/runtime/bufferpool/buffer-pool-test.cc M be/src/runtime/bufferpool/buffer-pool.cc M be/src/runtime/test-env.cc M be/src/runtime/test-env.h M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/service/query-options.cc M be/src/util/parse-util.cc M be/src/util/parse-util.h M bin/jenkins/dockerized-impala-run-tests.sh M tests/custom_cluster/test_scratch_disk.py M tests/verifiers/metric_verifier.py 15 files changed, 778 insertions(+), 224 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/15454/11 -- To view, visit http://gerrit.cloudera.org:8080/15454 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c08ff9504097f0fee8c32316c5c150136abe659 Gerrit-Change-Number: 15454 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15297 ) Change subject: IMPALA-9426 Download Python dependencies even skipping bootstrap toolchain .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5579/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I012314793ffb521001951ab7ec3d7a3ba737c405 Gerrit-Change-Number: 15297 Gerrit-PatchSet: 6 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 24 Mar 2020 02:39:11 + Gerrit-HasComments: No
