[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9208 ) Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH query .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Gerrit-Change-Number: 9208 Gerrit-PatchSet: 2 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Sat, 03 Feb 2018 05:22:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9208 ) Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH query .. IMPALA-4924 addendum: Change result type to decimal in a TPCH query Change the expected result type of Kudu TPCH Q17 to Decimal because DECIMAL_V2 is now enabled by default. This was not done earlier because we were not running TPCH on Kudu regularly. Cherry-picks: not for 2.x Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Reviewed-on: http://gerrit.cloudera.org:8080/9208 Reviewed-by: Taras BobrovytskyTested-by: Impala Public Jenkins --- M testdata/workloads/tpch/queries/tpch-kudu-q17.test 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Taras Bobrovytsky: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Gerrit-Change-Number: 9208 Gerrit-PatchSet: 3 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-6476: Skip krpc mem test that uses stress options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9207 ) Change subject: IMPALA-6476: Skip krpc mem test that uses stress options .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89ae25ee8c1aca3833c2d98e902ddaad2dd01aad Gerrit-Change-Number: 9207 Gerrit-PatchSet: 1 Gerrit-Owner: Lars VolkerGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 03 Feb 2018 05:10:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6476: Skip krpc mem test that uses stress options
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9207 ) Change subject: IMPALA-6476: Skip krpc mem test that uses stress options .. IMPALA-6476: Skip krpc mem test that uses stress options Some of the tests added in IMPALA-6193 rely on flags that are only compiled for debug binaries. This change marks those tests as debug-only so that they do not break the release tests. Change-Id: I89ae25ee8c1aca3833c2d98e902ddaad2dd01aad Reviewed-on: http://gerrit.cloudera.org:8080/9207 Reviewed-by: Alex BehmTested-by: Impala Public Jenkins --- M tests/custom_cluster/test_krpc_mem_usage.py 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Alex Behm: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I89ae25ee8c1aca3833c2d98e902ddaad2dd01aad Gerrit-Change-Number: 9207 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9206 ) Change subject: IMPALA-6475: Enable running TPCH on Kudu .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1870/ -- To view, visit http://gerrit.cloudera.org:8080/9206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c Gerrit-Change-Number: 9206 Gerrit-PatchSet: 1 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 03 Feb 2018 04:40:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/9191 ) Change subject: IMPALA-5269: Fix issue with final line of query followed by a comment .. IMPALA-5269: Fix issue with final line of query followed by a comment The patch is to remove any comments in a statement when checking if a statement ends with a semicolon delimiter. For example: Before (semicolon delimiter is needed at the end): select 1 + 1; -- comment\n; After (semicolon delimiter is no longer needed): select 1 + 1; -- comment Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258 --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 29 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/9191/4 -- To view, visit http://gerrit.cloudera.org:8080/9191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258 Gerrit-Change-Number: 9191 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy WijayaGerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/9206 ) Change subject: IMPALA-6475: Enable running TPCH on Kudu .. Patch Set 1: Code-Review+2 Yeah, this is basically the solution path I had started to go down when Alex had chatted with me about this. -- To view, visit http://gerrit.cloudera.org:8080/9206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c Gerrit-Change-Number: 9206 Gerrit-PatchSet: 1 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 03 Feb 2018 03:57:34 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9198 ) Change subject: IMPALA-3916: Reserve SQL:2016 reserved words .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1875/ -- To view, visit http://gerrit.cloudera.org:8080/9198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8 Gerrit-Change-Number: 9198 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Sat, 03 Feb 2018 03:49:52 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/9198 ) Change subject: IMPALA-3916: Reserve SQL:2016 reserved words .. Patch Set 2: oops the regex_escape change is already in this branch. -- To view, visit http://gerrit.cloudera.org:8080/9198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8 Gerrit-Change-Number: 9198 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Sat, 03 Feb 2018 03:49:28 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9198 ) Change subject: IMPALA-3916: Reserve SQL:2016 reserved words .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1869/ -- To view, visit http://gerrit.cloudera.org:8080/9198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8 Gerrit-Change-Number: 9198 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Sat, 03 Feb 2018 03:33:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. IMPALA-6113: Skip row groups with predicates on NULL columns Based on the existing Parquet column chunk level statistics null_count, Impala's Parquet scanner is enhanced to skip an entire row group if the null_count statistics indicate that all the values under the predicated column are NULL as we wouldn't get any result rows from that row group anyway. Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Reviewed-on: http://gerrit.cloudera.org:8080/9140 Reviewed-by: Tim ArmstrongTested-by: Impala Public Jenkins --- M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/parquet-column-stats.cc M be/src/exec/parquet-column-stats.h M testdata/workloads/functional-query/queries/QueryTest/parquet-stats.test 4 files changed, 59 insertions(+), 0 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Sat, 03 Feb 2018 03:24:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9153 ) Change subject: IMPALA-5293: Turn insert clustering on by default .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1874/ -- To view, visit http://gerrit.cloudera.org:8080/9153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9 Gerrit-Change-Number: 9153 Gerrit-PatchSet: 11 Gerrit-Owner: Lars VolkerGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 03 Feb 2018 02:11:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/9153 ) Change subject: IMPALA-5293: Turn insert clustering on by default .. Patch Set 11: Code-Review+2 Hit IMPALA-6472, rebased, carrying Alex's +2. -- To view, visit http://gerrit.cloudera.org:8080/9153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9 Gerrit-Change-Number: 9153 Gerrit-PatchSet: 11 Gerrit-Owner: Lars VolkerGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 03 Feb 2018 02:10:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9208 ) Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH query .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1873/ -- To view, visit http://gerrit.cloudera.org:8080/9208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Gerrit-Change-Number: 9208 Gerrit-PatchSet: 2 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Sat, 03 Feb 2018 01:42:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query
Taras Bobrovytsky has posted comments on this change. ( http://gerrit.cloudera.org:8080/9208 ) Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH query .. Patch Set 2: Code-Review+2 For some reason Change-Id was placed before "Cherry-picks" in the commit message. Fixed. Forwarding the +2 -- To view, visit http://gerrit.cloudera.org:8080/9208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Gerrit-Change-Number: 9208 Gerrit-PatchSet: 2 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Sat, 03 Feb 2018 01:42:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9153 ) Change subject: IMPALA-5293: Turn insert clustering on by default .. Patch Set 10: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1866/ -- To view, visit http://gerrit.cloudera.org:8080/9153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9 Gerrit-Change-Number: 9153 Gerrit-PatchSet: 10 Gerrit-Owner: Lars VolkerGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 03 Feb 2018 01:41:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query
Taras Bobrovytsky has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/9208 ) Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH query .. IMPALA-4924 addendum: Change result type to decimal in a TPCH query Change the expected result type of Kudu TPCH Q17 to Decimal because DECIMAL_V2 is now enabled by default. This was not done earlier because we were not running TPCH on Kudu regularly. Cherry-picks: not for 2.x Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 --- M testdata/workloads/tpch/queries/tpch-kudu-q17.test 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/9208/2 -- To view, visit http://gerrit.cloudera.org:8080/9208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Gerrit-Change-Number: 9208 Gerrit-PatchSet: 2 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9208 ) Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH query .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1872/ -- To view, visit http://gerrit.cloudera.org:8080/9208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Gerrit-Change-Number: 9208 Gerrit-PatchSet: 1 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 03 Feb 2018 01:35:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9208 ) Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH query .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Gerrit-Change-Number: 9208 Gerrit-PatchSet: 1 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Comment-Date: Sat, 03 Feb 2018 01:34:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6476: Skip krpc mem test that uses stress options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9207 ) Change subject: IMPALA-6476: Skip krpc mem test that uses stress options .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1871/ -- To view, visit http://gerrit.cloudera.org:8080/9207 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89ae25ee8c1aca3833c2d98e902ddaad2dd01aad Gerrit-Change-Number: 9207 Gerrit-PatchSet: 1 Gerrit-Owner: Lars VolkerGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 03 Feb 2018 01:32:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4924 addendum: Change result type to decimal in a TPCH query
Taras Bobrovytsky has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9208 Change subject: IMPALA-4924 addendum: Change result type to decimal in a TPCH query .. IMPALA-4924 addendum: Change result type to decimal in a TPCH query Change the expected result type of Kudu TPCH Q17 to Decimal because DECIMAL_V2 is now enabled by default. This was not done earlier because we were not running TPCH on Kudu regularly. Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Cherry-picks: not for 2.x --- M testdata/workloads/tpch/queries/tpch-kudu-q17.test 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/9208/1 -- To view, visit http://gerrit.cloudera.org:8080/9208 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I46fc038d40969547622707ce77a037494f0ed0a9 Gerrit-Change-Number: 9208 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-6448: Re-enable kerberized testing with KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9164 ) Change subject: IMPALA-6448: Re-enable kerberized testing with KRPC .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1865/ -- To view, visit http://gerrit.cloudera.org:8080/9164 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18 Gerrit-Change-Number: 9164 Gerrit-PatchSet: 3 Gerrit-Owner: Sailesh MukilGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Sat, 03 Feb 2018 01:21:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9206 ) Change subject: IMPALA-6475: Enable running TPCH on Kudu .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1870/ -- To view, visit http://gerrit.cloudera.org:8080/9206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c Gerrit-Change-Number: 9206 Gerrit-PatchSet: 1 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 03 Feb 2018 00:53:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9206 ) Change subject: IMPALA-6475: Enable running TPCH on Kudu .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c Gerrit-Change-Number: 9206 Gerrit-PatchSet: 1 Gerrit-Owner: Taras BobrovytskyGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 03 Feb 2018 00:50:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/9191 ) Change subject: IMPALA-5269: Fix issue with final line of query followed by a comment .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/9191/3/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/9191/3/shell/impala_shell.py@394 PS3, Line 394: statements = sqlparse.parse(line) I'm concerned that this is a fairly heavy-weight operation to do for every query. Have you done any performance tests to see how much time this takes, particularly with a large query (eg. something from tpcds)? Maybe sqlparse.format(line, strip_comments=True) would be work? I would expect it to be faster (you can check this) and it would also save you having to do the logic below where you check tokens. -- To view, visit http://gerrit.cloudera.org:8080/9191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258 Gerrit-Change-Number: 9191 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy WijayaGerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 03 Feb 2018 00:50:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6475: Enable running TPCH on Kudu
Taras Bobrovytsky has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9206 Change subject: IMPALA-6475: Enable running TPCH on Kudu .. IMPALA-6475: Enable running TPCH on Kudu Change-Id: I88b66f5db105694b3bcf33360887265996f9059c --- M testdata/workloads/tpch/tpch_core.csv M testdata/workloads/tpch/tpch_exhaustive.csv M testdata/workloads/tpch/tpch_pairwise.csv 3 files changed, 3 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/9206/1 -- To view, visit http://gerrit.cloudera.org:8080/9206 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I88b66f5db105694b3bcf33360887265996f9059c Gerrit-Change-Number: 9206 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8971 ) Change subject: IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/8971/9/be/src/service/query-options.cc File be/src/service/query-options.cc: http://gerrit.cloudera.org:8080/#/c/8971/9/be/src/service/query-options.cc@376 PS9, Line 376: // last condition is to unblock the highly improbable case where the : // min_buffer_size is greater than RuntimeFilterBank::MAX_BLOOM_FILTER_SIZE. : && FLAGS_min_buffer_size <= RuntimeFilterBank::MAX_BLOOM_FILTER_SIZE pretty ugly workaround, but had to put this so it doesn't block the query for this highly improbable case. Any comments? -- To view, visit http://gerrit.cloudera.org:8080/8971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f Gerrit-Change-Number: 8971 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet VigGerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 03 Feb 2018 00:21:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8971 ) Change subject: IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool .. Patch Set 8: (5 comments) > (5 comments) > > Did you look at the tests that Alex mentioned for creating a test > table with different cardinalities and nvds? initially we discussed including those tests to check if the limits are being enforced in case the size calculated from ndvs is outside those limits. It turns out that we infact are already testing that code path when we check if the query limits are enforced (in bloom_filters.test). Also, since we are taking care of enforcing the hard limits({MIN,MAX}_BLOOM_FILTER_SIZE) on the query options itself, we dont need to write any tests with mock table stats anymore. http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@95 PS8, Line 95: private static final long MIN_BLOOM_FILTER_SIZE = 4 * 1024; : private static final long MAX_BLOOM_FILTER_SIZE = 512 * 1024 * 1024; > You may remove them and add a check instead. Done http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@109 PS8, Line 109: _ > nit: no need for '_' when the fields are public Done http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@125 PS8, Line 125: bloomFilterSizeLimits_.max_ = tQueryOptions.getRuntime_filter_max_size(); : bloomFilterSizeLimits_.max_ = Math.max(bloomFilterSizeLimits_.max_, : MIN_BLOOM_FILTER_SIZE); : bloomFilterSizeLimits_.max_ = Math.max(bloomFilterSizeLimits_.max_, : BackendConfig.INSTANCE.getMinBufferSize()); : bloomFilterSizeLimits_.max_ = Math.min( : BitUtil.roundUpToPowerOf2(bloomFilterSizeLimits_.max_), MAX_BLOOM_FILTER_SIZE); > You may want to see if it makes sense to add this logic to FilterSizeLimits I moved this whole part into the FilterSizeLimits ctor, since i need to retain the order in which these are set. http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@482 PS8, Line 482: sizeLimits > 'filterSizeLimits' Done http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@494 PS8, Line 494: filterSizeBytes_ = Math.min(filterSizeBytes_, filterSizeLimits.max_); > There is always the case that the size needed to achieve the desired fp rat I would not recommend logging it here because although this is a valid case, it still does not guarantee that the filter will be disabled, as the disabling logic uses the actual ndv to check for fpp during query execution. Also, there might be a lot of runtime filters generated here but might not end up in the plan eventually, so the log might be filled with extra messages. -- To view, visit http://gerrit.cloudera.org:8080/8971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f Gerrit-Change-Number: 8971 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet VigGerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 03 Feb 2018 00:18:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool
Hello Dimitris Tsirogiannis, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8971 to look at the new patch set (#9). Change subject: IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool .. IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool This patch adds changes to the planner to account for memory used by bloom filters at the fragment instance level. Also adds changes to allocate memory for those bloom filters from the buffer pool. Testing: - Modified Planner Tests and end to end tests to account for memory reservation for the runtime filters. - Modified backend tests and benchmarks to use the bufferpool for bloom filter allocation. - Add an end to end test. - Ran rest of the core tests. Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f --- M be/src/benchmarks/bloom-filter-benchmark.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-filter.h M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/service/fe-support.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/util/backend-gflag-util.cc M be/src/util/bloom-filter-test.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M common/thrift/BackendGflags.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/PlanNodes.thrift M common/thrift/Planner.thrift M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test M testdata/workloads/functional-query/queries/QueryTest/bloom_filters.test M testdata/workloads/functional-query/queries/QueryTest/bloom_filters_wait.test M testdata/workloads/functional-query/queries/QueryTest/explain-level0.test M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test M testdata/workloads/functional-query/queries/QueryTest/spilling.test 38 files changed, 906 insertions(+), 549 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/8971/9 -- To view, visit http://gerrit.cloudera.org:8080/8971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f Gerrit-Change-Number: 8971 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet VigGerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6228: Control stats extrapolation via tbl prop.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9139 ) Change subject: IMPALA-6228: Control stats extrapolation via tbl prop. .. Patch Set 4: Code-Review+2 Rebase and resolve conflicts -- To view, visit http://gerrit.cloudera.org:8080/9139 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie49597bf1b93b7572106abc620d91f199cba0cfd Gerrit-Change-Number: 9139 Gerrit-PatchSet: 4 Gerrit-Owner: Alex BehmGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 02 Feb 2018 23:57:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6228: Control stats extrapolation via tbl prop.
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9139 to look at the new patch set (#4). Change subject: IMPALA-6228: Control stats extrapolation via tbl prop. .. IMPALA-6228: Control stats extrapolation via tbl prop. Introduces a new TBLPROPERTY for controlling stats extrapolation on a per-table basis: impala.enable.stats.extrapolation=true/false The property key was chosen to be consistent with the impalad startup flag --enable_stats_extrapolation and to indicate that the property was set and is used by Impala. Behavior: - If the property is not set, then the extrapolation behavior is determined by the impalad startup flag. - If the property is set, it overrides the impalad startup flag, i.e., extrapolation can be explicitly enabled or disabled regardless of the startup flag. Testing: - added new unit tests - code/hdfs run passed Change-Id: Ie49597bf1b93b7572106abc620d91f199cba0cfd --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/planner/StatsExtrapolationTest.java M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/custom_cluster/test_stats_extrapolation.py A tests/metadata/test_stats_extrapolation.py 9 files changed, 329 insertions(+), 278 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/9139/4 -- To view, visit http://gerrit.cloudera.org:8080/9139 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie49597bf1b93b7572106abc620d91f199cba0cfd Gerrit-Change-Number: 9139 Gerrit-PatchSet: 4 Gerrit-Owner: Alex BehmGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9198 ) Change subject: IMPALA-3916: Reserve SQL:2016 reserved words .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1869/ -- To view, visit http://gerrit.cloudera.org:8080/9198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8 Gerrit-Change-Number: 9198 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 02 Feb 2018 23:53:18 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/9198 ) Change subject: IMPALA-3916: Reserve SQL:2016 reserved words .. Patch Set 2: Code-Review+2 Carry over Philip's +2 -- To view, visit http://gerrit.cloudera.org:8080/9198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8 Gerrit-Change-Number: 9198 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 02 Feb 2018 23:52:47 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-3916: Change reserved words version to 2.11.0
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9201 ) Change subject: IMPALA-3916: Change reserved_words_version to 2.11.0 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9201 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I5a3e5eac8c09de4517beccf79dcea111264668c2 Gerrit-Change-Number: 9201 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Alex Behm Gerrit-Comment-Date: Fri, 02 Feb 2018 23:51:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/9191 ) Change subject: IMPALA-5269: Fix issue with final line of query followed by a comment .. IMPALA-5269: Fix issue with final line of query followed by a comment The patch is to fix a statement that is terminated with SQL terminated delimiter, i.e. ; followed by a comment. For example: Before: select 1 + 1; -- comment\n; After: select 1 + 1; -- comment Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258 --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 39 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/9191/3 -- To view, visit http://gerrit.cloudera.org:8080/9191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258 Gerrit-Change-Number: 9191 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy WijayaGerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-6337: Fix infinite loop in Impala shell
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/9195 ) Change subject: IMPALA-6337: Fix infinite loop in Impala shell .. IMPALA-6337: Fix infinite loop in Impala shell The patch uses a workaround for a bug in the sqlparse.split() function by joining the statements that contain error tokens into a single statement Change-Id: I9142f21a888189d351f00ce09baeba123bc0959b --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 42 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/9195/2 -- To view, visit http://gerrit.cloudera.org:8080/9195 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9142f21a888189d351f00ce09baeba123bc0959b Gerrit-Change-Number: 9195 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6456: Add flags to configure rpc negotiation timeout ms and negotiation thread count in KRPC
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/9186 ) Change subject: IMPALA-6456: Add flags to configure rpc_negotiation_timeout_ms and negotiation thread count in KRPC .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc File be/src/rpc/rpc-mgr.cc: http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@70 PS1, Line 70: Number of threads to dedicate to process connection negotiations > Maximum number of threads dedicated to handling RPC connection negotiations Done http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@84 PS1, Line 84: bld.set_rpc_negotiation_timeout_ms(FLAGS_rpc_negotiation_timeout_ms); > Does it make sense to also call bld.set_min_negotiation_threads(1) ? The min number of threads is 0 by default. What that means is that there won't be any "permanent" thread that is always waiting to do negotiation. They will be spawned as required (up to max_thread_count) and live for some small period after which it times out and kills itself. Given our workload, we would have bursts of negotiations at different points in time with potentially vast intervals in between. So maybe it's better to save the resources of that one thread. What do you think? http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@84 PS1, Line 84: FLAGS_rpc_negotiation_timeout_ms > What happens if this flag is set to negative by accident ? Will the code cr There's a CHECK to make sure that this isn't negative: https://github.com/apache/impala/blob/master/be/src/kudu/util/threadpool.cc#L79 So it will cause a crash. http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@85 PS1, Line 85: FLAGS_rpc_negotiation_thread_count > Will it be safer to do max(1, FLAGS_rpc_negotiation_thread_count) to guaran Yes that makes sense. Done. -- To view, visit http://gerrit.cloudera.org:8080/9186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I108d700e7eac04b678e21a3a920aac81ba8eede5 Gerrit-Change-Number: 9186 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh MukilGerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 02 Feb 2018 23:46:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/9191 ) Change subject: IMPALA-5269: Fix issue with final line of query followed by a comment .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/9191/2/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/9191/2/shell/impala_shell.py@394 PS2, Line 394: line = line[0: line.index('--')].strip() if '--' in line else line > I don't think that this is sufficient to determine the location of a commen I changed the logic in my 3rd patch to use an SQL parser to remove the last comment token. -- To view, visit http://gerrit.cloudera.org:8080/9191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258 Gerrit-Change-Number: 9191 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy WijayaGerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 02 Feb 2018 23:45:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6456: Add flags to configure rpc negotiation timeout ms and negotiation thread count in KRPC
Hello Michael Ho, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9186 to look at the new patch set (#2). Change subject: IMPALA-6456: Add flags to configure rpc_negotiation_timeout_ms and negotiation thread count in KRPC .. IMPALA-6456: Add flags to configure rpc_negotiation_timeout_ms and negotiation thread count in KRPC With the fix for KUDU-2228, the FLAGS_rpc_negotiation_timeout_ms was retired in KRPC. This patch introduces a flag to be able to configure that from the Impala side (FLAGS_rpc_negotiation_timeout_ms). It also introduces a flag to configure the negotiation thread count (FLAGS_rpc_negotiation_thread_count). Added a test to verify that setting FLAGS_rpc_negotiation_timeout_ms to 0 causes negotiation failures. We unfortunately can't write a test to check the same for FLAGS_rpc_negotiation_thread_count due to DCHECKS present in the code. Change-Id: I108d700e7eac04b678e21a3a920aac81ba8eede5 --- M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr.cc 2 files changed, 31 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/9186/2 -- To view, visit http://gerrit.cloudera.org:8080/9186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I108d700e7eac04b678e21a3a920aac81ba8eede5 Gerrit-Change-Number: 9186 Gerrit-PatchSet: 2 Gerrit-Owner: Sailesh MukilGerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-6449: Use CLOCK MONOTONIC in ConditionVariable
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/9158 ) Change subject: IMPALA-6449: Use CLOCK_MONOTONIC in ConditionVariable .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG@13 PS1, Line 13: CLOCK_MONOTONIC > Thanks for doing the research. I believe CLOCK_MONOTONIC_RAW is only availa Makes sense. Let's just add that to the commit message then, so it's useful for future reference. -- To view, visit http://gerrit.cloudera.org:8080/9158 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81611cfd5e7c5347203fe7fa6b0f615602257f87 Gerrit-Change-Number: 9158 Gerrit-PatchSet: 1 Gerrit-Owner: Michael HoGerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 02 Feb 2018 23:36:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Fri, 02 Feb 2018 23:31:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1868/ -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Fri, 02 Feb 2018 23:31:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6338: Fix flaky test profile fragment instances
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8997 ) Change subject: IMPALA-6338: Fix flaky test_profile_fragment_instances .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8997 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a1e3c62952003f37f88fe2b662bb11889ed Gerrit-Change-Number: 8997 Gerrit-PatchSet: 6 Gerrit-Owner: Thomas Tauber-MarshallGerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 02 Feb 2018 23:22:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6338: Fix flaky test profile fragment instances
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8997 ) Change subject: IMPALA-6338: Fix flaky test_profile_fragment_instances .. IMPALA-6338: Fix flaky test_profile_fragment_instances test_profile_fragment_instances checks that, once all the results have been returned, every fragment instance appears in the query profile for a query that internally cancels fragment instances that are still executing when the results have been fully returned. Every fis is guaranteed to send a profile to the coordinator in Finalize(), but previously fragment profiles were not applied by the coordinator if the backend was 'done', defined as either all instances having completed or one has entered an error state (including cancelled). So, the test could fail by the following sequence: - Some fragment for a particular backend sends an update to the coordinator. 'returned_all_results_' is true, so the coordinator responds indicating the the backend should cancel its remaining fragments. - Another fragment from that backend executes Finalize() and reports that it was cancelled. This causes the coordinator to consider the entire backend to be 'done'. - A third fragment, which had not previously sent a report from the reporting thread, from the same backend executes Finalize(). This report will not be applied by the coordinator as the backend is considered 'done', so this fragment will not appear in the final profile. The solution is to change the definition of 'done' to not include a backend that has been cancelled but still has fragments that haven't completed. This guarantees that for queries that complete successfully and are cancelled internally, all fis will send a report and have it applied by the coordinator before all results have been returned, since if eos is true Coordinator::GetNext() calls WaitForBackendCompletion(), which in this situation will now wait for all fis to Finalize(). Returning results for queries that are cancelled by the user is unaffected as the manual cancel path causes WaitForBackendCompletion(). Testing: - Ran test_profile_fragment_instances in a loop with no failures. I can reliably repro the original problem with a few carefully placed sleeps. Change-Id: I3a1e3c62952003f37f88fe2b662bb11889ed Reviewed-on: http://gerrit.cloudera.org:8080/8997 Reviewed-by: Tim ArmstrongTested-by: Impala Public Jenkins --- M be/src/common/status.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M tests/query_test/test_observability.py 6 files changed, 30 insertions(+), 18 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8997 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3a1e3c62952003f37f88fe2b662bb11889ed Gerrit-Change-Number: 8997 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5990: End-to-end compression of metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8825 ) Change subject: IMPALA-5990: End-to-end compression of metadata .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae Gerrit-Change-Number: 8825 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 02 Feb 2018 23:17:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6449: Use CLOCK MONOTONIC in ConditionVariable
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/9158 ) Change subject: IMPALA-6449: Use CLOCK_MONOTONIC in ConditionVariable .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG@13 PS1, Line 13: CLOCK_MONOTONIC > Reading the man pages, it basically says that even CLOCK_MONOTONIC can be a Thanks for doing the research. I believe CLOCK_MONOTONIC_RAW is only available after Linux kernel 2.6.28 so CLOCK_MONOTONIC seems more widely available. https://www.systutorials.com/docs/linux/man/2-clock_settime/ FWIW, our monotonic stopwatch is also using CLOCK_MONOTONIC. The monotonicity of the clock is the major point here. I agree that CLOCK_MONOTONIC_RAW would be even better but we may need to do more work to detect whether it's available on the platform. -- To view, visit http://gerrit.cloudera.org:8080/9158 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81611cfd5e7c5347203fe7fa6b0f615602257f87 Gerrit-Change-Number: 9158 Gerrit-PatchSet: 1 Gerrit-Owner: Michael HoGerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 02 Feb 2018 23:09:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9187 ) Change subject: IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table .. IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c Reviewed-on: http://gerrit.cloudera.org:8080/9187 Reviewed-by: Thomas Tauber-MarshallTested-by: Impala Public Jenkins --- M docs/topics/impala_alter_table.xml 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9187 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c Gerrit-Change-Number: 9187 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: John Russell Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9187 ) Change subject: IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9187 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c Gerrit-Change-Number: 9187 Gerrit-PatchSet: 1 Gerrit-Owner: Alex RodoniGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: John Russell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 02 Feb 2018 22:37:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9187 ) Change subject: IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/194/ -- To view, visit http://gerrit.cloudera.org:8080/9187 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c Gerrit-Change-Number: 9187 Gerrit-PatchSet: 1 Gerrit-Owner: Alex RodoniGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: John Russell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 02 Feb 2018 22:28:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6396: Exchange node's memory usage should include its receiver's
Michael Ho has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/9202 ) Change subject: IMPALA-6396: Exchange node's memory usage should include its receiver's .. IMPALA-6396: Exchange node's memory usage should include its receiver's A DataStreamRecvr is co-owned by the DataStreamMgr and an Exchange node. However, the life time of the memory allocations (e.g. row batches) of a DataStreamRecvr never exceeds that of its owning Exchange node. Previously, we used the fragment instance's MemTracker as the parent of the DataStreamRecvr's MemTracker. This change switches to using the MemTracker of the owning Exchange node as the parent tracker of the DataStreamRecvr. This makes it easier to identify the peak memory usage of the receivers of different exchange nodes in the runtime profile and query summary. Sample output from TPCH-Q21: EXCHANGE_NODE (id=18):(Total: 1s448ms, non-child: 265.818ms, % non-child: 18.35%) - ConvertRowBatchTime: 223.895ms - PeakMemoryUsage: 10.04 MB (10524943) - RowsReturned: 1.27M (1267464) - RowsReturnedRate: 875.19 K/sec RecvrSide: BytesReceived(500.000ms): 0, 1.64 MB, 9.98 MB, 9.98 MB, 10.01 MB, 10.01 MB, 10.01 MB, 31.79 MB, 60.19 MB, 87.84 MB - FirstBatchArrivalWaitTime: 0.000ns - TotalBytesReceived: 93.07 MB (97594728) - TotalGetBatchTime: 1s194ms - DataArrivalTimer: 1s183ms SenderSide: - DeserializeRowBatchTime: 344.343ms - NumBatchesAccepted: 3.80K (3796) - NumBatchesDeferred: 5 (5) - NumEarlySenders: 0 (0) Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45 --- M be/src/exec/exchange-node.cc M be/src/runtime/data-stream-mgr-base.h M be/src/runtime/data-stream-mgr.cc M be/src/runtime/data-stream-mgr.h M be/src/runtime/data-stream-test.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-mgr.h M tests/query_test/test_observability.py 8 files changed, 35 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/9202/2 -- To view, visit http://gerrit.cloudera.org:8080/9202 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45 Gerrit-Change-Number: 9202 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] IMPALA-6369: Exchange node's memory usage should include its receiver's
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9202 Change subject: IMPALA-6369: Exchange node's memory usage should include its receiver's .. IMPALA-6369: Exchange node's memory usage should include its receiver's A DataStreamRecvr is co-owned by the DataStreamMgr and an Exchange node. However, the life time of the memory allocations (e.g. row batches) of a DataStreamRecvr never exceeds that of its owning Exchange node. Previously, we used the fragment instance's MemTracker as the parent of the DataStreamRecvr's MemTracker. This change switches to using the MemTracker of the owning Exchange node as the parent tracker of the DataStreamRecvr. This makes it easier to identify the peak memory usage of the receivers of different exchange nodes in the runtime profile and query summary. Sample output from TPCH-Q21: EXCHANGE_NODE (id=18):(Total: 1s448ms, non-child: 265.818ms, % non-child: 18.35%) - ConvertRowBatchTime: 223.895ms - PeakMemoryUsage: 10.04 MB (10524943) - RowsReturned: 1.27M (1267464) - RowsReturnedRate: 875.19 K/sec RecvrSide: BytesReceived(500.000ms): 0, 1.64 MB, 9.98 MB, 9.98 MB, 10.01 MB, 10.01 MB, 10.01 MB, 31.79 MB, 60.19 MB, 87.84 MB - FirstBatchArrivalWaitTime: 0.000ns - TotalBytesReceived: 93.07 MB (97594728) - TotalGetBatchTime: 1s194ms - DataArrivalTimer: 1s183ms SenderSide: - DeserializeRowBatchTime: 344.343ms - NumBatchesAccepted: 3.80K (3796) - NumBatchesDeferred: 5 (5) - NumEarlySenders: 0 (0) Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45 --- M be/src/exec/exchange-node.cc M be/src/runtime/data-stream-mgr-base.h M be/src/runtime/data-stream-mgr.cc M be/src/runtime/data-stream-mgr.h M be/src/runtime/data-stream-test.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-mgr.h M tests/query_test/test_observability.py 8 files changed, 35 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/9202/1 -- To view, visit http://gerrit.cloudera.org:8080/9202 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45 Gerrit-Change-Number: 9202 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] IMPALA-6204: Remove external DataSource
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9192 ) Change subject: IMPALA-6204: Remove external DataSource .. Patch Set 3: I think this is now reviewable. I ran core tests successfully with this before rebasing. I'm running them again after rebase (https://jenkins.impala.io/job/gerrit-verify-dryrun-external/71/). There were some conflicts with the keyword change, but nothing interesting. BTW: +95, -4667!!! -- To view, visit http://gerrit.cloudera.org:8080/9192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6 Gerrit-Change-Number: 9192 Gerrit-PatchSet: 3 Gerrit-Owner: Philip ZeyligerGerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 02 Feb 2018 22:24:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6204: Remove external DataSource
Hello Dimitris Tsirogiannis, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9192 to look at the new patch set (#3). Change subject: IMPALA-6204: Remove external DataSource .. IMPALA-6204: Remove external DataSource Removes DataSourceScanNode, external data sources, and all affiliated code, tests, and documentation. When a data source table is encountered, we now throw an exception. To the user, this looks like: [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1'); Query: create table t (x int) stored as textfile tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1') Fetched 0 row(s) in 0.11s [pannier.ca.cloudera.com:21000] > select * from t; Query: select * from t Query submitted at: 2018-02-01 17:16:26 (Coordinator: http://pannier.ca.cloudera.com:25000) ERROR: AnalysisException: Failed to load metadata for table: 't' CAUSED BY: TableLoadingException: Failed to load metadata for table: default.t. Running 'invalidate metadata default.t' may resolve this problem. CAUSED BY: UnsupportedOperationException: Eternal Data source table not supported. A test has been added to capture this behavior. For the most part, I deleted the unused code. In a few places, a renamed the Thrift enums and threw errors if they're encountered. For Thrift structs, I left a comment about the now-skipped id that used to represent a data-source related entry. Cherry-picks: not for 2.x Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6 --- M CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/catalog/catalog-util.cc M be/src/exec/CMakeLists.txt M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h D be/src/exec/data-source-scan-node.cc D be/src/exec/data-source-scan-node.h M be/src/exec/exec-node.cc D be/src/exec/external-data-source-executor.cc D be/src/exec/external-data-source-executor.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M bin/clean-cmake.sh M bin/clean.sh M buildall.sh M common/thrift/CMakeLists.txt M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M common/thrift/Data.thrift M common/thrift/Descriptors.thrift D common/thrift/ExternalDataSource.thrift M common/thrift/Frontend.thrift M common/thrift/JniCatalog.thrift M common/thrift/PlanNodes.thrift M docs/impala.ditamap D docs/topics/impala_create_data_source.xml D docs/topics/impala_data_sources.xml D docs/topics/impala_drop_data_source.xml D ext-data-source/.gitignore D ext-data-source/CMakeLists.txt D ext-data-source/api/pom.xml D ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java D ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java D ext-data-source/pom.xml D ext-data-source/sample/pom.xml D ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java D ext-data-source/test/pom.xml D ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java M fe/CMakeLists.txt M fe/pom.xml M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java D fe/src/main/java/org/apache/impala/catalog/DataSource.java D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java D fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/jflex/sql-scanner.flex M
[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9153 ) Change subject: IMPALA-5293: Turn insert clustering on by default .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1866/ -- To view, visit http://gerrit.cloudera.org:8080/9153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9 Gerrit-Change-Number: 9153 Gerrit-PatchSet: 10 Gerrit-Owner: Lars VolkerGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 02 Feb 2018 22:04:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/9153 ) Change subject: IMPALA-5293: Turn insert clustering on by default .. Patch Set 10: Code-Review+2 PS10 is a final rebase before submitting the change. Carrying Alex's +2. -- To view, visit http://gerrit.cloudera.org:8080/9153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9 Gerrit-Change-Number: 9153 Gerrit-PatchSet: 10 Gerrit-Owner: Lars VolkerGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 02 Feb 2018 22:04:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6449: Use CLOCK MONOTONIC in ConditionVariable
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/9158 ) Change subject: IMPALA-6449: Use CLOCK_MONOTONIC in ConditionVariable .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9158/1//COMMIT_MSG@13 PS1, Line 13: CLOCK_MONOTONIC Reading the man pages, it basically says that even CLOCK_MONOTONIC can be affected by sources like NTP. However, the difference is that with CLOCK_MONOTONIC, the time will not jump backwards, but rather experience "time slew", which means that the frequency of clock ticks can change momentarily to reflect adjustments in NTP. CLOCK_MONOTONIC_RAW on the other hand shows the absolute wall clock time that's not affected by NTP. But it could be affected by things like temperature, environment, etc. that will not be corrected (CLOCK_MONOTONIC would have that corrected apparently) It seems a bit confusing which would be considered "right" while measuring time like we do. I'm okay with both, as long as we've considered both the options. Whatever you think is the right option, I think we can just add to the commit message that we've considered the other one and give a reason as to why we didn't choose that. -- To view, visit http://gerrit.cloudera.org:8080/9158 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81611cfd5e7c5347203fe7fa6b0f615602257f87 Gerrit-Change-Number: 9158 Gerrit-PatchSet: 1 Gerrit-Owner: Michael HoGerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 02 Feb 2018 22:03:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5293: Turn insert clustering on by default
Hello Tim Armstrong, Alex Behm, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9153 to look at the new patch set (#10). Change subject: IMPALA-5293: Turn insert clustering on by default .. IMPALA-5293: Turn insert clustering on by default This change enables clustering by default. IMPALA-2521 introduced the 'clustered' hint which inserts a local sort by the partitioning columns to a query plan. The hint is only effective for HDFS and Kudu tables. Like before, the 'noclustered' hint prevents clustering. If a table has ordering columns defined, the 'noclustered' hint is ignored and we issue a warning. This change removes some tests that were added specifically to test that clustering can be enabled using the 'clustered' hint. It changes some tests to use the 'noclustered' hint to make sure that clustering can be disabled. It also adds tests to make sure that we cover the 'noclustered' case properly. Cherry-picks: not for 2.x. Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9 --- M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/empty.test M testdata/workloads/functional-planner/queries/PlannerTest/insert.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-planner/queries/PlannerTest/order.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/with-clause.test M testdata/workloads/functional-query/queries/QueryTest/insert.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test 12 files changed, 248 insertions(+), 66 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/9153/10 -- To view, visit http://gerrit.cloudera.org:8080/9153 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idbf2368cf4415e6ecfa65058daf6ff87ef62f9d9 Gerrit-Change-Number: 9153 Gerrit-PatchSet: 10 Gerrit-Owner: Lars VolkerGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6456: Add flags to configure rpc negotiation timeout ms and negotiation thread count in KRPC
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/9186 ) Change subject: IMPALA-6456: Add flags to configure rpc_negotiation_timeout_ms and negotiation thread count in KRPC .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc File be/src/rpc/rpc-mgr.cc: http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@70 PS1, Line 70: Number of threads to dedicate to process connection negotiations Maximum number of threads dedicated to handling RPC connection negotiations http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@84 PS1, Line 84: bld.set_rpc_negotiation_timeout_ms(FLAGS_rpc_negotiation_timeout_ms); Does it make sense to also call bld.set_min_negotiation_threads(1) ? http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@84 PS1, Line 84: FLAGS_rpc_negotiation_timeout_ms What happens if this flag is set to negative by accident ? Will the code crash or does it translate to a very large unsigned value ? http://gerrit.cloudera.org:8080/#/c/9186/1/be/src/rpc/rpc-mgr.cc@85 PS1, Line 85: FLAGS_rpc_negotiation_thread_count Will it be safer to do max(1, FLAGS_rpc_negotiation_thread_count) to guarantee there is at least on negotiation thread available ? -- To view, visit http://gerrit.cloudera.org:8080/9186 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I108d700e7eac04b678e21a3a920aac81ba8eede5 Gerrit-Change-Number: 9186 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh MukilGerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Fri, 02 Feb 2018 21:43:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6448: Re-enable kerberized testing with KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9164 ) Change subject: IMPALA-6448: Re-enable kerberized testing with KRPC .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1865/ -- To view, visit http://gerrit.cloudera.org:8080/9164 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18 Gerrit-Change-Number: 9164 Gerrit-PatchSet: 3 Gerrit-Owner: Sailesh MukilGerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 02 Feb 2018 21:34:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6448: Re-enable kerberized testing with KRPC
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/9164 ) Change subject: IMPALA-6448: Re-enable kerberized testing with KRPC .. Patch Set 3: Code-Review+2 (1 comment) Rebase, carry +2. http://gerrit.cloudera.org:8080/#/c/9164/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9164/2//COMMIT_MSG@14 PS2, Line 14: . : This behavior can be found in SaslInit() in kudu/rpc/sasl_common.cc > nit: as SaslInit() which is called once per process will hard code some con Done -- To view, visit http://gerrit.cloudera.org:8080/9164 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18 Gerrit-Change-Number: 9164 Gerrit-PatchSet: 3 Gerrit-Owner: Sailesh MukilGerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 02 Feb 2018 21:34:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6448: Re-enable kerberized testing with KRPC
Hello Michael Ho, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9164 to look at the new patch set (#3). Change subject: IMPALA-6448: Re-enable kerberized testing with KRPC .. IMPALA-6448: Re-enable kerberized testing with KRPC For the patch for IMPALA-5054, we realized that we needed to make the kudu::rpc::Messenger configurable. A patch was done on the Kudu side which is tracked by KUDU-2228. As part of that patch, one of the design decisions taken was to only allow kerberos either on or off for the entirety of the process life. This means that we cannot switch kerberos on and off in the same process any more with KRPC. This behavior can be found in SaslInit() in kudu/rpc/sasl_common.cc as SaslInit() which is called once per process will hard code some configuration which cannot be toggled. This affected our kerberized rpc-mgr-tests. This patch splits out the kerberized part of rpc-mgr-test into rpc-mgr-kerberized-test. It also puts the common code between both the files into rpc-mgr-test-base.h Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18 --- M be/src/rpc/CMakeLists.txt A be/src/rpc/rpc-mgr-kerberized-test.cc A be/src/rpc/rpc-mgr-test-base.h M be/src/rpc/rpc-mgr-test.cc 4 files changed, 380 insertions(+), 316 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/9164/3 -- To view, visit http://gerrit.cloudera.org:8080/9164 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6412978316de90875c98f8fbe51c8d215c227b18 Gerrit-Change-Number: 9164 Gerrit-PatchSet: 3 Gerrit-Owner: Sailesh MukilGerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/9198 ) Change subject: IMPALA-3916: Reserve SQL:2016 reserved words .. IMPALA-3916: Reserve SQL:2016 reserved words This patch reserves SQL:2016 reserved words, excluding: 1. Impala builtin function names. 2. Time unit words(year, month, etc.). 3. An exception list based on a discussion. Some test cases are modified to avoid these words. A impalad and catalogd startup option reserved_words_version is added. The words are reserved if the option is set to "3.0.0". Change-Id: If1b295e6a77e840cf1b794c2eb73e1b9d2b8ddd6 Reviewed-on: http://gerrit.cloudera.org:8080/9096 Reviewed-by: Alex BehmReviewed-by: Philip Zeyliger Tested-by: Impala Public Jenkins (cherry picked from commit f0b3d9d122f2c6eb4137bf93e3512a489ff8fab0) Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8 --- M be/src/common/global-flags.cc M be/src/common/init.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/planner/StatsExtrapolationTest.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M testdata/workloads/functional-query/queries/QueryTest/empty-build-joins.test M testdata/workloads/functional-query/queries/QueryTest/exprs.test M testdata/workloads/functional-query/queries/QueryTest/partition-col-types.test M testdata/workloads/functional-query/queries/QueryTest/single-node-nlj.test M testdata/workloads/functional-query/queries/QueryTest/spilling-large-rows.test M testdata/workloads/functional-query/queries/QueryTest/values.test A tests/custom_cluster/test_reserved_words_version.py M tests/custom_cluster/test_stats_extrapolation.py M tests/query_test/test_sort.py 28 files changed, 570 insertions(+), 442 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/9198/2 -- To view, visit http://gerrit.cloudera.org:8080/9198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8 Gerrit-Change-Number: 9198 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR](2.x) IMPALA-3916: Change reserved words version to 2.11.0
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9201 Change subject: IMPALA-3916: Change reserved_words_version to 2.11.0 .. IMPALA-3916: Change reserved_words_version to 2.11.0 Impala 2.x doesn't reserve any unused words by default. Change-Id: I5a3e5eac8c09de4517beccf79dcea111264668c2 --- M be/src/common/global-flags.cc M fe/src/main/jflex/sql-scanner.flex 2 files changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/9201/1 -- To view, visit http://gerrit.cloudera.org:8080/9201 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I5a3e5eac8c09de4517beccf79dcea111264668c2 Gerrit-Change-Number: 9201 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/9191 ) Change subject: IMPALA-5269: Fix issue with final line of query followed by a comment .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/9191/2/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/9191/2/shell/impala_shell.py@394 PS2, Line 394: line = line[0: line.index('--')].strip() if '--' in line else line I don't think that this is sufficient to determine the location of a comment. For example, consider the query: select "--"; -- To view, visit http://gerrit.cloudera.org:8080/9191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258 Gerrit-Change-Number: 9191 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy WijayaGerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 02 Feb 2018 20:52:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) IMPALA-3916: Reserve SQL:2016 reserved words
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9198 ) Change subject: IMPALA-3916: Reserve SQL:2016 reserved words .. Patch Set 1: Code-Review+2 The conflict was boring, yes? I'd prefer you change the default in a separate commit. -- To view, visit http://gerrit.cloudera.org:8080/9198 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Ifd29d6aebe793d4eb37c092c1364f3acfebd91f8 Gerrit-Change-Number: 9198 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 02 Feb 2018 20:23:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1862/ -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Fri, 02 Feb 2018 19:59:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9187 Change subject: IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table .. IMPALA-6471: [docs] Corrected ALTER TABLE ADD PARTITION syntax for Kudu table Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c --- M docs/topics/impala_alter_table.xml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/9187/1 -- To view, visit http://gerrit.cloudera.org:8080/9187 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I70c49286ed6e250707a6edb5ecd77448d1142d0c Gerrit-Change-Number: 9187 Gerrit-PatchSet: 1 Gerrit-Owner: Alex RodoniGerrit-Reviewer: John Russell Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-5990: End-to-end compression of metadata
Tianyi Wang has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/8825 ) Change subject: IMPALA-5990: End-to-end compression of metadata .. IMPALA-5990: End-to-end compression of metadata Currently the catalog data is compressed in the statestore, but uncompressed when passed between FE and BE. It results in a ~2GB limit on the metadata. IMPALA-3499 introduced a workaround in the impalad but there isn't one in the catalogd. This patch aims to increase the size limit for statestore updates, reduce the copying of the metadata and reduce the memory footprint. With this patch, the catalog objects are passed and (de)compressed between FE and BE one at a time. The new limits are: - A single catalog object cannot be larger than ~2GB. - A statestore catalog update cannot be larger than ~4GB. It is compressed size if FLAGS_compact_catalog_topic is true. The behavior of the catalog op executer is not changed. The data is not compressed and the size limit is still 2GB. Testing: Ran existing tests. A test for compressing and decompressing catalog objects is added. Manually tested with a 1.95GB catalog object and a 3.90 GB uncompressed statestore update. Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae --- M be/src/catalog/CMakeLists.txt M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h A be/src/catalog/catalog-util-test.cc M be/src/catalog/catalog-util.cc M be/src/catalog/catalog-util.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/rpc/thrift-util.h M be/src/service/fe-support.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/util/jni-util.cc M be/src/util/jni-util.h M common/thrift/CatalogInternalService.thrift M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/util/TByteBuffer.java M testdata/workloads/functional-query/queries/QueryTest/exprs.test 27 files changed, 815 insertions(+), 563 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/8825/12 -- To view, visit http://gerrit.cloudera.org:8080/8825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae Gerrit-Change-Number: 8825 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-5990: End-to-end compression of metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8825 ) Change subject: IMPALA-5990: End-to-end compression of metadata .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1863/ -- To view, visit http://gerrit.cloudera.org:8080/8825 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a8819cad734b3a416eef6c954e55b73cc6023ae Gerrit-Change-Number: 8825 Gerrit-PatchSet: 12 Gerrit-Owner: Tianyi WangGerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 02 Feb 2018 19:29:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6337: Fix inifinte loop in Impala shell
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9195 Change subject: IMPALA-6337: Fix inifinte loop in Impala shell .. IMPALA-6337: Fix inifinte loop in Impala shell The patch uses a workaround for a bug in the sqlparse.split() function by joining the statements that contain error tokens into a single statement Change-Id: I9142f21a888189d351f00ce09baeba123bc0959b --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 42 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/9195/1 -- To view, visit http://gerrit.cloudera.org:8080/9195 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9142f21a888189d351f00ce09baeba123bc0959b Gerrit-Change-Number: 9195 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6204: Remove external DataSource
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9192 ) Change subject: IMPALA-6204: Remove external DataSource .. Patch Set 2: Code-Review-1 Sorry, hold off on this please. I rebased and then did some more grepping and found some more tentacles! -- To view, visit http://gerrit.cloudera.org:8080/9192 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6 Gerrit-Change-Number: 9192 Gerrit-PatchSet: 2 Gerrit-Owner: Philip ZeyligerGerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 02 Feb 2018 18:33:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3651: [DOCS] Doc for MURMUR HASH() function
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/9031 ) Change subject: IMPALA-3651: [DOCS] Doc for MURMUR_HASH() function .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/9031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I784a5a080d7d13192aac2fca67f841d2d19fc99b Gerrit-Change-Number: 9031 Gerrit-PatchSet: 2 Gerrit-Owner: Kim Jin ChulGerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: John Russell Gerrit-Reviewer: Kim Jin Chul Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 02 Feb 2018 18:28:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8971 ) Change subject: IMPALA-5519: Allocate fragment's runtime filter memory from Buffer pool .. Patch Set 8: (5 comments) Did you look at the tests that Alex mentioned for creating a test table with different cardinalities and nvds? http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java File fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java: http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@95 PS8, Line 95: private static final long MIN_BLOOM_FILTER_SIZE = 4 * 1024; : private static final long MAX_BLOOM_FILTER_SIZE = 512 * 1024 * 1024; > the query option restrictions already take care of this limit, by making su You may remove them and add a check instead. http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@109 PS8, Line 109: _ nit: no need for '_' when the fields are public http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@125 PS8, Line 125: bloomFilterSizeLimits_.max_ = tQueryOptions.getRuntime_filter_max_size(); : bloomFilterSizeLimits_.max_ = Math.max(bloomFilterSizeLimits_.max_, : MIN_BLOOM_FILTER_SIZE); : bloomFilterSizeLimits_.max_ = Math.max(bloomFilterSizeLimits_.max_, : BackendConfig.INSTANCE.getMinBufferSize()); : bloomFilterSizeLimits_.max_ = Math.min( : BitUtil.roundUpToPowerOf2(bloomFilterSizeLimits_.max_), MAX_BLOOM_FILTER_SIZE); You may want to see if it makes sense to add this logic to FilterSizeLimits class. Something like setMax(), setMin(), setDefault() functions. http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@482 PS8, Line 482: sizeLimits 'filterSizeLimits' http://gerrit.cloudera.org:8080/#/c/8971/8/fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java@494 PS8, Line 494: filterSizeBytes_ = Math.min(filterSizeBytes_, filterSizeLimits.max_); There is always the case that the size needed to achieve the desired fp rate is higher than the max limit size. In that case we should add a log message. -- To view, visit http://gerrit.cloudera.org:8080/8971 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea2759665fb2e8bef9433014a8d42a7ebf99ce1f Gerrit-Change-Number: 8971 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet VigGerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 02 Feb 2018 18:12:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6204: Remove external DataSource
Hello Dimitris Tsirogiannis, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9192 to look at the new patch set (#2). Change subject: IMPALA-6204: Remove external DataSource .. IMPALA-6204: Remove external DataSource Removes DataSourceScanNode, external data sources, and all affiliated code, tests, and documentation. When a data source table is encountered, we now throw an exception. To the user, this looks like: [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1'); Query: create table t (x int) stored as textfile tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1') Fetched 0 row(s) in 0.11s [pannier.ca.cloudera.com:21000] > select * from t; Query: select * from t Query submitted at: 2018-02-01 17:16:26 (Coordinator: http://pannier.ca.cloudera.com:25000) ERROR: AnalysisException: Failed to load metadata for table: 't' CAUSED BY: TableLoadingException: Failed to load metadata for table: default.t. Running 'invalidate metadata default.t' may resolve this problem. CAUSED BY: UnsupportedOperationException: Eternal Data source table not supported. A test has been added to capture this behavior. For the most part, I deleted the unused code. In a few places, a renamed the Thrift enums and threw errors if they're encountered. For Thrift structs, I left a comment about the now-skipped id that used to represent a data-source related entry. Cherry-picks: not for 2.x Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6 --- M CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/catalog/catalog-util.cc M be/src/exec/CMakeLists.txt M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h D be/src/exec/data-source-scan-node.cc D be/src/exec/data-source-scan-node.h M be/src/exec/exec-node.cc D be/src/exec/external-data-source-executor.cc D be/src/exec/external-data-source-executor.h M be/src/runtime/descriptors.cc M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M bin/clean-cmake.sh M bin/clean.sh M buildall.sh M common/thrift/CMakeLists.txt M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M common/thrift/Data.thrift M common/thrift/Descriptors.thrift D common/thrift/ExternalDataSource.thrift M common/thrift/Frontend.thrift M common/thrift/JniCatalog.thrift M common/thrift/PlanNodes.thrift M docs/impala.ditamap D docs/topics/impala_create_data_source.xml D docs/topics/impala_data_sources.xml D docs/topics/impala_drop_data_source.xml D ext-data-source/.gitignore D ext-data-source/CMakeLists.txt D ext-data-source/api/pom.xml D ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java D ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java D ext-data-source/pom.xml D ext-data-source/sample/pom.xml D ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java D ext-data-source/test/pom.xml D ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java M fe/CMakeLists.txt M fe/pom.xml M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java D fe/src/main/java/org/apache/impala/catalog/DataSource.java D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java D fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M
[Impala-ASF-CR] IMPALA-6204: Remove external DataSource
Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9192 Change subject: IMPALA-6204: Remove external DataSource .. IMPALA-6204: Remove external DataSource Removes DataSourceScanNode, external data sources, and all affiliated code, tests, and documentation. When a data source table is encountered, we now throw an exception. To the user, this looks like: [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1'); Query: create table t (x int) stored as textfile tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1') Fetched 0 row(s) in 0.11s [pannier.ca.cloudera.com:21000] > select * from t; Query: select * from t Query submitted at: 2018-02-01 17:16:26 (Coordinator: http://pannier.ca.cloudera.com:25000) ERROR: AnalysisException: Failed to load metadata for table: 't' CAUSED BY: TableLoadingException: Failed to load metadata for table: default.t. Running 'invalidate metadata default.t' may resolve this problem. CAUSED BY: UnsupportedOperationException: Eternal Data source table not supported. A test has been added to capture this behavior. For the most part, I deleted the unused code. In a few places, a renamed the Thrift enums and threw errors if they're encountered. For Thrift structs, I left a comment about the now-skipped id that used to represent a data-source related entry. Cherry-picks: not for 2.x Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6 --- M CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/catalog/catalog-util.cc M be/src/exec/CMakeLists.txt M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h D be/src/exec/data-source-scan-node.cc D be/src/exec/data-source-scan-node.h M be/src/exec/exec-node.cc D be/src/exec/external-data-source-executor.cc D be/src/exec/external-data-source-executor.h M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M bin/clean-cmake.sh M bin/clean.sh M buildall.sh M common/thrift/CMakeLists.txt M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M common/thrift/Data.thrift M common/thrift/Descriptors.thrift D common/thrift/ExternalDataSource.thrift M common/thrift/Frontend.thrift M common/thrift/JniCatalog.thrift M common/thrift/PlanNodes.thrift M docs/impala.ditamap D docs/topics/impala_create_data_source.xml D docs/topics/impala_data_sources.xml D docs/topics/impala_drop_data_source.xml D ext-data-source/.gitignore D ext-data-source/CMakeLists.txt D ext-data-source/api/pom.xml D ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java D ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java D ext-data-source/pom.xml D ext-data-source/sample/pom.xml D ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java D ext-data-source/test/pom.xml D ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java M fe/CMakeLists.txt M fe/pom.xml M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java D fe/src/main/java/org/apache/impala/catalog/DataSource.java D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java D fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
[Impala-ASF-CR] IMPALA-6449: Use CLOCK MONOTONIC in ConditionVariable
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/9158 ) Change subject: IMPALA-6449: Use CLOCK_MONOTONIC in ConditionVariable .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/9158/1/be/src/util/condition-variable.h File be/src/util/condition-variable.h: http://gerrit.cloudera.org:8080/#/c/9158/1/be/src/util/condition-variable.h@39 PS1, Line 39: pthread_condattr_setclock(, CLOCK_MONOTONIC); > CLOCK_MONOTONIC_COARSE may be a bit too coarse for our purpose here given s WFM -- To view, visit http://gerrit.cloudera.org:8080/9158 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I81611cfd5e7c5347203fe7fa6b0f615602257f87 Gerrit-Change-Number: 9158 Gerrit-PatchSet: 1 Gerrit-Owner: Michael HoGerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 02 Feb 2018 16:40:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5269: Fix issue with final line of query followed by a comment
fwij...@cloudera.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9191 Change subject: IMPALA-5269: Fix issue with final line of query followed by a comment .. IMPALA-5269: Fix issue with final line of query followed by a comment The patch is to fix a statement that is terminated with SQL terminated delimiter, i.e. ; followed by a comment. For example: Before: select 1 + 1; -- comment\n; After: select 1 + 1; -- comment Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258 --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 21 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/9191/2 -- To view, visit http://gerrit.cloudera.org:8080/9191 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I54f9a8f65214023520eaa010fc462a663d02d258 Gerrit-Change-Number: 9191 Gerrit-PatchSet: 2 Gerrit-Owner: fwij...@cloudera.com
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1862/ -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Fri, 02 Feb 2018 16:19:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Fri, 02 Feb 2018 16:18:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Fri, 02 Feb 2018 16:18:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5717: Support for ORC data files
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/9134 ) Change subject: IMPALA-5717: Support for ORC data files .. Patch Set 3: (24 comments) Thanks, Tim! Your comments are really useful! If we finally decide to move the ORC library into native-tool-chain project, is there any document about how to contribute to this? I think I may need the ORC library merged first than I can use it like other tools. There're still comments I haven't deal with. Forgive me to reply them later. http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.h File be/src/exec/hdfs-orc-scanner.h: http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.h@28 PS3, Line 28: class CollectionValueBuilder; > Not needed? Yeah, just added it when I try to support complex types. Will remove it. http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.h@259 PS3, Line 259: ProcessFileTail > It might be helpful to define what the "footer", "file tail" and "postscrip I was confused as well at first :) They're concepts in ORC. Here is their definitions: https://orc.apache.org/docs/file-tail.html You can also find them in be/src/orc/orc_proto.proto http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc File be/src/exec/hdfs-orc-scanner.cc: http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@1 PS3, Line 1: // Copyright 2012 Cloudera Inc. > Don't need cloudera copyrights! Done http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@24 PS3, Line 24: #include "common/object-pool.h" > Many of these headers look unused. Yes, will remove them http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@53 PS3, Line 53: using boost::algorithm::split; > Some of these boost "using" declarations don't seem to be needed. Done http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@59 PS3, Line 59: DEFINE_double(orc_min_filter_reject_ratio, 0.1, "(Advanced) If the percentage of " > I don't know why we made this flag option parquet-specific. Having an optio agree with you. There're many logics in the parquet scanner that can share with the ORC scanner. Not only this var, but also functions like IssueInitialRanges and FindFooterSplit. http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@73 PS3, Line 73: for (int i = 0; i < files.size(); ++i) { > Can we convert this to a range for? We generally prefer that in new code. Done http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@151 PS3, Line 151: mem_tracker_->CloseAndUnregisterFromParent(); > We only want to use CloseAndUnregisterFromParent() for the query-level MemT Just add the comment since I made it crashed when use Close at first... I can remove the comment if you're all clear of it :) http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@157 PS3, Line 157: std:: > Don't need std:: prefix. We generally prefer avoiding it when it isn't need We need this prefix because this class has a free function (see below) as well. http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@162 PS3, Line 162: if (ImpaladMetrics::MEM_POOL_TOTAL_BYTES != nullptr) { > You should be able to assume it's non-NULL. A lot of older code checks if m So the non-NULL checks in mem-pool.cc are redundant too? I learn this from the impala::MemPool implememtation. I found this metric useful when I ran test_failpoints.py individually. It won't come back to zero in 2 minutes. So I found bug mentioned in IMPALA-6423 http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@196 PS3, Line 196: void HdfsOrcScanner::ScanRangeInputStream::read(void* buf, uint64_t length, > It's unfortunate that the ORC code was designed to issue only synchronous r yes, quite a pity. So I hope we can include the ORC codes and modify the logics of getting input. http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@223 PS3, Line 223: memset(buf, 0, length); > Is the memset needed? If so, should document why in a comment. Just let the orc-reader to throw an exception for parse error if it read this later. Not needed actually since we throw the following exception immediately. http://gerrit.cloudera.org:8080/#/c/9134/3/be/src/exec/hdfs-orc-scanner.cc@224 PS3, Line 224: throw std::runtime_error("Cannot read from file: " + status.GetDetail()); > The fact we need to use exceptions is unfortunate. I need to think a bit mo I don't like exceptions as well. One solution is to insert the stream context into the orc-reader, and check if cancel in the loops inside it. This need to modify codes in the ORC-reader so I haven't started yet. If you guys decide to include the ORC codes, I can implement this.
[Impala-ASF-CR] IMPALA-6416: extend Thread::Create to track instance id
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/9053 ) Change subject: IMPALA-6416: extend Thread::Create to track instance id .. Patch Set 2: (7 comments) http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h File be/src/common/thread-debug-info.h: http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@79 PS2, Line 79: ExtractInfoFromParent > Extract... reads like something will be returned. How about "SetParentInfo" yeah I like that. Done. http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@80 PS2, Line 80: if (parent != nullptr) { > We usually early return instead of scoping the whole function, i.e. if (par Done http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@93 PS2, Line 93: t > nit: capital T Done http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@121 PS2, Line 121: struct ParentTdi { > Can you add a comment to this struct, too? The test has a ThreadDebugInfo p Added comment, and renamed it to "ParentInfo". http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@124 PS2, Line 124: char thread_name_[THREAD_NAME_SIZE] = {}; > I wonder if we can streamline the selective duplication of the parent threa I chose the first alternative, i.e. to store the system thread id and a pointer to the parent TDI. If it is a common case that the parent exits, we can switch to the second alternative, but also adding a pointer to the parent for convenience. http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/common/thread-debug-info.h@128 PS2, Line 128: > nit: I think this newline and the next one don't add much to readability. Done http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/util/thread.h File be/src/util/thread.h: http://gerrit.cloudera.org:8080/#/c/9053/2/be/src/util/thread.h@191 PS2, Line 191: const ThreadDebugInfo* parent_thread_info > This should go to the front (behind category or functor) now since it's str oh I see. I placed it behind functor. -- To view, visit http://gerrit.cloudera.org:8080/9053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I27de2962cf0b224c17b685d77dcba3bf2e9db187 Gerrit-Change-Number: 9053 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-NagyGerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 02 Feb 2018 14:38:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6416: extend Thread::Create to track instance id
Hello Lars Volker, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9053 to look at the new patch set (#3). Change subject: IMPALA-6416: extend Thread::Create to track instance id .. IMPALA-6416: extend Thread::Create to track instance id This commit builds upon IMPALA-3703. Each thread that was created through Thread::Create() has a ThreadDebugInfo object on the stack frame of Thread::SuperviseThread(). This object has stack allocated char buffers that can be read during a debug session even if we only have minidumps. However, with the old solution ThreadDebugInfo::instance_id was set manually for each thread. It is too easy to forget to set instance_id every time we create a new thread. This commit has the assumption that if a thread has an instance id associated, then the threads spawned by it will always work on the same instance id. In Thread::StartThread the parent thread passes its ThreadDebugInfo object to its child who copies the instance id and also saves its parent's name. This means if we set ThreadDebugInfo::instance_id in some "root thread", then all descendant threads will annotate themselves with the instance id automatically. Since threads also record their parent's name, it might be also possible to reconstruct the thread creation graph. With GDB I tested if it copies the instance id at every place where we previously needed to set it manually. I added an automated test to thread-debug-info-test.cc Change-Id: I27de2962cf0b224c17b685d77dcba3bf2e9db187 --- M be/src/common/thread-debug-info-test.cc M be/src/common/thread-debug-info.cc M be/src/common/thread-debug-info.h M be/src/exec/blocking-join-node.cc M be/src/exec/hdfs-scan-node.cc M be/src/runtime/fragment-instance-state.cc M be/src/util/thread.cc M be/src/util/thread.h 8 files changed, 81 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/9053/3 -- To view, visit http://gerrit.cloudera.org:8080/9053 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I27de2962cf0b224c17b685d77dcba3bf2e9db187 Gerrit-Change-Number: 9053 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-NagyGerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. Patch Set 6: Code-Review+1 LGTM! -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Fri, 02 Feb 2018 13:28:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5237: Support a quoted string in date/time format
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/8508 ) Change subject: IMPALA-5237: Support a quoted string in date/time format .. Patch Set 8: Code-Review+1 Thanks for taking care of this. For me it's fine. -- To view, visit http://gerrit.cloudera.org:8080/8508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie34055ac695748bcfb110bfa6ed5308f469ea178 Gerrit-Change-Number: 8508 Gerrit-PatchSet: 8 Gerrit-Owner: Kim Jin ChulGerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Kim Jin Chul Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 02 Feb 2018 10:09:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/9140 ) Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/9140/3/be/src/exec/parquet-column-stats.cc File be/src/exec/parquet-column-stats.cc: http://gerrit.cloudera.org:8080/#/c/9140/3/be/src/exec/parquet-column-stats.cc@132 PS3, Line 132: bool ColumnStatsBase::ReadNullCountStat(const parquet::ColumnChunk& col_chunk, > Why do we need to return a "const int64_t*" to the caller? Thanks Tim for the explanation. Makes sense. Done. http://gerrit.cloudera.org:8080/#/c/9140/3/tests/query_test/test_parquet_stats.py File tests/query_test/test_parquet_stats.py: http://gerrit.cloudera.org:8080/#/c/9140/3/tests/query_test/test_parquet_stats.py@49 PS3, Line 49: self.run_test_case('QueryTest/parquet-stats', vector, use_db=unique_database) > Sorry about the confusion. I thought that adding the create table statement Thanks Tim, Anuj! Dropped the $DATABASE prefix. -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Fri, 02 Feb 2018 09:20:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6113: Skip row groups with predicates on NULL columns
Hello Lars Volker, Zoltan Borok-Nagy, anujphadke, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9140 to look at the new patch set (#6). Change subject: IMPALA-6113: Skip row groups with predicates on NULL columns .. IMPALA-6113: Skip row groups with predicates on NULL columns Based on the existing Parquet column chunk level statistics null_count, Impala's Parquet scanner is enhanced to skip an entire row group if the null_count statistics indicate that all the values under the predicated column are NULL as we wouldn't get any result rows from that row group anyway. Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf --- M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/parquet-column-stats.cc M be/src/exec/parquet-column-stats.h M testdata/workloads/functional-query/queries/QueryTest/parquet-stats.test 4 files changed, 59 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/9140/6 -- To view, visit http://gerrit.cloudera.org:8080/9140 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I141317af0e0df30da8f220b29b0bfba364f40ddf Gerrit-Change-Number: 9140 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor KaszabGerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: anujphadke