[Impala-ASF-CR] IMPALA-6119: Fix issue with multiple partitions sharing same location
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/10543 )
Change subject: IMPALA-6119: Fix issue with multiple partitions sharing same
location
..
Patch Set 10:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/10543/10/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/10543/10/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1220
PS10, Line 1220: partitionSet = Sets.newHashSet();
: partitionSet.add(partition);
: locationToPartMap_.put(location, partitionSet);
locationToPartMap_.put(location, Sets.newHashSet(partition));
http://gerrit.cloudera.org:8080/#/c/10543/1/tests/metadata/test_partition_metadata.py
File tests/metadata/test_partition_metadata.py:
http://gerrit.cloudera.org:8080/#/c/10543/1/tests/metadata/test_partition_metadata.py@159
PS1, Line 159: assert data.split('\t')[1] == '6'
> I managed to check this in Hive: It also drops the folder of the partition
Sorry, I misunderstood your comment. I was actually trying this out myself in
Hive and I think I understand Hive's behavior now. Like you mentioned Hive
just drops the directory irrespective of whether anything else points to it
*but* still retains the other partitions and they remain in a dangling state.
Querying them, we get no results back.
I think Impala should also do the same thing. This can be done by calling
for each (partition: locationToPartMap_(partitionTobedropped.location))
partition.setFileDescriptors(emptyList)
IMO that behavior makes more sense and makes the code simpler too while keeping
us consistent with Hive. Thoughts?
http://gerrit.cloudera.org:8080/#/c/10543/10/tests/metadata/test_partition_metadata.py
File tests/metadata/test_partition_metadata.py:
http://gerrit.cloudera.org:8080/#/c/10543/10/tests/metadata/test_partition_metadata.py@175
PS10, Line 175: Then Hive
: drops one of these partitions and as a result both are
expected to be dropped.
Like I mentioned in the other comment, this is not expected from a Hive POV
AFAICT. Also, reading the comments from Sergey in
https://issues.apache.org/jira/browse/HIVE-19830 gives the same impression.
So, we should probably just drop the partition we were asked to drop and
invalidate the filedescriptors of all the partitions that point to the location
that just dropped/deleted.
--
To view, visit http://gerrit.cloudera.org:8080/10543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2a54bc8224bcefe65b83de2df58bb84629f2aa4a
Gerrit-Change-Number: 10543
Gerrit-PatchSet: 10
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Thu, 14 Jun 2018 05:48:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Add missing PrintId() calls
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10716 ) Change subject: Add missing PrintId() calls .. Add missing PrintId() calls For consistency, add PrintId() around query_id() used in a stream, which was missing from this commit but needed for 2.x backport: IMPALA-5216: Make admission control queuing async This change was put into the cherry-pick for 2.x, so: Cherry-picks: not for 2.x Change-Id: Ifa53bf46411b09f33ccf38b080cd1c06dd3717ef Reviewed-on: http://gerrit.cloudera.org:8080/10716 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/admission-controller.cc M be/src/service/client-request-state.cc 2 files changed, 7 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifa53bf46411b09f33ccf38b080cd1c06dd3717ef Gerrit-Change-Number: 10716 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] Add missing PrintId() calls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10716 ) Change subject: Add missing PrintId() calls .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa53bf46411b09f33ccf38b080cd1c06dd3717ef Gerrit-Change-Number: 10716 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 14 Jun 2018 04:49:02 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10715 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2666/ -- To view, visit http://gerrit.cloudera.org:8080/10715 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10715 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 04:28:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Bharath Vissapragada has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9933 ) Change subject: IMPALA-2195: Improper handling of comments in queries .. IMPALA-2195: Improper handling of comments in queries This patch fixes an issue where parseline is unable to deduce the correct command when a statement has a leading comment. Before: > -- comment > insert into table t values(100); Fetched 1 row(s) in 0.01s After: > -- comment > insert into table t values(100); Modified 1 row(s) in 0.01s Before (FE syntax error): > /*comment*/ help; After (show help correctly): > /*comment*/ help; Testing: - Added shell tests - Ran end-to-end shell tests on Python 2.6 and Python 2.7 Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Reviewed-on: http://gerrit.cloudera.org:8080/9933 Tested-by: Impala Public Jenkins Reviewed-by: Bharath Vissapragada --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 173 insertions(+), 12 deletions(-) Approvals: Impala Public Jenkins: Verified Bharath Vissapragada: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/9933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Gerrit-Change-Number: 9933 Gerrit-PatchSet: 16 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/9933 ) Change subject: IMPALA-2195: Improper handling of comments in queries .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Gerrit-Change-Number: 9933 Gerrit-PatchSet: 15 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Thu, 14 Jun 2018 04:11:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9933 ) Change subject: IMPALA-2195: Improper handling of comments in queries .. Patch Set 15: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Gerrit-Change-Number: 9933 Gerrit-PatchSet: 15 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Thu, 14 Jun 2018 03:57:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10629 ) Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 03:47:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10365 ) Change subject: IMPALA-6035: Add query options to limit thread reservation .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 03:25:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7161: Fix impala-config.sh's handling of JAVA HOME
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10702 ) Change subject: IMPALA-7161: Fix impala-config.sh's handling of JAVA_HOME .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/10702/2/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/10702/2/bin/impala-config.sh@230 PS2, Line 230: # bin/impala-config-local.sh to set JAVA_HOME, so it is important to pick up that We could remove this from bootstrap_development as part of this change. http://gerrit.cloudera.org:8080/#/c/10702/2/bin/impala-config.sh@238 PS2, Line 238: SYSTEM_JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:bin/javac::") > How about using "which javac" and "JAVA_HOME=$(which javac|xargs readlink - I think Tianyi is right here in that we should honor the javac that's on the PATH if we're picking a default. http://gerrit.cloudera.org:8080/#/c/10702/2/bin/impala-config.sh@241 PS2, Line 241: # Prefer the JAVA_HOME set in the environment, but use the system's JAVA_HOME otherwise If you don't mind, could you remove the following line in this change. $git grep JAVA_HOME | grep entrypoi docker/entrypoint.sh: export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 -- To view, visit http://gerrit.cloudera.org:8080/10702 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idf3521b4f44fdbdc841a90fd00c477c9423a75bb Gerrit-Change-Number: 10702 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Thu, 14 Jun 2018 03:28:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10365 ) Change subject: IMPALA-6035: Add query options to limit thread reservation .. IMPALA-6035: Add query options to limit thread reservation Adds two options: THREAD_RESERVATION_LIMIT and THREAD_RESERVATION_AGGREGATE_LIMIT, which are both enforced by admission control based on planner resource requirements and the schedule. The mechanism used is the same as the minimum reservation checks. THREAD_RESERVATION_LIMIT limits the total number of reserved threads in fragments scheduled on a single backend. THREAD_RESERVATION_AGGREGATE_LIMIT limits the sum of reserved threads across all fragments. This also slightly improves the minimum reservation error message to include the host name. Testing: Added end-to-end tests that exercise the code paths. Ran core tests. Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Reviewed-on: http://gerrit.cloudera.org:8080/10365 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test A testdata/workloads/functional-query/queries/QueryTest/thread-limits.test A tests/query_test/test_resource_limits.py 12 files changed, 254 insertions(+), 29 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10471 ) Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Gerrit-Change-Number: 10471 Gerrit-PatchSet: 11 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 01:36:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. Patch Set 18: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/9986/18/be/src/util/zip-util.h File be/src/util/zip-util.h: http://gerrit.cloudera.org:8080/#/c/9986/18/be/src/util/zip-util.h@33 PS18, Line 33: //Extract files from a zip archive to a destination directory in local filesystem. nit: comment formatting - should start with /// http://gerrit.cloudera.org:8080/#/c/9986/10/testdata/tzdb/2017c/Africa/Abidjan File testdata/tzdb/2017c/Africa/Abidjan: http://gerrit.cloudera.org:8080/#/c/9986/10/testdata/tzdb/2017c/Africa/Abidjan@1 PS10, Line 1: > These are binary data files created from text data files that are in the pu How does they pass the rat checks? We run those precommit. I expected to see something in bin/rat_exclude_files.txt. -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 Gerrit-Change-Number: 9986 Gerrit-PatchSet: 18 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 14 Jun 2018 01:44:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10471 ) Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/10471/9//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10471/9//COMMIT_MSG@13 PS9, Line 13: implementing a feature > add a jira for this to make it concrete. Done -- To view, visit http://gerrit.cloudera.org:8080/10471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Gerrit-Change-Number: 10471 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 01:30:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Fredy Wijaya has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/10471 ) Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER .. IMPALA-7016: Implement ALTER DATABASE SET OWNER Alter the database owner to either user or role. On database creation, the database owner will be set to the current user, which can be viewed via DESCRIBE DATABASE db command. Having an owner information allows implementing a feature where an owner can be given certain privileges automatically upon a database creation (https://issues.apache.org/jira/browse/IMPALA-7075). The ALTER DATABASE SET OWNER will be a useful command for transferring ownership (a set of owner privileges) from the current owner to another owner. Syntax: ALTER DATABASE db SET OWNER USER user ALTER DATABASE db SET OWNER ROLE role Testing: - Added new front-end tests - Added new end-to-end tests Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 --- M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup A fe/src/main/java/org/apache/impala/analysis/AlterDbSetOwnerStmt.java A fe/src/main/java/org/apache/impala/analysis/AlterDbStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/Owner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 15 files changed, 361 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10471/10 -- To view, visit http://gerrit.cloudera.org:8080/10471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Gerrit-Change-Number: 10471 Gerrit-PatchSet: 10 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] Add missing PrintId() calls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10716 ) Change subject: Add missing PrintId() calls .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa53bf46411b09f33ccf38b080cd1c06dd3717ef Gerrit-Change-Number: 10716 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 14 Jun 2018 01:28:27 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing PrintId() calls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10716 ) Change subject: Add missing PrintId() calls .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2667/ -- To view, visit http://gerrit.cloudera.org:8080/10716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa53bf46411b09f33ccf38b080cd1c06dd3717ef Gerrit-Change-Number: 10716 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 14 Jun 2018 01:28:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition This refactors out interfaces in the frontend for the interaction between the analysis/planning code and the classes that implement these catalog objects. This takes care of the most commonly used objects but defers some others (e.g. functions, cache pools, data sources, etc) to follow-on patches. There are a few spots remaining in the frontend that still downcast to implementation classes, particularly where the frontend actually makes modifications to catalog objects in-place. I left TODOs in those spots and will come back later as necessary. Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Reviewed-on: http://gerrit.cloudera.org:8080/10611 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddReplaceColsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetRowFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateFunctionStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DropFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionDef.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/analysis/PartitionSpecBase.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java A fe/src/main/java/org/apache/impala/catalog/FeCatalog.java A
[Impala-ASF-CR] Add missing PrintId() calls
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10716 ) Change subject: Add missing PrintId() calls .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifa53bf46411b09f33ccf38b080cd1c06dd3717ef Gerrit-Change-Number: 10716 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 14 Jun 2018 00:58:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7161: Fix impala-config.sh's handling of JAVA HOME
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10702 ) Change subject: IMPALA-7161: Fix impala-config.sh's handling of JAVA_HOME .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10702/2/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/10702/2/bin/impala-config.sh@238 PS2, Line 238: SYSTEM_JAVA_HOME=$(readlink -f /usr/bin/javac | sed "s:bin/javac::") How about using "which javac" and "JAVA_HOME=$(which javac|xargs readlink -f|xargs dirname|xargs dirname)"? It will work for javac not in /usr/bin as well. -- To view, visit http://gerrit.cloudera.org:8080/10702 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idf3521b4f44fdbdc841a90fd00c477c9423a75bb Gerrit-Change-Number: 10702 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Thu, 14 Jun 2018 01:01:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7161: Fix impala-config.sh's handling of JAVA HOME
Joe McDonnell has uploaded a new patch set (#1). ( http://gerrit.cloudera.org:8080/10702 ) Change subject: IMPALA-7161: Fix impala-config.sh's handling of JAVA_HOME .. IMPALA-7161: Fix impala-config.sh's handling of JAVA_HOME It is common for developers to specify JAVA_HOME in bin/impala-config-local.sh, so wait until after it is sourced to validate JAVA_HOME. Also, try harder to auto-detect the system's JAVA_HOME in case it has not been specified in the environment. Change-Id: Idf3521b4f44fdbdc841a90fd00c477c9423a75bb --- M bin/impala-config.sh 1 file changed, 26 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/10702/1 -- To view, visit http://gerrit.cloudera.org:8080/10702 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idf3521b4f44fdbdc841a90fd00c477c9423a75bb Gerrit-Change-Number: 10702 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10627 ) Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 01:19:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7161: Fix impala-config.sh's handling of JAVA HOME
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10702 to look at the new patch set (#2). Change subject: IMPALA-7161: Fix impala-config.sh's handling of JAVA_HOME .. IMPALA-7161: Fix impala-config.sh's handling of JAVA_HOME It is common for developers to specify JAVA_HOME in bin/impala-config-local.sh, so wait until after it is sourced to validate JAVA_HOME. Also, try harder to auto-detect the system's JAVA_HOME in case it has not been specified in the environment. Here is a run through of different scenarios: 1. Not set in environment, not set in impala-config-local.sh: Didn't work before, now tries to autodetect 2. Set in environment, not set in impala-config-local.sh: No change 3. Not set in environment, set in impala-config-local.sh: Didn't work before, now works 4. Set in environment and set in impala-config-local.sh: This used to be potentially inconsistent (i.e. JAVA comes from the environment's JAVA_HOME, but JAVA_HOME is overwritten by impala-config-local.sh), now it always uses the value from impala-config-local.sh. Change-Id: Idf3521b4f44fdbdc841a90fd00c477c9423a75bb --- M bin/impala-config.sh 1 file changed, 26 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/10702/2 -- To view, visit http://gerrit.cloudera.org:8080/10702 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idf3521b4f44fdbdc841a90fd00c477c9423a75bb Gerrit-Change-Number: 10702 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](2.x) IMPALA-5216: Make admission control queuing async
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10715 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 1: Code-Review+2 Thanks for doing the backport, Tim. -- To view, visit http://gerrit.cloudera.org:8080/10715 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10715 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 00:57:16 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-5216: Make admission control queuing async
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10715 ) Change subject: IMPALA-5216: Make admission control queuing async .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2666/ -- To view, visit http://gerrit.cloudera.org:8080/10715 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10715 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 00:50:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Add missing PrintId() calls
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10716 Change subject: Add missing PrintId() calls .. Add missing PrintId() calls For consistency, add PrintId() around query_id() used in a stream, which was missing from this commit but needed for 2.x backport: IMPALA-5216: Make admission control queuing async This change was put into the cherry-pick for 2.x, so: Cherry-picks: not for 2.x Change-Id: Ifa53bf46411b09f33ccf38b080cd1c06dd3717ef --- M be/src/scheduling/admission-controller.cc M be/src/service/client-request-state.cc 2 files changed, 7 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/10716/1 -- To view, visit http://gerrit.cloudera.org:8080/10716 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifa53bf46411b09f33ccf38b080cd1c06dd3717ef Gerrit-Change-Number: 10716 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 00:46:27 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-5216: Make admission control queuing async
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/10715 )
Change subject: IMPALA-5216: Make admission control queuing async
..
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/10715/1/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/10715/1/be/src/scheduling/admission-controller.cc@535
PS1, Line 535: PrintId
Some PrintId() were missing. This was a compile error on 2.x because of the
different thrift version.
http://gerrit.cloudera.org:8080/#/c/10715/1/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/10715/1/be/src/service/client-request-state.cc@496
PS1, Line 496: if (exec_env_->admission_controller() != nullptr) {
There was a conflict here - this is a DCHECK on master because we don't support
running with AC disabled.
http://gerrit.cloudera.org:8080/#/c/10715/1/tests/query_test/test_observability.py
File tests/query_test/test_observability.py:
http://gerrit.cloudera.org:8080/#/c/10715/1/tests/query_test/test_observability.py@92
PS1, Line 92: handle = self.execute_query_async(query,
There was a conflict here because of using a different query option. I just
switched to the master version of the test, since it works on both branches.
--
To view, visit http://gerrit.cloudera.org:8080/10715
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf
Gerrit-Change-Number: 10715
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 14 Jun 2018 00:48:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) IMPALA-5216: Make admission control queuing async
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10715 Change subject: IMPALA-5216: Make admission control queuing async .. IMPALA-5216: Make admission control queuing async Implement asynchronous admission control queuing. This is achieved by running the admission control code-path in a separate thread. Major changes include: propagating cancellation to the admission control thread and dequeuing thread, and adding a new Query Operation State called "PENDING" that represents the state between completion of planning and starting of query execution. Testing: - Added a deterministic end to end test and a session expiry test. - Ran multiple stress tests successfully with a cancellation probability of 60% and with different values for the following parameters: max_requests, queue_wait_timeout_ms. Ensured that the impalad was in a valid state afterwards (no orphan fragments or wrong metrics). - Ran all exhaustive tests and ASAN core tests successfully. - Ran data load successfully. Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf --- M be/src/common/atomic.h M be/src/common/logging.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/query-state.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/promise-test.cc M be/src/util/promise.h M common/thrift/ImpalaService.thrift M tests/authorization/test_authorization.py M tests/beeswax/impala_beeswax.py M tests/common/impala_connection.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_krpc_mem_usage.py M tests/custom_cluster/test_session_expiration.py M tests/hs2/hs2_test_suite.py M tests/hs2/test_hs2.py M tests/query_test/test_observability.py M tests/query_test/test_udfs.py M www/query_backends.tmpl 31 files changed, 690 insertions(+), 245 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/10715/1 -- To view, visit http://gerrit.cloudera.org:8080/10715 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf Gerrit-Change-Number: 10715 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/10571/14/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java File fe/src/main/java/org/apache/impala/analysis/UnionStmt.java: http://gerrit.cloudera.org:8080/#/c/10571/14/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java@562 PS14, Line 562: withClause_ > We currently do any SQL rewrite for WITH clause: https://github.com/apache/ I meant to say do not do* -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 16 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 00:41:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 16: (1 comment) > Patch Set 14: > > (1 comment) > > Just skimming through the patch, have a quick question. Is the plan to do it > for all statements? For ex, InsertStmt is not covered? I updated all statements that may require a rewrite. Please review it again. http://gerrit.cloudera.org:8080/#/c/10571/14/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java File fe/src/main/java/org/apache/impala/analysis/UnionStmt.java: http://gerrit.cloudera.org:8080/#/c/10571/14/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java@562 PS14, Line 562: withClause_ > what about this? Does the following work? We currently do any SQL rewrite for WITH clause: https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java#L362-L369 However, I'll still update the code in case we decide to run SQL rewrite for WITH clause in the future. -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 16 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 00:39:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. IMPALA-7106: Print rewritten SQL correctly when log trace is enabled toSql() method is used to print SQL string that is close to the original SQL string, which is something that users want to see. When debugging issues related to SQL rewrites, it can be very useful to be able to print/get the SQL string that is being rewritten. This patch adds a new method toSql(boolean rewritten) to get the rewritten SQL string. The LOG.trace statement that prints the rewritten SQL is also updated to use toSql(true). Testing: - Added FE test for the rewritten SQL string - Ran all FE tests Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java 17 files changed, 206 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10571/16 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 16 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-4908: NULL floats don't compare equal to other NULL floats
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10707 ) Change subject: IMPALA-4908: NULL floats don't compare equal to other NULL floats .. IMPALA-4908: NULL floats don't compare equal to other NULL floats This change ensures that comparing two NULL floats with different val fields returns true. Although this is undefined behavior, it is now consistent with other types. Along with the change, a unit test was added to ensure that equality checking of floats returns results as expected. Change-Id: Ie7310645e5752d8203be5abc22a6562a59b6e975 Reviewed-on: http://gerrit.cloudera.org:8080/10707 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/udf/udf-test.cc M be/src/udf/udf.h 2 files changed, 28 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie7310645e5752d8203be5abc22a6562a59b6e975 Gerrit-Change-Number: 10707 Gerrit-PatchSet: 3 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4908: NULL floats don't compare equal to other NULL floats
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10707 ) Change subject: IMPALA-4908: NULL floats don't compare equal to other NULL floats .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7310645e5752d8203be5abc22a6562a59b6e975 Gerrit-Change-Number: 10707 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 00:33:54 + Gerrit-HasComments: No
[Impala-ASF-CR] Add a missing PrintId()
Tim Armstrong has abandoned this change. ( http://gerrit.cloudera.org:8080/10714 ) Change subject: Add a missing PrintId() .. Abandoned Actually I need to add it in a couple more places.. will abandon for now. -- To view, visit http://gerrit.cloudera.org:8080/10714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: If01b72c66a3ae505942128580d8177af337a8b69 Gerrit-Change-Number: 10714 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10629 ) Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2664/ -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 14 Jun 2018 00:11:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9933 ) Change subject: IMPALA-2195: Improper handling of comments in queries .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2665/ -- To view, visit http://gerrit.cloudera.org:8080/9933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Gerrit-Change-Number: 9933 Gerrit-PatchSet: 15 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Thu, 14 Jun 2018 00:20:32 + Gerrit-HasComments: No
[Impala-ASF-CR] Add a missing PrintId()
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10714 Change subject: Add a missing PrintId() .. Add a missing PrintId() For consistency, add a PrintId() around a query_id() used in a stream, which was missing from this commit but needed for 2.x backport: IMPALA-5216: Make admission control queuing asynca This change was put into the cherry-pick for 2.x, so: Cherry-picks: not for 2.x Change-Id: If01b72c66a3ae505942128580d8177af337a8b69 --- M be/src/service/client-request-state.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/10714/1 -- To view, visit http://gerrit.cloudera.org:8080/10714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If01b72c66a3ae505942128580d8177af337a8b69 Gerrit-Change-Number: 10714 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10365 ) Change subject: IMPALA-6035: Add query options to limit thread reservation .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 00:08:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10365 ) Change subject: IMPALA-6035: Add query options to limit thread reservation .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2663/ -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 00:08:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Hello Bikramjeet Vig, Mostafa Mokhtar, Dan Hecht, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10365 to look at the new patch set (#12). Change subject: IMPALA-6035: Add query options to limit thread reservation .. IMPALA-6035: Add query options to limit thread reservation Adds two options: THREAD_RESERVATION_LIMIT and THREAD_RESERVATION_AGGREGATE_LIMIT, which are both enforced by admission control based on planner resource requirements and the schedule. The mechanism used is the same as the minimum reservation checks. THREAD_RESERVATION_LIMIT limits the total number of reserved threads in fragments scheduled on a single backend. THREAD_RESERVATION_AGGREGATE_LIMIT limits the sum of reserved threads across all fragments. This also slightly improves the minimum reservation error message to include the host name. Testing: Added end-to-end tests that exercise the code paths. Ran core tests. Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test A testdata/workloads/functional-query/queries/QueryTest/thread-limits.test A tests/query_test/test_resource_limits.py 12 files changed, 254 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/10365/12 -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10365 ) Change subject: IMPALA-6035: Add query options to limit thread reservation .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2662/ -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 00:07:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10365 ) Change subject: IMPALA-6035: Add query options to limit thread reservation .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 00:07:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10365 ) Change subject: IMPALA-6035: Add query options to limit thread reservation .. Patch Set 10: Code-Review+2 rebased and resolved conflicts related to Scheduler* -> const Scheduler& -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 14 Jun 2018 00:06:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Hello Bikramjeet Vig, Mostafa Mokhtar, Dan Hecht, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10365 to look at the new patch set (#10). Change subject: IMPALA-6035: Add query options to limit thread reservation .. IMPALA-6035: Add query options to limit thread reservation Adds two options: THREAD_RESERVATION_LIMIT and THREAD_RESERVATION_AGGREGATE_LIMIT, which are both enforced by admission control based on planner resource requirements and the schedule. The mechanism used is the same as the minimum reservation checks. THREAD_RESERVATION_LIMIT limits the total number of reserved threads in fragments scheduled on a single backend. THREAD_RESERVATION_AGGREGATE_LIMIT limits the sum of reserved threads across all fragments. This also slightly improves the minimum reservation error message to include the host name. Testing: Added end-to-end tests that exercise the code paths. Ran core tests. Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test A testdata/workloads/functional-query/queries/QueryTest/thread-limits.test A tests/query_test/test_resource_limits.py 12 files changed, 254 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/10365/10 -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6802 (part 4): Clean up authorization tests
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10442 ) Change subject: IMPALA-6802 (part 4): Clean up authorization tests .. Patch Set 4: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/10442/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java: http://gerrit.cloudera.org:8080/#/c/10442/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@946 PS3, Line 946: ctional", : "alltypes", privilege)); > This is the pattern we've been using in these tests to be explicit for each hmm.. ok, guess it takes a bit of getting used to http://gerrit.cloudera.org:8080/#/c/10442/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java: http://gerrit.cloudera.org:8080/#/c/10442/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@919 PS4, Line 919: // use /** style comments for methods. -- To view, visit http://gerrit.cloudera.org:8080/10442 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4cf3a4751b41908ef81ec35d89a2713d9fa0dc4 Gerrit-Change-Number: 10442 Gerrit-PatchSet: 4 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 23:53:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7165: [DOCS] Correct example for dynamic partition pruning
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10703 ) Change subject: IMPALA-7165: [DOCS] Correct example for dynamic partition pruning .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/10703/2/docs/shared/impala_common.xml File docs/shared/impala_common.xml: http://gerrit.cloudera.org:8080/#/c/10703/2/docs/shared/impala_common.xml@a1883 PS2, Line 1883: > did this text get moved to the other file? This text was now restored. http://gerrit.cloudera.org:8080/#/c/10703/2/docs/topics/impala_runtime_filtering.xml File docs/topics/impala_runtime_filtering.xml: http://gerrit.cloudera.org:8080/#/c/10703/2/docs/topics/impala_runtime_filtering.xml@348 PS2, Line 348: labelled > labeled Done -- To view, visit http://gerrit.cloudera.org:8080/10703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 Gerrit-Change-Number: 10703 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 23:27:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7140 (part 3): load partitions for FS tables
Hello Vuk Ercegovac, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10713 to review the following change. Change subject: IMPALA-7140 (part 3): load partitions for FS tables .. IMPALA-7140 (part 3): load partitions for FS tables This implements the loading of partitions from the HMS for FS-based tables. The LocalPartitionSpec class implements PrunablePartition based on parsing the partition names returned from the HMS. After partitions are pruned, the remaining partitions can be loaded by name. With this patch, I am able to connect to an impalad running with --use_local_catalog and issue 'show partitions functional.alltypes' with the expected results. A new unit test is added which shares some code with CatalogTest to ensure that partitions can be loaded and parsed properly. Testing partition pruning via end-to-end planning is not quite supported yet: we need to implement 'toThriftDescriptor()' before we can run those end-to-end tests. Change-Id: Iddf2edbd6bdc0684560b2ecca9c4c6b6819ef1d3 --- M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/PartitionStatsUtil.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java A fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java A fe/src/main/java/org/apache/impala/catalog/local/LocalPartitionSpec.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 12 files changed, 637 insertions(+), 88 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/10713/1 -- To view, visit http://gerrit.cloudera.org:8080/10713 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iddf2edbd6bdc0684560b2ecca9c4c6b6819ef1d3 Gerrit-Change-Number: 10713 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7165: [DOCS] Correct example for dynamic partition pruning
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10703 ) Change subject: IMPALA-7165: [DOCS] Correct example for dynamic partition pruning .. Patch Set 3: ah, thanks for the explanation! -- To view, visit http://gerrit.cloudera.org:8080/10703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 Gerrit-Change-Number: 10703 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 23:36:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10510 ) Change subject: IMPALA-5552: Add support for authorized proxy groups .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2661/ -- To view, visit http://gerrit.cloudera.org:8080/10510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Gerrit-Change-Number: 10510 Gerrit-PatchSet: 17 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 23:35:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10510 ) Change subject: IMPALA-5552: Add support for authorized proxy groups .. Patch Set 16: Code-Review+1 This looks fine to me at this point. I'm going to kick off GVO, but I've not +2'd it because Tim's still gating commits. I'll poke him to see if he wants to hold off on this or not. -- To view, visit http://gerrit.cloudera.org:8080/10510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Gerrit-Change-Number: 10510 Gerrit-PatchSet: 16 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 23:35:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 ) Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 23:35:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7165: [DOCS] Correct example for dynamic partition pruning
Hello Henry Robinson, Gabor Kaszab, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10703 to look at the new patch set (#3). Change subject: IMPALA-7165: [DOCS] Correct example for dynamic partition pruning .. IMPALA-7165: [DOCS] Correct example for dynamic partition pruning Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 --- M docs/shared/impala_common.xml M docs/topics/impala_partitioning.xml M docs/topics/impala_runtime_filtering.xml 3 files changed, 96 insertions(+), 58 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/10703/3 -- To view, visit http://gerrit.cloudera.org:8080/10703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 Gerrit-Change-Number: 10703 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7121: Clean up partitionIds from HdfsTable
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10654 ) Change subject: IMPALA-7121: Clean up partitionIds_ from HdfsTable .. Patch Set 2: Code-Review+1 Looks reasonable to me. Should be easy to combine with my patch linked above regardless of which one is committed first. -- To view, visit http://gerrit.cloudera.org:8080/10654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da Gerrit-Change-Number: 10654 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 13 Jun 2018 23:20:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7121: Clean up partitionIds from HdfsTable
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10654 ) Change subject: IMPALA-7121: Clean up partitionIds_ from HdfsTable .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10654/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10654/2//COMMIT_MSG@16 PS2, Line 16: One thing needs extra attention here is that modifying the result of It looks like this adds an extra copy in a lot of cases though, e.g. matchingIds.addAll(tbl_.getPartitionIds()); This may not matter that much, but it's hard to know for sure. What about wrapping it in Collections.unmodifableSet() instead of making a copy: https://docs.oracle.com/javase/7/docs/api/java/util/Collections.html#unmodifiableSet(java.util.Set) -- To view, visit http://gerrit.cloudera.org:8080/10654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da Gerrit-Change-Number: 10654 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 13 Jun 2018 23:20:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7165: [DOCS] Correct example for dynamic partition pruning
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10703 ) Change subject: IMPALA-7165: [DOCS] Correct example for dynamic partition pruning .. Patch Set 2: (2 comments) what was missing/incorrect in the example? http://gerrit.cloudera.org:8080/#/c/10703/2/docs/shared/impala_common.xml File docs/shared/impala_common.xml: http://gerrit.cloudera.org:8080/#/c/10703/2/docs/shared/impala_common.xml@a1883 PS2, Line 1883: did this text get moved to the other file? http://gerrit.cloudera.org:8080/#/c/10703/2/docs/topics/impala_runtime_filtering.xml File docs/topics/impala_runtime_filtering.xml: http://gerrit.cloudera.org:8080/#/c/10703/2/docs/topics/impala_runtime_filtering.xml@348 PS2, Line 348: labelled labeled -- To view, visit http://gerrit.cloudera.org:8080/10703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 Gerrit-Change-Number: 10703 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 23:19:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7165: [DOCS] Correct example for dynamic partition pruning
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10703 ) Change subject: IMPALA-7165: [DOCS] Correct example for dynamic partition pruning .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/316/ -- To view, visit http://gerrit.cloudera.org:8080/10703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 Gerrit-Change-Number: 10703 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 23:16:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7121: Clean up partitionIds from HdfsTable
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10654 ) Change subject: IMPALA-7121: Clean up partitionIds_ from HdfsTable .. Patch Set 2: Ah, I just did some work in this same area in https://gerrit.cloudera.org/#/c/10711/1 . Let me take a look at this, and you might want to take a look at that one, and we can combine efforts. Didn't notice your review before I wrote mine. -- To view, visit http://gerrit.cloudera.org:8080/10654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da Gerrit-Change-Number: 10654 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 13 Jun 2018 23:14:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7140 (part 2). Create skeleton for LocalFsTable
Hello Vuk Ercegovac, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10712 to review the following change. Change subject: IMPALA-7140 (part 2). Create skeleton for LocalFsTable .. IMPALA-7140 (part 2). Create skeleton for LocalFsTable This adds a skeleton implementation of FeFsTable for the LocalCatalog. Most of the implementation in this patch is stubbed out -- only the bare bones necessary to instantiate the subclass when a table is loaded and the simplest table-level methods. Change-Id: Id2b184104d92e128250df5a08ac7ffb3dde011a8 --- M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalogException.java A fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 10 files changed, 263 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/10712/1 -- To view, visit http://gerrit.cloudera.org:8080/10712 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id2b184104d92e128250df5a08ac7ffb3dde011a8 Gerrit-Change-Number: 10712 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10631 to look at the new patch set (#6). Change subject: IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading .. IMPALA-7141 (part 2). Extract interfaces for partition pruning without loading This adds a new PrunablePartition interface which HdfsPartition implements. The interface includes only the partition ID and the partition key/values. For the case of the current catalog implementation, this provides no benefit. However, for LocalCatalog, we want to defer loading partition information until after pruning. With this interface, we can construct PrunablePartition objects using just the partition names, and then load the partitions once pruning is complete. Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java 18 files changed, 173 insertions(+), 50 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10631/6 -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7141 (part 1): clean up handling of default/dummy partition
Hello Vuk Ercegovac, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10711 to review the following change. Change subject: IMPALA-7141 (part 1): clean up handling of default/dummy partition .. IMPALA-7141 (part 1): clean up handling of default/dummy partition Currently, HdfsTable inconsistently uses the term "default partition" to refer to two different concepts: 1) For unpartitioned tables, a single partition with ID 1 and no partition keys is created and added to the partition map. 2) All tables have an additional partition added with partition ID -1 which acts as a sort of prototype for partition creation: when new partitions are created during an INSERT operation, the file format and other related options are copied out of this special partition. This partition is inconsistently referred to as either the "default partition" or the "dummy partition". The handling of this second case (the partition with id -1) was somewhat messy: - the partition shows up in the partitionMap_ member, but does not show up in the partitionIds_ member. - almost all of the call sites that iterate through the partitions of an HdfsTable instance ended up skipping over the dummy partition. - several call sites called getPartitions().size() but then had to adjust the result by subtracting one in order to actually count the number of partitions in a table. - similarly, test assertions had to assert that tables with 24 partitions had an expected partition map size of 25. In order to address the above, this patch makes the following changes: - getPartitions() and getPartitionMap() no longer include the dummy partition. This removes a bunch of special case checks to skip over the dummy partition or to adjust partition counts based on it. - to clarify the purpose of this partition, references to it are renamed to "prototype partition" instead of "default partition". - when converting the HdfsTable to/from Thrift, the prototype partition is included in its own field in the struct, instead of being stuffed into the same map with the true partitions of the table. This reflects the fact that this partition is special (eg missing fields like 'location' which otherwise are required for real partitions) This change should should be entirely internal with no functional differences. As such, the only testing changes are some fixes for assertions on the Thrift serialized structures and other internals. Change-Id: I15e91b50eb7c2a5e0bac8c33d603d6cd8cbaca2e --- M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M common/thrift/CatalogObjects.thrift M common/thrift/ImpalaInternalService.thrift M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java 18 files changed, 156 insertions(+), 202 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/10711/1 -- To view, visit http://gerrit.cloudera.org:8080/10711 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I15e91b50eb7c2a5e0bac8c33d603d6cd8cbaca2e Gerrit-Change-Number: 10711 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10630 to look at the new patch set (#6). Change subject: IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog .. IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog This adds support for loading the Table object from HMS and parsing out the Column and partitioning information from it. With this change, I'm able to connect to an impalad running in "local catalog" mode and run DESCRIBE, DESCRIBE EXTENDED, and SHOW CREATE TABLE commands. Other commands like SHOW PARTITIONS don't work properly yet, and type-specific table functionality (eg views, HBase tables, etc) are not yet supported. Again a simple unit test is included to check that column information is loaded. More thorough testing is deferred until we've reached enough coverage that we can start running e2e tests against a cluster running in "local" mode. Change-Id: I640f27e36198955e057da62a3ce25a858406e496 --- A fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalogException.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 9 files changed, 300 insertions(+), 79 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/10630/6 -- To view, visit http://gerrit.cloudera.org:8080/10630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I640f27e36198955e057da62a3ce25a858406e496 Gerrit-Change-Number: 10630 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7165: [DOCS] Correct example for dynamic partition pruning
Henry Robinson has posted comments on this change. ( http://gerrit.cloudera.org:8080/10703 ) Change subject: IMPALA-7165: [DOCS] Correct example for dynamic partition pruning .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 Gerrit-Change-Number: 10703 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 23:09:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 ) Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading .. Patch Set 5: (9 comments) http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@136 PS5, Line 136: hdfs > nit: fsTable, or just table Done http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@141 PS5, Line 141: assert > precondition it's not really a precondition, though - just an internal sanity check. I reorganized the method a bit and had it throw AssertionError so it's always enabled. http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@120 PS5, Line 120: @return > nit: Returns Done http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@138 PS5, Line 138: @return > nit: Returns Done http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java File fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@30 PS5, Line 30: complet > nit: complete Done http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@35 PS5, Line 35: @return > nit: Returns Done http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@41 PS5, Line 41: @return > nit: Returns Done http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java File fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@a158 PS5, Line 158: > any cleanup/simplification is a good thing. hopefully this is not one of th didn't end up being a huge patch. Assuming it passes the exhaustive tests I think it won't be too much yak-shaving http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@101 PS5, Line 101: If 'allowEmpty' is False, empty partitions are not returned. > heh... how many negations are here? :) -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 22:58:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10630 ) Change subject: IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/10630/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java: http://gerrit.cloudera.org:8080/#/c/10630/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@62 PS5, Line 62: fullly > nit: fully Done http://gerrit.cloudera.org:8080/#/c/10630/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@87 PS5, Line 87: @param > lets get rid of these (or at least, not add new ones). numClusteringCols is Done http://gerrit.cloudera.org:8080/#/c/10630/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@109 PS5, Line 109: FeCatalogUtils. > remove Done http://gerrit.cloudera.org:8080/#/c/10630/5/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java File fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java: http://gerrit.cloudera.org:8080/#/c/10630/5/fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java@191 PS5, Line 191: table-level stats > I see that these are constructed lazily when accessed (L177) whereas other no reason, I'll change that to be eager as well. -- To view, visit http://gerrit.cloudera.org:8080/10630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I640f27e36198955e057da62a3ce25a858406e496 Gerrit-Change-Number: 10630 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 22:50:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7165: [DOCS] Correct example for dynamic partition pruning
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10703 ) Change subject: IMPALA-7165: [DOCS] Correct example for dynamic partition pruning .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/10703/1/docs/shared/impala_common.xml File docs/shared/impala_common.xml: http://gerrit.cloudera.org:8080/#/c/10703/1/docs/shared/impala_common.xml@1873 PS1, Line 1873: STORED AS PARQUET > This doesn't depend on file format, so that can be dropped as well. Done http://gerrit.cloudera.org:8080/#/c/10703/1/docs/shared/impala_common.xml@1878 PS1, Line 1878: PARTITIONED > this table doesn't need to be partitioned. It would make sense not to have Done http://gerrit.cloudera.org:8080/#/c/10703/1/docs/shared/impala_common.xml@1884 PS1, Line 1884: SUMMARY; -- Shows 2 rows in yy were pruned, and 3 out of 5 rows were scanned. > I think it's helpful to show the plan for the SELECT query. It shows that r Done http://gerrit.cloudera.org:8080/#/c/10703/1/docs/shared/impala_common.xml@1884 PS1, Line 1884: SUMMARY > agreed that explain and an extract from "profile;" (not summary) would be m Done -- To view, visit http://gerrit.cloudera.org:8080/10703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 Gerrit-Change-Number: 10703 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 22:15:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5168: Codegen HASH PARTITIONED KrpcDataStreamSender::Send()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10421 ) Change subject: IMPALA-5168: Codegen HASH_PARTITIONED KrpcDataStreamSender::Send() .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2655/ -- To view, visit http://gerrit.cloudera.org:8080/10421 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1c44cc9312c062cc7a5a4ac9156ceaa31fb887ff Gerrit-Change-Number: 10421 Gerrit-PatchSet: 7 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 22:16:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7165: [DOCS] Correct example for dynamic partition pruning
Hello Henry Robinson, Gabor Kaszab, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10703 to look at the new patch set (#2). Change subject: IMPALA-7165: [DOCS] Correct example for dynamic partition pruning .. IMPALA-7165: [DOCS] Correct example for dynamic partition pruning Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 --- M docs/shared/impala_common.xml M docs/topics/impala_partitioning.xml M docs/topics/impala_runtime_filtering.xml 3 files changed, 92 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/10703/2 -- To view, visit http://gerrit.cloudera.org:8080/10703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 Gerrit-Change-Number: 10703 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10627 ) Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2660/ -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 21:59:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7165: [DOCS] Correct example for dynamic partition pruning
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10703 ) Change subject: IMPALA-7165: [DOCS] Correct example for dynamic partition pruning .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/10703/1/docs/shared/impala_common.xml File docs/shared/impala_common.xml: http://gerrit.cloudera.org:8080/#/c/10703/1/docs/shared/impala_common.xml@1873 PS1, Line 1873: STORED AS PARQUET This doesn't depend on file format, so that can be dropped as well. http://gerrit.cloudera.org:8080/#/c/10703/1/docs/shared/impala_common.xml@1884 PS1, Line 1884: SUMMARY agreed that explain and an extract from "profile;" (not summary) would be more helpful. -- To view, visit http://gerrit.cloudera.org:8080/10703 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I44d1054f55d3dc7947ccf4c2ef440e506c41f963 Gerrit-Change-Number: 10703 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Henry Robinson Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 21:55:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 14: I stopped the build. -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 21:47:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10709 ) Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 21:38:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 ) Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java File fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@a158 PS5, Line 158: > hm, the issue I was facing wasn't with this, so much as with the assertions any cleanup/simplification is a good thing. hopefully this is not one of those threads that unravels too many other things. -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 21:28:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 14: (1 comment) Just skimming through the patch, have a quick question. Is the plan to do it for all statements? For ex, InsertStmt is not covered? http://gerrit.cloudera.org:8080/#/c/10571/14/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java File fe/src/main/java/org/apache/impala/analysis/UnionStmt.java: http://gerrit.cloudera.org:8080/#/c/10571/14/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java@562 PS14, Line 562: withClause_ what about this? Does the following work? with t1 as (select 1+1) select * from t1 union select * from foo -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 21:27:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3816: (prep) Move TupleSorter to sorter-ir.cc
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10679 ) Change subject: IMPALA-3816: (prep) Move TupleSorter to sorter-ir.cc .. Patch Set 3: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaaf2b75c2f789002c42939865c018f728d29a113 Gerrit-Change-Number: 10679 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 21:26:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Hello Lars Volker, Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10709 to look at the new patch set (#3). Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. IMPALA-7111: avoid use of boost::split in CheckPluginEnabled This is an attempt to either avoid the bug or make it easier to diagnose if it reoccurs. My suspicion is that somehow boost::split() is accessing the input string in a non-thread-safe manner, but the implementation is opaque enough that it's not obvious how. Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 --- M be/src/exec/hdfs-plugin-text-scanner.cc M be/src/util/string-util-test.cc M be/src/util/string-util.cc M be/src/util/string-util.h 4 files changed, 48 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/10709/3 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc
File be/src/exec/hdfs-plugin-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc@107
PS1, Line 107: if
(!CommaSeparatedContains(FLAGS_enabled_hdfs_text_scanner_plugins, plugin_name))
{
> Up to you, but since this isn't performance sensitive, you could use split.
I figured it would be good to avoid allocating any intermediate strings in case
this doesn't fix the issue and we need to interpret more ASAN use-after-free
warnings.
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc
File be/src/util/string-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@83
PS1, Line 83: // Basic tests with string present.
> Test explicitly that this doesn't whitespace-trim the entries. I don't reme
It doesn't - you have to explicitly specify the characters to split on.
I added a test at the bottom.
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@84
PS1, Line 84: EXPECT_TRUE(CommaSeparatedContains("LZO", "LZO"));
> Should we have a test with different length in string and pattern, eg. "PLU
Done
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@90
PS1, Line 90: EXPECT_FALSE(CommaSeparatedContains("", "LZO"));
> Nit: also test "," and ",,"?
Done
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:25:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2659/ -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 21:22:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 21:22:28 + Gerrit-HasComments: No
[native-toolchain-CR] WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 4: > Patch Set 4: > > (1 comment) > > > Patch Set 4: > > > > The build worked on newer linux distros but failed on older ones when > > building flat buffers: > > > > > > 22:40:05 ./flatc: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version > > `GLIBCXX_3.4.17' not found (required by ./flatc) > > It might be because it's not using the binutil in native-toolchain Sent a random inline comment out. You can ignore it for now -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 21:18:51 + Gerrit-HasComments: No
[native-toolchain-CR] WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tianyi Wang has posted comments on this change. (
http://gerrit.cloudera.org:8080/6521 )
Change subject: WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
..
Patch Set 4:
(1 comment)
> Patch Set 4:
>
> The build worked on newer linux distros but failed on older ones when
> building flat buffers:
>
>
> 22:40:05 ./flatc: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version
> `GLIBCXX_3.4.17' not found (required by ./flatc)
It might be because it's not using the binutil in native-toolchain
http://gerrit.cloudera.org:8080/#/c/6521/4/functions.sh
File functions.sh:
http://gerrit.cloudera.org:8080/#/c/6521/4/functions.sh@490
PS4, Line 490: if ! local required_libs=$(objdump -p "$file" 2>/dev/null |
grep NEEDED | awk '{print $2}')
long line
--
To view, visit http://gerrit.cloudera.org:8080/6521
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67
Gerrit-Change-Number: 6521
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:17:22 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 ) Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java File fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@a158 PS5, Line 158: > this will have that default partition whereas matchingPatitionIds will not hm, the issue I was facing wasn't with this, so much as with the assertions in CatalogTest and with EXPLAIN output where we assume that 'getPartitionsMap()' returns an extra partition. I found the whole "dummy partition" thing really confusing, with special cases in ~10-15 places around the code to skip over them, so I took a whack at a patch to clean that stuff up a bit. I'll put that patch in order before this one in the patch series. Let me know what you think. -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 21:14:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
..
Patch Set 1:
(2 comments)
Thanks for looking into this. I noticed that we also have gutil/strings/split.h
in our sources. Should we use that for a simpler implementation? I don't feel
strongly about it and am ok with the custom code, too.
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc
File be/src/util/string-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@84
PS1, Line 84: EXPECT_TRUE(CommaSeparatedContains("LZO", "LZO"));
Should we have a test with different length in string and pattern, eg.
"PLUGIN2", "LZO" and "A,B,C", "PLUGIN2"?
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@90
PS1, Line 90: EXPECT_FALSE(CommaSeparatedContains("", "LZO"));
Nit: also test "," and ",,"?
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:11:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Philip Zeyliger has posted comments on this change. (
http://gerrit.cloudera.org:8080/10709 )
Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc
File be/src/exec/hdfs-plugin-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/exec/hdfs-plugin-text-scanner.cc@107
PS1, Line 107: if
(!CommaSeparatedContains(FLAGS_enabled_hdfs_text_scanner_plugins, plugin_name))
{
Up to you, but since this isn't performance sensitive, you could use split.h
from gutil.
I think the following is similar.
vector x = strings::Split(FLAGS_..., ",");
if (x.find(plugin_name) != ...::npos) { ...
}
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc
File be/src/util/string-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/10709/1/be/src/util/string-util-test.cc@83
PS1, Line 83: // Basic tests with string present.
Test explicitly that this doesn't whitespace-trim the entries. I don't remember
if boost:split does or doesn't trim.
--
To view, visit http://gerrit.cloudera.org:8080/10709
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8
Gerrit-Change-Number: 10709
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Wed, 13 Jun 2018 21:11:38 +
Gerrit-HasComments: Yes
[native-toolchain-CR] WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 4: The build worked on newer linux distros but failed on older ones when building flat buffers: 22:40:05 ./flatc: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.17' not found (required by ./flatc) -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 21:07:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10631 ) Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading .. Patch Set 5: (9 comments) http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java File fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@136 PS5, Line 136: hdfs nit: fsTable, or just table http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java@141 PS5, Line 141: assert precondition http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@120 PS5, Line 120: @return nit: Returns http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@138 PS5, Line 138: @return nit: Returns http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java File fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@30 PS5, Line 30: complet nit: complete http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@35 PS5, Line 35: @return nit: Returns http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java@41 PS5, Line 41: @return nit: Returns http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java File fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java: http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@a158 PS5, Line 158: this will have that default partition whereas matchingPatitionIds will not have it since getPatitionIds (L150), which is the only place I see here it can sneak in. why not add qualifying partitions as done here instead of getting all and filtering? http://gerrit.cloudera.org:8080/#/c/10631/5/fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java@101 PS5, Line 101: If 'allowEmpty' is False, empty partitions are not returned. heh... how many negations are here? -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 21:06:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4908: NULL floats don't compare equal to other NULL floats
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10707 ) Change subject: IMPALA-4908: NULL floats don't compare equal to other NULL floats .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2658/ -- To view, visit http://gerrit.cloudera.org:8080/10707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7310645e5752d8203be5abc22a6562a59b6e975 Gerrit-Change-Number: 10707 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 21:05:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4908: NULL floats don't compare equal to other NULL floats
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10707 ) Change subject: IMPALA-4908: NULL floats don't compare equal to other NULL floats .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7310645e5752d8203be5abc22a6562a59b6e975 Gerrit-Change-Number: 10707 Gerrit-PatchSet: 2 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 21:05:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4908: NULL floats don't compare equal to other NULL floats
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10707 ) Change subject: IMPALA-4908: NULL floats don't compare equal to other NULL floats .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7310645e5752d8203be5abc22a6562a59b6e975 Gerrit-Change-Number: 10707 Gerrit-PatchSet: 1 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 21:05:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7111: avoid use of boost::split in CheckPluginEnabled
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10709 Change subject: IMPALA-7111: avoid use of boost::split in CheckPluginEnabled .. IMPALA-7111: avoid use of boost::split in CheckPluginEnabled This is an attempt to either avoid the bug or make it easier to diagnose if it reoccurs. My suspicion is that somehow boost::split() is accessing the input string in a non-thread-safe manner, but the implementation is opaque enough that it's not obvious how. Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 --- M be/src/exec/hdfs-plugin-text-scanner.cc M be/src/util/string-util-test.cc M be/src/util/string-util.cc M be/src/util/string-util.h 4 files changed, 38 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/10709/1 -- To view, visit http://gerrit.cloudera.org:8080/10709 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I17b7f083731a33b832035f24900e351e2cb2feb8 Gerrit-Change-Number: 10709 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 20:58:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2657/ -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 20:58:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 20:51:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/10571/12/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/10571/12/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@940 PS12, Line 940: // Return the SQL string before inline-view expression substitution. > It's still here Oops. I forgot to commit the change for this file. Done. -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 13 Jun 2018 20:50:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. IMPALA-7106: Print rewritten SQL correctly when log trace is enabled toSql() method is used to print SQL string that is close to the original SQL string, which is something that users want to see. When debugging issues related to SQL rewrites, it can be very useful to be able to print/get the SQL string that is being rewritten. This patch adds a new method toSql(boolean rewritten) to get the rewritten SQL string. The LOG.trace statement that prints the rewritten SQL is also updated to use toSql(true). Testing: - Added FE test for the rewritten SQL string - Ran all FE tests Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java 10 files changed, 105 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10571/13 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6835: Add table name and node id to Kudu scanner errors
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10671 ) Change subject: IMPALA-6835: Add table name and node id to Kudu scanner errors .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 Gerrit-Change-Number: 10671 Gerrit-PatchSet: 4 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 13 Jun 2018 20:49:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6835: Add table name and node id to Kudu scanner errors
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10671 ) Change subject: IMPALA-6835: Add table name and node id to Kudu scanner errors .. IMPALA-6835: Add table name and node id to Kudu scanner errors Previously, the error messages in KuduScanner only contained the reason for failure. They did not contain the KuduTable name or the TPlanNode id which made it inconveient to debug. This change adds the TPlanNode id to all error messages and the KuduTable name whenever applicable. This change was manually tested by explicitly returning failure while scanning kudu tables. Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 Reviewed-on: http://gerrit.cloudera.org:8080/10671 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/kudu-scanner.cc M be/src/exec/kudu-scanner.h M be/src/exec/kudu-util.h 3 files changed, 27 insertions(+), 14 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 Gerrit-Change-Number: 10671 Gerrit-PatchSet: 5 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10571 )
Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is
enabled
..
Patch Set 12:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10571/11/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/11/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@941
PS11, Line 941: return toSql(false);
> Remove because the implementation is the same as in StatementBase
Done
http://gerrit.cloudera.org:8080/#/c/10571/11/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java
File fe/src/main/java/org/apache/impala/analysis/UnionStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/11/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java@554
PS11, Line 554: public String toSql(boolean rewritten) {
> Remove because the implementation is the same as in StatementBase
Done
--
To view, visit http://gerrit.cloudera.org:8080/10571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53
Gerrit-Change-Number: 10571
Gerrit-PatchSet: 12
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 13 Jun 2018 20:41:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. IMPALA-7106: Print rewritten SQL correctly when log trace is enabled toSql() method is used to print SQL string that is close to the original SQL string, which is something that users want to see. When debugging issues related to SQL rewrites, it can be very useful to be able to print/get the SQL string that is being rewritten. This patch adds a new method toSql(boolean rewritten) to get the rewritten SQL string. The LOG.trace statement that prints the rewritten SQL is also updated to use toSql(true). Testing: - Added FE test for the rewritten SQL string - Ran all FE tests Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java 10 files changed, 108 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10571/12 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 12 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-4908: NULL floats don't compare equal to other NULL floats
Pooja Nilangekar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10707 Change subject: IMPALA-4908: NULL floats don't compare equal to other NULL floats .. IMPALA-4908: NULL floats don't compare equal to other NULL floats This change ensures that comparing two NULL floats with different val fields returns true. Although this is undefined behavior, it is now consistent with other types. Along with the change, a unit test was added to ensure that equality checking of floats returns results as expected. Change-Id: Ie7310645e5752d8203be5abc22a6562a59b6e975 --- M be/src/udf/udf-test.cc M be/src/udf/udf.h 2 files changed, 28 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/10707/1 -- To view, visit http://gerrit.cloudera.org:8080/10707 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie7310645e5752d8203be5abc22a6562a59b6e975 Gerrit-Change-Number: 10707 Gerrit-PatchSet: 1 Gerrit-Owner: Pooja Nilangekar
