[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 05 Sep 2019 04:22:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. IMPALA-8915: reapply IMPALA-8901 fix This reapplies the below fix and resolves conflicts correctly. IMPALA-8901: Fix # links on /catalog page. After IMPALA-7935, the database links on the top of /catalog page that use # links to jump to the part of the page corresponding to the particular database is broken because the id is ommitted. This is fixed by moving id to a tag which doesn't change if local catalog is turned on. Testing: Manually tested the links are working in both local catalog mode and V1 mode. Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Reviewed-on: http://gerrit.cloudera.org:8080/14176 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M www/catalog.tmpl 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4466/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 12 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Sep 2019 03:20:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14134 to look at the new patch set (#12). Change subject: IMPALA-8797: Support database and table blacklist .. IMPALA-8797: Support database and table blacklist Add catalogd and coordinator startup options for database and table blacklist. Blacklisted dbs/tables will be skipped in loading. Users won't see them when getting database/table list, e.g. in SHOW DATABASES/TABLES. Dropping/creating blacklisted databases/tables/views are not allowed too. Implementation: Catalogd and coordinators parses the --blacklisted_dbs and --blacklisted_tables options in startup. Blacklist checks are added in catalogd when loading the metadata and in coordinators when analysing DDL requests for create/drop. Catalogd will also check blacklist when executing DDL requests from coordinators in case their blacklists are inconsistent. Motivation: By default, it's used to blacklist "sys" and "information_schema" databases from Hive. Admin can use them to specify any databases/tables that are not suitable for Impala to query. Tests: - Add java unit tests for blacklist related functions - Add a custom cluster test: test_blacklisted_dbs_and_tables.py - Ran CORE tests Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java A fe/src/test/java/org/apache/impala/util/CatalogBlacklistUtilsTest.java A tests/custom_cluster/test_blacklisted_dbs_and_tables.py 14 files changed, 542 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14134/12 -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 12 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/14134 )
Change subject: IMPALA-8797: Support database and table blacklist
..
Patch Set 10:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/analysis/TableName.java
File fe/src/main/java/org/apache/impala/analysis/TableName.java:
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/analysis/TableName.java@81
PS10, Line 81: Preconditions.checkNotNull(db_);
> Like I mentioned in the other comment, Preconditions.checkState(isFullyQual
Done
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java
File fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java:
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java@104
PS10, Line 104: if (BLACKLISTED_TABLES.contains(table)) {
> Ah, I see what you are saying. I think thats a deep refactor. Suggested a n
Sure.
--
To view, visit http://gerrit.cloudera.org:8080/14134
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc
Gerrit-Change-Number: 14134
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 05 Sep 2019 02:35:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14134 )
Change subject: IMPALA-8797: Support database and table blacklist
..
Patch Set 10: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/analysis/TableName.java
File fe/src/main/java/org/apache/impala/analysis/TableName.java:
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/analysis/TableName.java@81
PS10, Line 81: Preconditions.checkNotNull(db_);
Like I mentioned in the other comment,
Preconditions.checkState(isFullyQualified())?
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java
File fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java:
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java@104
PS10, Line 104: if (BLACKLISTED_TABLES.contains(table)) {
> But there are no "analyzer" object here... Even if we move this "verifyTabl
Ah, I see what you are saying. I think thats a deep refactor. Suggested a nit
in the Preconditions check.
--
To view, visit http://gerrit.cloudera.org:8080/14134
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc
Gerrit-Change-Number: 14134
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 05 Sep 2019 02:29:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Thu, 05 Sep 2019 02:27:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. IMPALA-8907: TestResultSpooling.test_slow_query is flaky De-flake test_result_spooling.py::TestResultsSpooling::test_slow_query by increasing the delay in RowBatch production. This patch makes two fixes to ensure that RowBatchGetWaitTime is a non-zero value: * Add the DELAY DEBUG_ACTION to the SCAN_NODE rather than the EXCHANGE_NODE. Since the EXCHANGE_NODE only processes a few rows, adding the delay to the SCAN_NODE decreases the rate at which results are produced. * Wait for all rows to be fetched before checking if RowBatchGetWaitTime is in the profile. This fixes a possible race condition where the fetch thread was not able to issue the fetch RPC before the test checked if RowBatchGetWaitTime was in the runtime profile. Testing: * Looped test_slow_query for several hours with 0 failures Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Reviewed-on: http://gerrit.cloudera.org:8080/14170 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_result_spooling.py 1 file changed, 4 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/14134 )
Change subject: IMPALA-8797: Support database and table blacklist
..
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java
File fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java:
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java@104
PS10, Line 104: if (BLACKLISTED_TABLES.contains(table)) {
> Can we just do TableName temp = analyzer.getFqTableName(table);
But there are no "analyzer" object here... Even if we move this
"verifyTableName" function into Analyzer, there are no analyzer objects in
TableName.analyze()... We still need to call "getFqTableName" in somewhere with
the analyzer object...
--
To view, visit http://gerrit.cloudera.org:8080/14134
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc
Gerrit-Change-Number: 14134
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 05 Sep 2019 00:52:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14134 )
Change subject: IMPALA-8797: Support database and table blacklist
..
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java
File fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java:
http://gerrit.cloudera.org:8080/#/c/14134/10/fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java@104
PS10, Line 104: if (BLACKLISTED_TABLES.contains(table)) {
Can we just do TableName temp = analyzer.getFqTableName(table);
if (BLACKLISTED_TABLES.contains(temp)) {
.
That saves all the callers from doing
analyzer.getFqTableName(tableName_).analyze();
--
To view, visit http://gerrit.cloudera.org:8080/14134
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc
Gerrit-Change-Number: 14134
Gerrit-PatchSet: 10
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 05 Sep 2019 00:46:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4465/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 05 Sep 2019 00:43:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4464/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Sep 2019 00:29:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8572: Log query events before Unregister() call.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14143 ) Change subject: IMPALA-8572: Log query events before Unregister() call. .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/14143/4//COMMIT_MSG Commit Message: PS4: > I had a couple of high level questions. 1) I've spent some time looking at it, especially reading git blame and older jiras, but I didn't totally understand why a fetch was necessary. I didn't see an obvious downside either. So I ended up preserving the existing functionality. 2) Isn't that a deadlock? FetchInternal() blocks on Wait(), Wait() blocks on LogQueryEvents() which blocks on first fetch. Am I missing something? -- To view, visit http://gerrit.cloudera.org:8080/14143 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I639b9c1acb9806b29292cd85be2863688453ca2e Gerrit-Change-Number: 14143 Gerrit-PatchSet: 4 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 05 Sep 2019 00:18:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 05 Sep 2019 00:14:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4881/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Thu, 05 Sep 2019 00:14:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14134 to look at the new patch set (#10). Change subject: IMPALA-8797: Support database and table blacklist .. IMPALA-8797: Support database and table blacklist Add catalogd and coordinator startup options for database and table blacklist. Blacklisted dbs/tables will be skipped in loading. Users won't see them when getting database/table list, e.g. in SHOW DATABASES/TABLES. Dropping/creating blacklisted databases/tables/views are not allowed too. Implementation: Catalogd and coordinators parses the --blacklisted_dbs and --blacklisted_tables options in startup. Blacklist checks are added in catalogd when loading the metadata and in coordinators when analysing DDL requests for create/drop. Catalogd will also check blacklist when executing DDL requests from coordinators in case their blacklists are inconsistent. Motivation: By default, it's used to blacklist "sys" and "information_schema" databases from Hive. Admin can use them to specify any databases/tables that are not suitable for Impala to query. Tests: - Add java unit tests for blacklist related functions - Add a custom cluster test: test_blacklisted_dbs_and_tables.py - Ran CORE tests Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java A fe/src/test/java/org/apache/impala/util/CatalogBlacklistUtilsTest.java A tests/custom_cluster/test_blacklisted_dbs_and_tables.py 14 files changed, 542 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14134/10 -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 10 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/14134 )
Change subject: IMPALA-8797: Support database and table blacklist
..
Patch Set 9:
(1 comment)
> Patch Set 9:
>
> (3 comments)
>
> lgtm, can +2 it once the TableName#analyze() is refactored to include the
> blacklist checks.
Sure. Just refactored in it.
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java
File fe/src/main/java/org/apache/impala/analysis/TableName.java:
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java@86
PS8, Line 86: throw new AnalysisException("Invalid table/view name: " +
tbl_);
> I think we can add the getFqTableName() logic into BlackListUtils.verifyTab
OK, agree with that. Avoiding possibility for bugs is more important.
--
To view, visit http://gerrit.cloudera.org:8080/14134
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc
Gerrit-Change-Number: 14134
Gerrit-PatchSet: 9
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Thu, 05 Sep 2019 00:01:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14157 ) Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc File be/src/exec/buffered-plan-root-sink.cc: http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@159 PS4, Line 159: > Don't need _ suffix on local variable. Done http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@172 PS4, Line 172: // wait_timeout_timer ensures the client only ever waits up to > Why not just start the timer and leave it running? I think the elapsed time Yeah makes sense. Moved the Start before the lock is acquired, and it is never stopped. This way it tracks the total time required to fetch, including waiting for rows to become available and waiting for rows to materialize. I have some additional tests in mind for this, but they require IMPALA-8825 first. http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@174 PS4, Line 174: uint64_t wait_duration = max((uint64_t) 1, > nit: could be a one liner: Done -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 04 Sep 2019 23:54:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14157 to look at the new patch set (#5). Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC .. IMPALA-7312: Non-blocking mode for Fetch() RPC Adds the query option FETCH_ROWS_TIMEOUT_MS to control the client timeout when fetching rows. Set to 10 seconds by default to avoid unnecessary fetch requests. Timeout applies when result spooling is enabled or disabled. When result spooling is disabled, the timeout controls how long the client thread will wait for a single RowBatch to be produced by the coordinator fragment. When result spooling is enabled, a client can fetch multiple RowBatches at a time, so the timeout controls the total time spent waiting for RowBatches to be produced. Testing: * Added new tests to test_fetch.py * Running core tests Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d --- M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/hs2/test_fetch.py 9 files changed, 161 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14157/5 -- To view, visit http://gerrit.cloudera.org:8080/14157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d Gerrit-Change-Number: 14157 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add observability of the query load on executor groups .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4463/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 04 Sep 2019 22:33:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add observability of the query load on executor groups .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4462/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 04 Sep 2019 22:26:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4880/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 04 Sep 2019 22:21:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 04 Sep 2019 22:21:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Hello Andrew Sherman, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14103 to look at the new patch set (#8). Change subject: IMPALA-8858: Add observability of the query load on executor groups .. IMPALA-8858: Add observability of the query load on executor groups With this patch, all executor groups with at least one executor will have a metric added that display the number of queries (admitted by the local coordinator) running on them. The metric is removed only when the group has no executors or no longer exists. It gets updated when either the cluster membership changes or a query gets admitted or released by the admission controller. Also adds the ability to delete metrics from a metric group after registration. Testing: Added a custom cluster test and a BE metric test. Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_executor_groups.py 12 files changed, 265 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14103/8 -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/14103 )
Change subject: IMPALA-8858: Add observability of the query load on executor
groups
..
Patch Set 7:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.h
File be/src/scheduling/admission-controller.h:
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.h@943
PS6, Line 943: /// Updates the list of executor groups for which we maintain
the query load metrics.
> "groups for which we maintain..."
Done
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.h@944
PS6, Line 944: from the metric gr
> Should we consider keeping groups with >0 executors around, even if they're
With how the current cancellation logic works, the query is still counted as
running against a backend by the impala-server till it gets un-registered,
which means that it would be cancelled regardless of whether it has completed
its fragments on a particular backend.
However, I still agree with keeping non-zero groups around since they can still
have an intermediate non zero value for query load which can prove useful.
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.cc@320
PS6, Line 320: dequeue_thread_->Join();
> Can the argument be a const&?
changed this mechanism, switched to sending a snapshotPtr instead.
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/admission-controller.cc@1852
PS6, Line 1852: for (const auto& group : snapshot->executor_groups) {
> I feel that this function's body could benefit from a comment explaining wh
Done
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/cluster-membership-mgr.h
File be/src/scheduling/cluster-membership-mgr.h:
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/scheduling/cluster-membership-mgr.h@135
PS6, Line 135: /// events. They may be called at any time before or after
calling Init() and are
> The comment should include what gets passed into the callback, or alternati
changed the update mechanism
http://gerrit.cloudera.org:8080/#/c/14103/3/be/src/scheduling/cluster-membership-mgr.cc
File be/src/scheduling/cluster-membership-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/14103/3/be/src/scheduling/cluster-membership-mgr.cc@342
PS3, Line 342: base_snapshot = current_membership_.get();
> I'm still in favor of this.
Done.
Note: this adds a little extra work to the critical path since now we cant
optimize work for specific calls:
- NotifyLocalServerForDeletedBackend called every time and not just when
update_local_server is true
- Admission controller needs to separately create a set a groups instead of
leveraging the set previously created during metric updates.
These are pretty minor + cluster changes would be way less frequent in practice
so shouldn't matter much.
http://gerrit.cloudera.org:8080/#/c/14103/3/be/src/scheduling/cluster-membership-mgr.cc@346
PS3, Line 346: // copying the Snapshot if it isn't.
> ?
Done
http://gerrit.cloudera.org:8080/#/c/14103/3/be/src/scheduling/cluster-membership-mgr.cc@551
PS3, Line 551:
:
:
> What was the reason for this change?
just seemed consistent with other update calls to send the whole snapshotPtr
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/util/metrics.h
File be/src/util/metrics.h:
http://gerrit.cloudera.org:8080/#/c/14103/6/be/src/util/metrics.h@455
PS6, Line 455: typedef std::unordered_map> MetricMap;
> Is there a reason this map needs to be ordered? If so, can you add it to th
oops, didnt notice that a map was being used instead of an unordered_map. Not
sure what the initial intention was but I dont see any reason why it would
require an ordered map. changing it to unordered_map. Same goes for
ChildGroupMap below.
http://gerrit.cloudera.org:8080/#/c/14103/6/common/thrift/metrics.json
File common/thrift/metrics.json:
http://gerrit.cloudera.org:8080/#/c/14103/6/common/thrift/metrics.json@2495
PS6, Line 2495: "description": "Total number of queries running on executor
group: $0",
> I'd suggest moving the $0 to the back of the string to make it slightly eas
makes sense to me. Moving it to the end.
http://gerrit.cloudera.org:8080/#/c/14103/6/common/thrift/metrics.json@2502
PS6, Line 2502: ad.$0"
do you think num-running-queries is more appropriate here?
--
To view, visit http://gerrit.cloudera.org:8080/14103
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539
Gerrit-Change-Number: 14103
Gerrit-PatchSet: 7
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Andrew
[Impala-ASF-CR] IMPALA-8858: Add observability of the query load on executor groups
Hello Andrew Sherman, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14103 to look at the new patch set (#7). Change subject: IMPALA-8858: Add observability of the query load on executor groups .. IMPALA-8858: Add observability of the query load on executor groups With this patch, all executor groups with at least one executor will have a metric added that display the number of queries (admitted by the local coordinator) running on them. The metric is removed only when the group has no executors or no longer exists. It gets updated when either the cluster membership changes or a query gets admitted or released by the admission controller. Also adds the ability to delete metrics from a metric group after registration. Testing: Added a custom cluster test and a BE metric test. Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 --- M be/src/runtime/exec-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/metrics-test.cc M be/src/util/metrics.cc M be/src/util/metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_executor_groups.py 12 files changed, 264 insertions(+), 135 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14103/7 -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-7984: Port runtime filter from Thrift RPC to KRPC
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/13882 )
Change subject: IMPALA-7984: Port runtime filter from Thrift RPC to KRPC
..
Patch Set 17:
(15 comments)
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.h
File be/src/runtime/coordinator.h:
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.h@48
PS17, Line 48: using kudu::rpc::RpcContext;
don't use 'using' in header files
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc@986
PS17, Line 986: {
: return;
: }
You can leave the formatting as-is
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc@1041
PS17, Line 1041: void
Coordinator::SetupSidecarForBloomFilter(PublishFilterParamsPB* rpc_params,
Its unfortunate that this logic is duplicated between here and
BloomFilter::ToProtobuf, eg. because duplicating code makes it more likely
there are bugs (for example I think that you have a bug here in the case where
AddOutboundSidecar fails, since you don't disable the bloom filter, even though
that bug is already fixed in BloomFilter::ToProtobuf)
There are a couple of possible solutions to this:
1. Move this to be a static function in BloomFilter, eg.
BloomFilter::AddSidecar(), and call it from BloomFilter::ToProtobuf()
2. Instead of having Coordinator::FilterState store a BloomFilterPB and 'string
directory' just actually reconstruct a BloomFilter object, i.e. around line
1110 below, by calling BloomFilter::Init() with sidecar_slice.data(). Then we
would be able to just call the existing BloomFilter::ToProtobuf()
3. some other idea I didn't think of
Option 2 seems cleaner overall to me, since it allows us to better hide the
details of how bloom filters work in the BloomFilter class instead of leaking
things like the existence of the 'directory' string into FilterState.
However, its slightly less efficient (cause we add a transition from
BloomFilterPB -> Bloom Filter object and back, with the required setup, eg. in
BloomFilter::Init()), though not a ton less efficient since I think it can be
done without adding any copies of the directory, and it probably also is more
work for you in terms of code changes (eg. BloomFilter::Or would have to be
slightly rewritten to take BloomFilter objects instead).
And, option 1 is consistent with the way things already work. So, I'll leave it
up to you to decide.
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc@1089
PS17, Line 1089: !params->bloom_filter().always_true() &&
: !params->bloom_filter().always_false())
Might be cleaner to check has_directory_sidecar_idx() here, which should be
equivalent
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/coordinator.cc@1096
PS17, Line 1096: return;
You need to handle this error, i.e. call Disable()
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.h
File be/src/runtime/runtime-filter-bank.h:
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.h@52
PS17, Line 52: class UniqueIdPB;
I don't think this is used anywhere
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.h@142
PS17, Line 142: void SendFilterToCoordinatorCompleteCb(const
kudu::rpc::RpcController* rpc_controller);
I think this can be private.
Lets also rename it UpdateFilterCompleteCb, to keep it consistent with the name
of the rpc.
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.h@154
PS17, Line 154: num_krpcs_
num_inflight_rpcs_
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.cc
File be/src/runtime/runtime-filter-bank.cc:
http://gerrit.cloudera.org:8080/#/c/13882/17/be/src/runtime/runtime-filter-bank.cc@133
PS17, Line 133: DCHECK_NE(state_->query_options().runtime_filter_mode,
TRuntimeFilterMode::OFF)
While thinking about the locking protocol in Close(), I noticed that we don't
check 'closed_' here like we do at the start of PublishGlobalFilter()
I spent a little time looking at the code, and I think that its correct as is,
because RuntimeFilterBank::Close() only gets called from
RuntimeState::ReleaseResources(), which is only called from
FragmentInstanceState::Close(), which can't be called until after Open() is run
on exec_tree_ and UpdateFilterFromLocal() is only called during Open() (via
PartitionedHashJoinNode::Open() ->
BlockingJoinNode::ProcessBuildInputAndOpenProbe(), which calls on a thread and
then blocks until completion for BlockingJoinNode::ProcessBuildInputAsync() ->
BlockingJoinNode::SendBuildInputToSink() -> PhjBuilder::FlushFinal())
It might be nice if you could confirm my
[Impala-ASF-CR] IMPALA-8907: TestResultSpooling.test slow query is flaky
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14170 ) Change subject: IMPALA-8907: TestResultSpooling.test_slow_query is flaky .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idcb4a6b38f85a9497f80e2674e1c1fa512be5940 Gerrit-Change-Number: 14170 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 04 Sep 2019 19:23:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7312: Non-blocking mode for Fetch() RPC
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/14157 )
Change subject: IMPALA-7312: Non-blocking mode for Fetch() RPC
..
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@159
PS4, Line 159: wait_timeout_timer_
Don't need _ suffix on local variable.
http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@172
PS4, Line 172: wait_timeout_timer_.Start();
Why not just start the timer and leave it running? I think the elapsed time is
likely more important to the client than the time spent blocked.
http://gerrit.cloudera.org:8080/#/c/14157/4/be/src/exec/buffered-plan-root-sink.cc@174
PS4, Line 174: if (!rows_available_.WaitFor(l, wait_duration)) {
nit: could be a one liner:
timed_out = !rows_available_.WaitFor(...);
--
To view, visit http://gerrit.cloudera.org:8080/14157
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I331acaba23a65dab43cca48e9dc0dc957b9c632d
Gerrit-Change-Number: 14157
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 04 Sep 2019 19:08:00 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14134 )
Change subject: IMPALA-8797: Support database and table blacklist
..
Patch Set 9:
(3 comments)
lgtm, can +2 it once the TableName#analyze() is refactored to include the
blacklist checks.
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java
File fe/src/main/java/org/apache/impala/analysis/TableName.java:
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java@86
PS8, Line 86: throw new AnalysisException("Invalid table/view name: " +
tbl_);
> I think they should always be called together. But BlacklistUtils.verifyTab
I think we can add the getFqTableName() logic into
BlackListUtils.verifyTableName().
My concern with separating these too is that if someone adds a new Statement in
the future that involves a tableName and they forget to to call
BlackListUtils.verify(), we might run into issues. Instead folding that logic
into TableName#analyze() makes life easier.
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1519
PS8, Line 1519: addSummary(resp, "Can't drop blacklisted database: " +
dbName);
> This is tested in tests/custom_cluster/test_blacklisted_dbs_and_tables.py:
ack, sorry missed it.
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/util/BlacklistUtils.java
File fe/src/main/java/org/apache/impala/util/BlacklistUtils.java:
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/util/BlacklistUtils.java@32
PS8, Line 32:
> Sure. What about CatalogBlacklistUtils?
sgtm
--
To view, visit http://gerrit.cloudera.org:8080/14134
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc
Gerrit-Change-Number: 14134
Gerrit-PatchSet: 9
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Wed, 04 Sep 2019 17:47:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 04 Sep 2019 17:02:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14134 ) Change subject: IMPALA-8797: Support database and table blacklist .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4461/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 04 Sep 2019 14:36:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14134 to look at the new patch set (#9). Change subject: IMPALA-8797: Support database and table blacklist .. IMPALA-8797: Support database and table blacklist Add catalogd and coordinator startup options for database and table blacklist. Blacklisted dbs/tables will be skipped in loading. Users won't see them when getting database/table list, e.g. in SHOW DATABASES/TABLES. Dropping/creating blacklisted databases/tables/views are not allowed too. Implementation: Catalogd and coordinators parses the --blacklisted_dbs and --blacklisted_tables options in startup. Blacklist checks are added in catalogd when loading the metadata and in coordinators when analysing DDL requests for create/drop. Catalogd will also check blacklist when executing DDL requests from coordinators in case their blacklists are inconsistent. Motivation: By default, it's used to blacklist "sys" and "information_schema" databases from Hive. Admin can use them to specify any databases/tables that are not suitable for Impala to query. Tests: - Add java unit tests for blacklist related functions - Add a custom cluster test: test_blacklisted_dbs_and_tables.py - Ran CORE tests Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateViewStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/TableName.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/main/java/org/apache/impala/util/CatalogBlacklistUtils.java A fe/src/test/java/org/apache/impala/util/CatalogBlacklistUtilsTest.java A tests/custom_cluster/test_blacklisted_dbs_and_tables.py 15 files changed, 540 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14134/9 -- To view, visit http://gerrit.cloudera.org:8080/14134 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc Gerrit-Change-Number: 14134 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8797: Support database and table blacklist
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/14134 )
Change subject: IMPALA-8797: Support database and table blacklist
..
Patch Set 8:
(5 comments)
> I think the issue with sys and information_schema not appearing is probably
> something to do with the version of hive pinned in the toolchain.
I've tried USE_CDP_HIVE=true to use Hive3, but there are still no "sys" and
"information_schema" databases. Maybe it requires some configuration change or
it's not shown for postgresql. Hope I can find the cause tomorrow.
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java
File fe/src/main/java/org/apache/impala/analysis/TableName.java:
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/analysis/TableName.java@86
PS8, Line 86: throw new AnalysisException("Invalid table/view name: " +
tbl_);
> Is there a reason not to fold the Blacklist.verifyTableName() into TableNam
I think they should always be called together. But
BlacklistUtils.verifyTableName() requires a fully-qualified TableName object. I
used to add it into TableName.analyze() in a previous patch version, then the
codes like this:
tableName_.analyze();
become something like:
analyzer.getFqTableName(tableName_).analyze();
Since "analyze" sounds like changing the state (to be analyzed) of the target.
Now we look like changing the state of a temp object... So I decided to
refactor the function name to "verify", meaning we are just verifying something
based on the temp object.
Finally, we decided to keep the name as "analyze" to be consistent with other
parts (in the review of patch set 7). So I moved out the logics of
BlacklistUtils.verifyTableName() since it requires more (valid db_) comparing
to TableName.analyze().
However, I'm ok if we should merge them together and writing codes like
analyzer.getFqTableName(tableName_).analyze();
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@270
PS8, Line 270: // Error string for inconsistent blacklisted dbs/tables
configs between catalogd and
> Yea, we don't have any checks that validate the configs across roles. That
Yes, I think for configs that read by both catalogd and coordinators, we should
always check the consistent. I used to just give the configs to catalogd and
found wired behaviors. So I decide to add these checks.
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1519
PS8, Line 1519: addSummary(resp, "Can't drop blacklisted database: " +
dbName);
> Add a test for this?
This is tested in tests/custom_cluster/test_blacklisted_dbs_and_tables.py:
self.__expect_error_in_result(
"drop database if exists functional_rc",
"Can't drop blacklisted database: functional_rc")
Note that it uses __expect_error_in_result, not __expect_error_in_query.
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1644
PS8, Line 1644:
> Similar check here?
Good point! I missed it since the bug of IMPALA-8918. Will add more checks.
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/util/BlacklistUtils.java
File fe/src/main/java/org/apache/impala/util/BlacklistUtils.java:
http://gerrit.cloudera.org:8080/#/c/14134/8/fe/src/main/java/org/apache/impala/util/BlacklistUtils.java@32
PS8, Line 32: BlacklistUtils
> nit: Call this CatalogUtils or something? BlackList seems out of place, doe
Sure. What about CatalogBlacklistUtils?
--
To view, visit http://gerrit.cloudera.org:8080/14134
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02dbb07f8e08793b57b2a88d09b30fd32cff26dc
Gerrit-Change-Number: 14134
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Wed, 04 Sep 2019 13:55:04 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4460/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 16 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 04 Sep 2019 09:43:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 16: (1 comment) http://gerrit.cloudera.org:8080/#/c/13722/15/be/src/runtime/datetime-iso-sql-format-parser.cc File be/src/runtime/datetime-iso-sql-format-parser.cc: http://gerrit.cloudera.org:8080/#/c/13722/15/be/src/runtime/datetime-iso-sql-format-parser.cc@144 PS15, Line 144: > Good catch this is actually a bug! In fact I think something is off with th Fixed in PS16 -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 16 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 04 Sep 2019 09:04:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Hello Attila Jeges, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13722
to look at the new patch set (#16).
Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
..
IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
This enhancement introduces FORMAT clause for CAST() operator that is
applicable for casts between string types and timestamp types. Instead
of accepting SimpleDateFormat patterns the FORMAT clause supports
datetime patterns following the ISO:SQL:2016 standard.
Note, the CAST() operator without the FORMAT clause still uses
Impala's implementation of SimpleDateFormat handling. Similarly, the
existing conversion functions such as to_timestamp(), from_timestamp()
etc. remain unchanged and use SimpleDateFormat. Contrary to how these
functions work the FORMAT clause must specify a string literal and
cannot be used with any other kind of a string expression.
Milestone 1 contains all the format tokens covered by the SQL
standard. Further milestones will add more functionality on top of
this list to cover functionality provided by other RDBMS systems.
List of tokens implemented by this change:
- , YYY, YY, Y: Year tokens
- , RR: Round year tokens
- MM: Month (1-12)
- DD: Day (1-31)
- DDD: Day of year (1-366)
- HH, HH12: Hour of day (1-12)
- HH24: Hour of day (0-23)
- MI: Minute (0-59)
- SS: Second (0-59)
- S: Second of day (0-86399)
- FF, FF1, ..., FF9: Fractional second
- AM, PM, A.M., P.M.: Meridiem indicators
- TZH: Timezone hour (-99-+99)
- TZM: Timezone minute (0-99)
- Separators: - . / , ' ; : space
- ISO8601 date indicators (T, Z)
Some notes about the matching algorithm:
- The parsing algorithm uses these tokens in a case insensitive
manner.
- The separators are interchangeable with each other. For example a
'-' separator in the format will match with a '.' character in the
input.
- The length of the separator sequences is handled flexibly meaning
that a single separator character in the format for instance would
match with a multi-separator sequence in the input.
- In a string type to timestamp conversion the timezone offset tokens
are parsed, expected to match with the input but they don't adjust
the result as the input is already expected to be in UTC format.
Usage example:
SELECT CAST('01-02-2019' AS TIMESTAMP FORMAT 'MM-DD-');
SELECT CAST('2019.10.10 13:30:40.123456 +01:30' AS TIMESTAMP
FORMAT '-MM-DD HH24:MI:SS.FF9 TZH:TZM');
SELECT CAST(timestamp_column as STRING
FORMAT " MM HH12 YY") from some_table;
Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23
---
M be/src/benchmarks/convert-timestamp-benchmark.cc
M be/src/benchmarks/parse-timestamp-benchmark.cc
M be/src/common/init.cc
M be/src/exec/text-converter.inline.h
M be/src/exprs/CMakeLists.txt
A be/src/exprs/cast-format-expr.cc
A be/src/exprs/cast-format-expr.h
M be/src/exprs/cast-functions-ir.cc
M be/src/exprs/date-functions-ir.cc
M be/src/exprs/expr-test.cc
M be/src/exprs/scalar-expr-evaluator.h
M be/src/exprs/scalar-expr.cc
M be/src/exprs/scalar-expr.h
M be/src/exprs/timestamp-functions-ir.cc
M be/src/exprs/timestamp-functions.cc
M be/src/exprs/timestamp-functions.h
M be/src/runtime/CMakeLists.txt
M be/src/runtime/date-parse-util.cc
M be/src/runtime/date-parse-util.h
M be/src/runtime/date-test.cc
M be/src/runtime/date-value.cc
M be/src/runtime/date-value.h
A be/src/runtime/datetime-iso-sql-format-parser.cc
A be/src/runtime/datetime-iso-sql-format-parser.h
A be/src/runtime/datetime-iso-sql-format-tokenizer.cc
A be/src/runtime/datetime-iso-sql-format-tokenizer.h
D be/src/runtime/datetime-parse-util.h
A be/src/runtime/datetime-parser-common.cc
A be/src/runtime/datetime-parser-common.h
R be/src/runtime/datetime-simple-date-format-parser.cc
A be/src/runtime/datetime-simple-date-format-parser.h
M be/src/runtime/runtime-state.cc
M be/src/runtime/timestamp-parse-util.cc
M be/src/runtime/timestamp-parse-util.h
M be/src/runtime/timestamp-test.cc
M be/src/runtime/timestamp-value.cc
M be/src/runtime/timestamp-value.h
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/testutil/random-vector-generators.h
M be/src/util/dict-test.cc
M be/src/util/min-max-filter-test.cc
M be/src/util/string-parser.h
M common/thrift/Exprs.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CastExpr.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
A
testdata/workloads/functional-query/queries/QueryTest/cast_format_from_table.test
M testdata/workloads/functional-query/queries/QueryTest/date.test
A tests/query_test/test_cast_with_format.py
54 files changed, 3,461 insertions(+), 865 deletions(-)
git
[Impala-ASF-CR] IMPALA-8913: Add query option to disable hbase row estimation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14169 ) Change subject: IMPALA-8913: Add query option to disable hbase row estimation .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I768ea1f560c5faab74d8772bc8aa9ddefdcf2e10 Gerrit-Change-Number: 14169 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 04 Sep 2019 08:37:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8913: Add query option to disable hbase row estimation
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14169 ) Change subject: IMPALA-8913: Add query option to disable hbase row estimation .. IMPALA-8913: Add query option to disable hbase row estimation IMPALA-8058 added a flag, disable_hbase_row_est, in the QueryCtx for tests to bypass the codes of estimating HBase row stats from HBase RPCs and fall back to HMS row count stats. This patch exposes it as a query option. Tests: - Replace the usage of disable_hbase_row_est in QueryCtx by the new query option in tests. Verified relative tests pass as before. Change-Id: I768ea1f560c5faab74d8772bc8aa9ddefdcf2e10 Reviewed-on: http://gerrit.cloudera.org:8080/14169 Reviewed-by: Quanlong Huang Tested-by: Impala Public Jenkins --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 7 files changed, 33 insertions(+), 16 deletions(-) Approvals: Quanlong Huang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14169 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I768ea1f560c5faab74d8772bc8aa9ddefdcf2e10 Gerrit-Change-Number: 14169 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 15: (1 comment) http://gerrit.cloudera.org:8080/#/c/13722/15/be/src/runtime/datetime-iso-sql-format-parser.cc File be/src/runtime/datetime-iso-sql-format-parser.cc: http://gerrit.cloudera.org:8080/#/c/13722/15/be/src/runtime/datetime-iso-sql-format-parser.cc@144 PS15, Line 144: 0, 99 > Ok, I checked out your change and run some tests. This works: Good catch this is actually a bug! In fact I think something is off with this TH handling as I observed another input to work incorrectly. Let me give this a second thought and come back with a fix. -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 15 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 04 Sep 2019 08:21:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8755: Frontend support for Z-ordering
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13955 ) Change subject: IMPALA-8755: Frontend support for Z-ordering .. Patch Set 11: Have you forgot to upload the latest patch set? :) -- To view, visit http://gerrit.cloudera.org:8080/13955 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie122002ca8f52ca2c1e1ec8ff1d476ae1f4f875d Gerrit-Change-Number: 13955 Gerrit-PatchSet: 11 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 04 Sep 2019 06:44:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8915: reapply IMPALA-8901 fix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14176 ) Change subject: IMPALA-8915: reapply IMPALA-8901 fix .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4459/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14176 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I42ea1ec4c386f2eb45f370cd35a104b49786a48c Gerrit-Change-Number: 14176 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 04 Sep 2019 06:04:41 + Gerrit-HasComments: No
