[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Jiawei Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14600 Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. IMPALA-9109: Add top-k metadata loading ranking on catalogd UI Add functions in CatalogUsageMonitor to monitor and report the catalog usage of the tables have the longest metadata loading time. Add the sorted table in Catalog server web-ui. The loading time is cacualted by the median from load_duration metrics. Testing: Launch Impala and activate some tables to see the table loading time showed successfully on the catalog debug UI page. Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M www/catalog.tmpl 8 files changed, 132 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/14600/2 -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 2 Gerrit-Owner: Jiawei Wang
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managed table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managed table without HMS config .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5158/ -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 8 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 31 Oct 2019 06:42:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9108: exclude leveldbjni mvn dependency
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14593 ) Change subject: IMPALA-9108: exclude leveldbjni mvn dependency .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4916/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14593 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I90b63bc03511545530e1506bc602623591c56e98 Gerrit-Change-Number: 14593 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 31 Oct 2019 06:38:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9108: exclude leveldbjni mvn dependency
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14593 Change subject: IMPALA-9108: exclude leveldbjni mvn dependency .. IMPALA-9108: exclude leveldbjni mvn dependency We don't need this at all - it's pulled in via some transitive dependencies, e.g. htrace and hive-serde. Add an exclusion and add it as a banned dependency. Change-Id: I90b63bc03511545530e1506bc602623591c56e98 --- M fe/pom.xml 1 file changed, 22 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/14593/1 -- To view, visit http://gerrit.cloudera.org:8080/14593 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I90b63bc03511545530e1506bc602623591c56e98 Gerrit-Change-Number: 14593 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9107: Add toolchain maven cache to speed up maven builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14562 ) Change subject: IMPALA-9107: Add toolchain maven cache to speed up maven builds .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4915/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I043912f5fbc7cf24ee80b2855354656aa587ca9f Gerrit-Change-Number: 14562 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 31 Oct 2019 05:34:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8815: fix ranger startup after set-classpath.sh
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14152 ) Change subject: IMPALA-8815: fix ranger startup after set-classpath.sh .. IMPALA-8815: fix ranger startup after set-classpath.sh Having non-existent or incompatible jars on the classpath can cause Ranger startup to fail. Update run-ranger-server.sh to clean the classpath so that it works after sourcing set-classpath.sh. Also remove a couple of legacy jars from 2013. Those jars no longer exist in Hive. Testing: In my development environment. $ . bin/set-classpath.sh $ ./testdata/bin/run-ranger-server.sh Change-Id: Ie7036f9a07e5c9b8d46bb7f459d0b9d1e7e9d0a7 Reviewed-on: http://gerrit.cloudera.org:8080/14152 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/set-classpath.sh M testdata/bin/run-ranger-server.sh 2 files changed, 7 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14152 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie7036f9a07e5c9b8d46bb7f459d0b9d1e7e9d0a7 Gerrit-Change-Number: 14152 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-9107: Add toolchain maven cache to speed up maven builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14562 ) Change subject: IMPALA-9107: Add toolchain maven cache to speed up maven builds .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/14562/4/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/14562/4/bin/bootstrap_toolchain.py@388 PS4, Line 388: class ToolchainMavenCache(EnvVersionedPackage): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/14562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I043912f5fbc7cf24ee80b2855354656aa587ca9f Gerrit-Change-Number: 14562 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 31 Oct 2019 04:51:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8815: fix ranger startup after set-classpath.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14152 ) Change subject: IMPALA-8815: fix ranger startup after set-classpath.sh .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14152 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7036f9a07e5c9b8d46bb7f459d0b9d1e7e9d0a7 Gerrit-Change-Number: 14152 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 31 Oct 2019 05:01:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9107: Add toolchain maven cache to speed up maven builds
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14562 to look at the new patch set (#4). Change subject: IMPALA-9107: Add toolchain maven cache to speed up maven builds .. IMPALA-9107: Add toolchain maven cache to speed up maven builds The maven build downloads a large number of artifacts from various maven repositories. When starting with an empty .m2 directory (like most upstream Jenkins jobs), downloading all the artifacts can take up to 30 minutes. This has been slowing down our precommit builds by 15-20 minutes. This adds a mode to download a tarball from the toolchain, unpack it, and use it as a maven repository. The tarball is constructed by doing a fresh Impala build with an empty .m2 directory, removing locations that should come from impala.cdp.repo or impala.cdh.repo, then packing that into a tarball. The tarball is currently around 240MB and unpacks to about 309MB, so the space requirement is fairly modest. The jars in the toolchain maven cache are copied to the .m2 directory, so the toolchain maven cache can be removed/replaced without impacting a running cluster. Testing: - Local testing shows a substantially faster build with an empty .m2 directory. - Ran gerrit-verify-dryrun-external and the jobs took about 15-20 minutes less. Change-Id: I043912f5fbc7cf24ee80b2855354656aa587ca9f --- M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/mvn-quiet.sh M impala-parent/pom.xml 4 files changed, 110 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/14562/4 -- To view, visit http://gerrit.cloudera.org:8080/14562 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I043912f5fbc7cf24ee80b2855354656aa587ca9f Gerrit-Change-Number: 14562 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8879: Fix wrong active tabs after bootstrap upgrade
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14582 ) Change subject: IMPALA-8879: Fix wrong active tabs after bootstrap upgrade .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c746bc4f49b30df54bd3c73ac6f1cf838260c65 Gerrit-Change-Number: 14582 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 31 Oct 2019 03:33:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9105: Catalog debug page top-n table has a URL generation issue
Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14589 to look at the new patch set (#2). Change subject: IMPALA-9105: Catalog debug page top-n table has a URL generation issue .. IMPALA-9105: Catalog debug page top-n table has a URL generation issue What's the problem: Previously, catalogd debug page does not have 'fqtn' property. Because CatalogServer::GetCatalogUsage() does not pass fqtn to the template. Thus, the generated page table detail page cannot be opened. Also, Someone might accedentially deleted which makes the catalogd UI top-K page unorganized. Fix: Correct the URL generation part. Test: Open the page and all the tables object can be correctly opened. Change-Id: If7416c79baf2e78d6790995e97d9802ec7a8cc37 --- M www/catalog.tmpl 1 file changed, 4 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/14589/2 -- To view, visit http://gerrit.cloudera.org:8080/14589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If7416c79baf2e78d6790995e97d9802ec7a8cc37 Gerrit-Change-Number: 14589 Gerrit-PatchSet: 2 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9105: Catalog debug page top-n table has a URL generation issue
Jiawei Wang has posted comments on this change. (
http://gerrit.cloudera.org:8080/14589 )
Change subject: IMPALA-9105: Catalog debug page top-n table has a URL
generation issue
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14589/1/www/catalog.tmpl
File www/catalog.tmpl:
http://gerrit.cloudera.org:8080/#/c/14589/1/www/catalog.tmpl@42
PS1, Line 42:
> Oops!
Yeah, I think someone accidentally deleted this.
http://gerrit.cloudera.org:8080/#/c/14589/1/www/catalog.tmpl@47
PS1, Line 47:{{name}}
> Is {{fqtn}} empty here causing the problem? Looks like we still use {{fqtn}
Yes, fqtn empty caused the issue. The problem is that these "Top-n" tables are
passed in a different function in CatalogServer::GetCatalogUsage. And we did
not pass fqtn there.
--
To view, visit http://gerrit.cloudera.org:8080/14589
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If7416c79baf2e78d6790995e97d9802ec7a8cc37
Gerrit-Change-Number: 14589
Gerrit-PatchSet: 1
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 31 Oct 2019 03:34:17 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8879: Fix wrong active tabs after bootstrap upgrade
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14582 ) Change subject: IMPALA-8879: Fix wrong active tabs after bootstrap upgrade .. IMPALA-8879: Fix wrong active tabs after bootstrap upgrade After upgrading Bootstrap to 4.3.1 in the previous patch, the query page no longer highlights which tab is being used. This is due to the removal of ".nav-tabs > li.active > a" in bootstrap.css. So li elements in "active" class no longer has additional styles. Instead, the styles of "active" class is moved into to "nav-link" class in Bootstrap 4. We should add the "active" class in the "nav-link" elements instead. Tests: - Manually click throught all tabs to check the active state Change-Id: I5c746bc4f49b30df54bd3c73ac6f1cf838260c65 Reviewed-on: http://gerrit.cloudera.org:8080/14582 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M www/jvm-threadz.tmpl M www/query_backends.tmpl M www/query_detail_tabs.tmpl M www/query_finstances.tmpl M www/query_memory.tmpl M www/query_plan.tmpl M www/query_plan_text.tmpl M www/query_profile.tmpl M www/query_stmt.tmpl M www/query_summary.tmpl M www/threadz.tmpl M www/threadz_tabs.tmpl 12 files changed, 28 insertions(+), 24 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5c746bc4f49b30df54bd3c73ac6f1cf838260c65 Gerrit-Change-Number: 14582 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managed table without HMS config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managed table without HMS config .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5158/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 8 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 31 Oct 2019 01:49:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8974: Fixed a bug when create kudu managed table without HMS config
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14398 ) Change subject: IMPALA-8974: Fixed a bug when create kudu managed table without HMS config .. Patch Set 8: Code-Review+2 Sorry for the delay, thanks for addressing my comments! -- To view, visit http://gerrit.cloudera.org:8080/14398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iacc53801a660c033869cb4747910c98a80e08297 Gerrit-Change-Number: 14398 Gerrit-PatchSet: 8 Gerrit-Owner: wangsheng Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 31 Oct 2019 01:48:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9105: Catalog debug page top-n table has a URL generation issue
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/14589 )
Change subject: IMPALA-9105: Catalog debug page top-n table has a URL
generation issue
..
Patch Set 1:
(2 comments)
Thanks for fixing this by the way. I tried the patch locally and it works well.
This patch looks good to me. It'd be better if we can explain the root cause in
commit message. I'm ok to give a +2 if the cause is not easy to find.
http://gerrit.cloudera.org:8080/#/c/14589/1/www/catalog.tmpl
File www/catalog.tmpl:
http://gerrit.cloudera.org:8080/#/c/14589/1/www/catalog.tmpl@42
PS1, Line 42:
Oops!
http://gerrit.cloudera.org:8080/#/c/14589/1/www/catalog.tmpl@47
PS1, Line 47:{{name}}
Is {{fqtn}} empty here causing the problem? Looks like we still use {{fqtn}} at
line 194 and line 198. It'd be better if we can explain why it doesn't work in
the commit message. Maybe the answer is in ImpalaHttpHandler::CatalogHandler().
--
To view, visit http://gerrit.cloudera.org:8080/14589
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If7416c79baf2e78d6790995e97d9802ec7a8cc37
Gerrit-Change-Number: 14589
Gerrit-PatchSet: 1
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Thu, 31 Oct 2019 01:42:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8815: fix ranger startup after set-classpath.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14152 ) Change subject: IMPALA-8815: fix ranger startup after set-classpath.sh .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5156/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14152 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7036f9a07e5c9b8d46bb7f459d0b9d1e7e9d0a7 Gerrit-Change-Number: 14152 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 31 Oct 2019 00:32:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8815: fix ranger startup after set-classpath.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14152 ) Change subject: IMPALA-8815: fix ranger startup after set-classpath.sh .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14152 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7036f9a07e5c9b8d46bb7f459d0b9d1e7e9d0a7 Gerrit-Change-Number: 14152 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 31 Oct 2019 00:32:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8815: fix ranger startup after set-classpath.sh
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/14152 ) Change subject: IMPALA-8815: fix ranger startup after set-classpath.sh .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14152 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie7036f9a07e5c9b8d46bb7f459d0b9d1e7e9d0a7 Gerrit-Change-Number: 14152 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 31 Oct 2019 00:31:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8065 Add OS version and Kernel version in OSInfo
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/14531 ) Change subject: IMPALA-8065 Add OS version and Kernel version in OSInfo .. Patch Set 3: (8 comments) Code looks good, Comments and commit message need more work, I think http://gerrit.cloudera.org:8080/#/c/14531/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14531/3//COMMIT_MSG@8 PS3, Line 8: I think it is important to write a commit message that can be understood by someone who has not read the code you are changing. Sometimes it helps to explain what the code is. OsInfo is used to get human readable information about the OS on which Impala is running. Before this change OsInfo::DebugString() would print two lines: - OS version: the long name of the Linux kernel from /proc/version - Clock: the type of clock used After this change OsInfo::DebugString() will print three lines: - OS version: the short name of the Linux distribution. If Docker is being used this is the name of the Container OS - Kernel version: the long name of the Linux kernel from /proc/version If Docker is being used this is the description of the Host OS. - Clock: the type of clock used http://gerrit.cloudera.org:8080/#/c/14531/3//COMMIT_MSG@15 PS3, Line 15: OS version: "Ubuntu 16.04.6 LTS" Why does OS Version have quotes but Kernel version does not? http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/CMakeLists.txt File be/src/util/CMakeLists.txt: http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/CMakeLists.txt@124 PS3, Line 124: os-info-test.cc These should be in alphabetical order http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/CMakeLists.txt@187 PS3, Line 187: ADD_UNIFIED_BE_LSAN_TEST(os-info-test "OsInfo.*") These should be in alphabetical order http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/os-info-test.cc File be/src/util/os-info-test.cc: http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/os-info-test.cc@25 PS3, Line 25: ASSERT_NE(osinfo.os_version(), "Unknown"); Add a simple comment explaining the test http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/os-info.h File be/src/util/os-info.h: http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/os-info.h@35 PS3, Line 35: /// Name of the OS release. Simple name of the OS. If Docker is used this is the name of the Container OS http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/os-info.h@42 PS3, Line 42: /// and the version of a GCC compiler used to build it. The version of Linux kernel and the version of the compiler used to build it. If Docker is used this is the Host OS. http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/os-info.cc File be/src/util/os-info.cc: http://gerrit.cloudera.org:8080/#/c/14531/3/be/src/util/os-info.cc@62 PS3, Line 62: os_path = "/etc/centos-release"; Add a comment saying "Only old distributions like Centos 6" -- To view, visit http://gerrit.cloudera.org:8080/14531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I848c9e53ee4e0bf8ae0874bb6da28e8efa7f7c8a Gerrit-Change-Number: 14531 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Thu, 31 Oct 2019 00:06:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8065 Add OS version and Kernel version in OSInfo
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14531 ) Change subject: IMPALA-8065 Add OS version and Kernel version in OSInfo .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4914/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I848c9e53ee4e0bf8ae0874bb6da28e8efa7f7c8a Gerrit-Change-Number: 14531 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Wed, 30 Oct 2019 23:54:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4913/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 30 Oct 2019 23:42:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4912/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 30 Oct 2019 23:36:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8065 Add OS version and Kernel version in OSInfo
Xiaomeng Zhang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/14531 ) Change subject: IMPALA-8065 Add OS version and Kernel version in OSInfo .. IMPALA-8065 Add OS version and Kernel version in OSInfo Original we get /proc/version displayed as OS version, while it's actually kernel version. We should correct it as Kernel version, and display OS version from /etc/os-release (for centos6 it's /etc/centos-release). Tested locally, the displayed OS Info in Ubuntu16 dev box is: OS version: "Ubuntu 16.04.6 LTS" Kernel version: Linux version 4.15.0-65-generic (buildd@lcy01-amd64-017) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)) Clock: clocksource: 'tsc', clockid_t: CLOCK_MONOTONIC Also tested with diff OS in docker: centos, redhat, ubuntu, oracle, debian. Each OS picked one version to test. Added new backend test os-info-test.cc. Change-Id: I848c9e53ee4e0bf8ae0874bb6da28e8efa7f7c8a --- M be/src/util/CMakeLists.txt A be/src/util/os-info-test.cc M be/src/util/os-info.cc M be/src/util/os-info.h 4 files changed, 71 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/14531/3 -- To view, visit http://gerrit.cloudera.org:8080/14531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I848c9e53ee4e0bf8ae0874bb6da28e8efa7f7c8a Gerrit-Change-Number: 14531 Gerrit-PatchSet: 3 Gerrit-Owner: Xiaomeng Zhang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Xiaomeng Zhang
[Impala-ASF-CR] IMPALA-8879: Fix wrong active tabs after bootstrap upgrade
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14582 ) Change subject: IMPALA-8879: Fix wrong active tabs after bootstrap upgrade .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5154/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c746bc4f49b30df54bd3c73ac6f1cf838260c65 Gerrit-Change-Number: 14582 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 23:07:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8768: [DOCS] Added conditions for queries recorded in audit log
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14575 ) Change subject: IMPALA-8768: [DOCS] Added conditions for queries recorded in audit log .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14575/1/docs/topics/impala_auditing.xml File docs/topics/impala_auditing.xml: http://gerrit.cloudera.org:8080/#/c/14575/1/docs/topics/impala_auditing.xml@269 PS1, Line 269: Queries with at least one fetch issued by the client. IMPALA-8572 (coming in 3.4) actually changed this slightly, so that queries are logged as soon as results are available to be fetched by clients, so before the first fetch occurs. Bharath said in the commit message: The conditions under which the events are logged are slightly modified by this patch. Without the patch, events are logged for unsuccessful queries if atleast a single fetch is perfomed. This patch relaxes this guarantee to log events for any query that reaches the FINISHED state (rows are available to fetch by the client) and does not wait for a fetch to be performed. This simplifies the coordinator state machine by avoiding unnecessary synchronization. -- To view, visit http://gerrit.cloudera.org:8080/14575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I809e716e66558db02e6401bd218b3dd2de49864c Gerrit-Change-Number: 14575 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Wed, 30 Oct 2019 23:07:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8879: Fix wrong active tabs after bootstrap upgrade
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14582 ) Change subject: IMPALA-8879: Fix wrong active tabs after bootstrap upgrade .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c746bc4f49b30df54bd3c73ac6f1cf838260c65 Gerrit-Change-Number: 14582 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 23:07:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Anurag Mantripragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14592 Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. This is the first of several changes to use informational, unenforced primary key(pk) and foreign key(fk) specifications in Impala. The parent JIRA for this effort is IMPALA-3531. This change adds support for adding pk/fk information during create table DDLs. There is only limited SQL syntax support as of now and will add various other SQL styles including ANSI syntax support in later changes. Currently the only supported way of adding fk/pk information is after the column definitions. Examples are: CREATE TABLE pk(col1 INT, col2 STRING, PRIMARY KEY(col1, col2)); CREATE TABLE fk(id INT, col1 INT, col2 STRING, PRIMARY KEY(id), FOREIGN KEY(col1, col2) REFERENCES pk(col1, col2)); In the current implementation, manual specification of constraint names is not supported. Internally we use UUIDs for contraint name generation. Additionally, three constraint states are supported to comply with Hive's implementation which were taken from Oracle. DISABLE (default true) NOVALIDATE (default true) RELY (default true) More info here: https://docs.oracle.com/database/121/DWHSG/schemas.htm#DWHSG9053 These constraints can be optionally specified after each PK/FK specification like: CREATE TABLE pk(id INT, PRIMARY KEY(id) DISABLE, NOVALIDATE, RELY); Basic toSql support is added but observability changes like showing PK/FK information in DESCRIBE output will be done separately. Tests: Added tests to: - AnalyzeDDLTest#TestCreateTable - ParserTest#TestCreateTable - ToSqlTest#TestCreateTable Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java 16 files changed, 719 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/14592/1 -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 1 Gerrit-Owner: Anurag Mantripragada
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Anurag Mantripragada has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. This is the first of several changes to use informational, unenforced primary key(pk) and foreign key(fk) specifications in Impala. The parent JIRA for this effort is IMPALA-3531. This change adds support for adding pk/fk information during create table DDLs. There is only limited SQL syntax support as of now and will add various other SQL styles including ANSI syntax support in later changes. Currently the only supported way of adding fk/pk information is after the column definitions. Examples are: CREATE TABLE pk(col1 INT, col2 STRING, PRIMARY KEY(col1, col2)); CREATE TABLE fk(id INT, col1 INT, col2 STRING, PRIMARY KEY(id), FOREIGN KEY(col1, col2) REFERENCES pk(col1, col2)); In the current implementation, manual specification of constraint names is not supported. Internally we use UUIDs for contraint name generation. Additionally, three constraint states are supported to comply with Hive's implementation which were taken from Oracle. DISABLE (default true) NOVALIDATE (default true) RELY (default true) More info here: https://docs.oracle.com/database/121/DWHSG/schemas.htm#DWHSG9053 These constraints can be optionally specified after each PK/FK specification like: CREATE TABLE pk(id INT, PRIMARY KEY(id) DISABLE, NOVALIDATE, RELY); Basic toSql support is added but observability changes like showing PK/FK information in DESCRIBE output will be done separately. Tests: Added tests to: - AnalyzeDDLTest#TestCreateTable - ParserTest#TestCreateTable - ToSqlTest#TestCreateTable Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java 16 files changed, 719 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/14592/2 -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 2 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14592 )
Change subject: IMPALA-2112: Support primary key/foreign key constraints as
part of create table in Impala.
..
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/14592/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/14592/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@959
PS1, Line 959: primaryKeys_.addAll(client.getPrimaryKeys(new
PrimaryKeysRequest(msTbl.getDbName(),
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/14592/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/14592/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2391
PS1, Line 2391: AnalysisError("create table fk(id int, foreign key(id)
references non_pk_table(id))", ctx,
line too long (94 > 90)
http://gerrit.cloudera.org:8080/#/c/14592/1/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java
File fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java:
http://gerrit.cloudera.org:8080/#/c/14592/1/fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java@384
PS1, Line 384: + "(year))", "CREATE TABLE test_pk_fk.fk ( id INT,
year STRING, FOREIGN KEY(id) "
line too long (93 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/14592
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7
Gerrit-Change-Number: 14592
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Wed, 30 Oct 2019 22:52:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5152/ -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 22:33:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9105: Catalog debug page top-n table has a URL generation issue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14589 ) Change subject: IMPALA-9105: Catalog debug page top-n table has a URL generation issue .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4911/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If7416c79baf2e78d6790995e97d9802ec7a8cc37 Gerrit-Change-Number: 14589 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 30 Oct 2019 19:24:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14538 ) Change subject: IMPALA-4400: aggregate runtime filters locally .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4909/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc Gerrit-Change-Number: 14538 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 19:01:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14538 ) Change subject: IMPALA-4400: aggregate runtime filters locally .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4910/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc Gerrit-Change-Number: 14538 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 19:03:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14538 ) Change subject: IMPALA-4400: aggregate runtime filters locally .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4908/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc Gerrit-Change-Number: 14538 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 18:53:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9105: Catalog debug page top-n table has a URL generation issue
Jiawei Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14589 Change subject: IMPALA-9105: Catalog debug page top-n table has a URL generation issue .. IMPALA-9105: Catalog debug page top-n table has a URL generation issue What's the problem: Previously, catalogd debug page does not have 'fqtn' property. Thus, the generated page table detail page cannot be opened. Fix: Correct the URL generation part. Test: Open the page and all the tables object can be correctly opened. Change-Id: If7416c79baf2e78d6790995e97d9802ec7a8cc37 --- M www/catalog.tmpl 1 file changed, 4 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/14589/1 -- To view, visit http://gerrit.cloudera.org:8080/14589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If7416c79baf2e78d6790995e97d9802ec7a8cc37 Gerrit-Change-Number: 14589 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14538 ) Change subject: IMPALA-4400: aggregate runtime filters locally .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4907/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc Gerrit-Change-Number: 14538 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 18:37:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14538 ) Change subject: IMPALA-4400: aggregate runtime filters locally .. Patch Set 9: (3 comments) http://gerrit.cloudera.org:8080/#/c/14538/9/be/src/runtime/runtime-filter-bank.h File be/src/runtime/runtime-filter-bank.h: http://gerrit.cloudera.org:8080/#/c/14538/9/be/src/runtime/runtime-filter-bank.h@193 PS9, Line 193: /// Contains references to all the min-max filters generated. Used in Close() to safely > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/14538/9/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/14538/9/fe/src/main/java/org/apache/impala/planner/Planner.java@444 PS9, Line 444: totalRuntimeFilterMemBytes += fragment.getProducedRuntimeFiltersMemReservationBytes(); > line too long (92 > 90) Done http://gerrit.cloudera.org:8080/#/c/14538/9/tests/custom_cluster/test_mt_dop.py File tests/custom_cluster/test_mt_dop.py: http://gerrit.cloudera.org:8080/#/c/14538/9/tests/custom_cluster/test_mt_dop.py@90 PS9, Line 90: > flake8: W391 blank line at end of file Done -- To view, visit http://gerrit.cloudera.org:8080/14538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc Gerrit-Change-Number: 14538 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 18:24:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5152/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 18:13:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Hello Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14538 to look at the new patch set (#12). Change subject: IMPALA-4400: aggregate runtime filters locally .. IMPALA-4400: aggregate runtime filters locally Move RuntimeFilterBank to QueryState(). Implement fine-grained locking for each filter to mitigate any increased lock contention from the change. Make RuntimeFilterBank handle multiple producers of the same filter, e.g. multiple instances of a partitioned join. It computes the expected number of filters upfront then sends the filter to the coordinator once all the local instances have been merged together. The merging can done in parallel locally to improve latency of filter propagation. Add Or() methods to MinMaxFilter and BloomFilter, since we now need to merge those, not just the thrift versions. Update coordinator filter routing to expect only one instance of a filter from each producer backend and to only send one instance to each consumer backend (instead of sending one per fragment). Update memory reservations and estimates to be lower to account for sharing of filters between fragment instances. mt_dop plans are modified to show these shared and non-shared resources separately. Enable waiting for runtime filters for kudu scanner with mt_dop. Made min/max filters const-correct. TODO: * Rebase onto KRPC runtime filter change Testing * Added unit tests for Or() methods. * Added some additional e2e test coverage for mt_dop queries * Updated planner tests with new estimates and reservation. * Ran a single node 3-impalad stress test with TPC-H kudu and TPC-DS parquet. * TODO: run tests under ASAN. Perf * Did a single-node perf run on TPC-H with default settings. No perf change. * Single-node perf run with mt_dop=8 showed significant speedups: +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(30) | parquet / none / none | 10.07 | -5.96% | 5.07 | -10.80%| +--+---+-++++ +--+--+---++-++---++---++-+-+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval| +--+--+---++-++---++---++-+-+ | TPCH(30) | TPCH-Q7 | parquet / none / none | 37.49 | 36.33 | +3.18% | 6.34% | 4.85%| 20| +1.90% | 3.96| 1.75| | TPCH(30) | TPCH-Q15 | parquet / none / none | 3.77 | 3.75| +0.61% | 1.20% | 1.03%| 20| +0.74% | 1.50| 1.72| | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.32 | 2.32| +0.05% | 1.62% | 2.14%| 20| -0.01% | -0.04 | 0.09| | TPCH(30) | TPCH-Q19 | parquet / none / none | 5.17 | 5.18| -0.20% | 1.56% | 1.63%| 20| -0.09% | -0.89 | -0.39 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.27 | 4.28| -0.29% | 1.09% | 1.80%| 20| -0.05% | -0.74 | -0.61 | | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.24 | 1.25| -0.35% | 3.47% | 2.95%| 20| -0.19% | -0.92 | -0.35 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 9.73 | 9.87| -1.38% | 1.22% | 1.05%| 20| -1.34% | -3.26 | -3.87 | | TPCH(30) | TPCH-Q16 | parquet / none / none | 2.49 | 2.54| -1.97% | 2.91% | 2.41%| 20| -2.07% | -2.09 | -2.36 | | TPCH(30) | TPCH-Q2 | parquet / none / none | 1.97 | 2.01| -1.91% | 2.14% | 2.57%| 20| -2.21% | -2.76 | -2.58 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 80.59 | 82.48 | -2.29% | 6.61% | 3.34%| 20| -3.67% | -3.17 | -1.41 | | TPCH(30) | TPCH-Q10 | parquet / none / none | 5.12 | 5.43| I -5.70% | 0.82% | 1.62%| 20| I -5.72% | -5.27 | -14.22 | | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.50 | 26.20 | I -6.49% | 0.47% | 0.43%| 20| I -7.00% | -5.27 | -47.60 | | TPCH(30) | TPCH-Q18 | parquet / none / none | 8.77 | 9.48| I -7.55% | 0.83% | 0.79%| 20| I -8.06% | -5.27 | -30.59 | | TPCH(30) | TPCH-Q3 | parquet / none / none | 6.05 | 6.61
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Hello Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14538 to look at the new patch set (#11). Change subject: IMPALA-4400: aggregate runtime filters locally .. IMPALA-4400: aggregate runtime filters locally Move RuntimeFilterBank to QueryState(). Implement fine-grained locking for each filter to mitigate any increased lock contention from the change. Make RuntimeFilterBank handle multiple producers of the same filter, e.g. multiple instances of a partitioned join. It computes the expected number of filters upfront then sends the filter to the coordinator once all the local instances have been merged together. The merging can done in parallel locally to improve latency of filter propagation. Add Or() methods to MinMaxFilter and BloomFilter, since we now need to merge those, not just the thrift versions. Update coordinator filter routing to expect only one instance of a filter from each producer backend and to only send one instance to each consumer backend (instead of sending one per fragment). Update memory reservations and estimates to be lower to account for sharing of filters between fragment instances. mt_dop plans are modified to show these shared and non-shared resources separately. Enable waiting for runtime filters for kudu scanner with mt_dop. Made min/max filters const-correct. TODO: * Rebase onto KRPC runtime filter change Testing * Added unit tests for Or() methods. * Added some additional e2e test coverage for mt_dop queries * Updated planner tests with new estimates and reservation. * Ran a single node 3-impalad stress test with TPC-H kudu and TPC-DS parquet. * TODO: run tests under ASAN. Perf * Did a single-node perf run on TPC-H with default settings. No perf change. * Single-node perf run with mt_dop=8 showed significant speedups: +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(30) | parquet / none / none | 10.07 | -5.96% | 5.07 | -10.80%| +--+---+-++++ +--+--+---++-++---++---++-+-+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval| +--+--+---++-++---++---++-+-+ | TPCH(30) | TPCH-Q7 | parquet / none / none | 37.49 | 36.33 | +3.18% | 6.34% | 4.85%| 20| +1.90% | 3.96| 1.75| | TPCH(30) | TPCH-Q15 | parquet / none / none | 3.77 | 3.75| +0.61% | 1.20% | 1.03%| 20| +0.74% | 1.50| 1.72| | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.32 | 2.32| +0.05% | 1.62% | 2.14%| 20| -0.01% | -0.04 | 0.09| | TPCH(30) | TPCH-Q19 | parquet / none / none | 5.17 | 5.18| -0.20% | 1.56% | 1.63%| 20| -0.09% | -0.89 | -0.39 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.27 | 4.28| -0.29% | 1.09% | 1.80%| 20| -0.05% | -0.74 | -0.61 | | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.24 | 1.25| -0.35% | 3.47% | 2.95%| 20| -0.19% | -0.92 | -0.35 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 9.73 | 9.87| -1.38% | 1.22% | 1.05%| 20| -1.34% | -3.26 | -3.87 | | TPCH(30) | TPCH-Q16 | parquet / none / none | 2.49 | 2.54| -1.97% | 2.91% | 2.41%| 20| -2.07% | -2.09 | -2.36 | | TPCH(30) | TPCH-Q2 | parquet / none / none | 1.97 | 2.01| -1.91% | 2.14% | 2.57%| 20| -2.21% | -2.76 | -2.58 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 80.59 | 82.48 | -2.29% | 6.61% | 3.34%| 20| -3.67% | -3.17 | -1.41 | | TPCH(30) | TPCH-Q10 | parquet / none / none | 5.12 | 5.43| I -5.70% | 0.82% | 1.62%| 20| I -5.72% | -5.27 | -14.22 | | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.50 | 26.20 | I -6.49% | 0.47% | 0.43%| 20| I -7.00% | -5.27 | -47.60 | | TPCH(30) | TPCH-Q18 | parquet / none / none | 8.77 | 9.48| I -7.55% | 0.83% | 0.79%| 20| I -8.06% | -5.27 | -30.59 | | TPCH(30) | TPCH-Q3 | parquet / none / none | 6.05 | 6.61
[Impala-ASF-CR] IMPALA-8879: Fix wrong active tabs after bootstrap upgrade
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14582 ) Change subject: IMPALA-8879: Fix wrong active tabs after bootstrap upgrade .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5c746bc4f49b30df54bd3c73ac6f1cf838260c65 Gerrit-Change-Number: 14582 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 18:06:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 18:13:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 ) Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4906/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13794 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d Gerrit-Change-Number: 13794 Gerrit-PatchSet: 7 Gerrit-Owner: Greg Rahn Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 30 Oct 2019 18:00:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Hello Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14538 to look at the new patch set (#10). Change subject: IMPALA-4400: aggregate runtime filters locally .. IMPALA-4400: aggregate runtime filters locally Move RuntimeFilterBank to QueryState(). Implement fine-grained locking for each filter to mitigate any increased lock contention from the change. Make RuntimeFilterBank handle multiple producers of the same filter, e.g. multiple instances of a partitioned join. It computes the expected number of filters upfront then sends the filter to the coordinator once all the local instances have been merged together. The merging can done in parallel locally to improve latency of filter propagation. Add Or() methods to MinMaxFilter and BloomFilter, since we now need to merge those, not just the thrift versions. Update coordinator filter routing to expect only one instance of a filter from each producer backend and to only send one instance to each consumer backend (instead of sending one per fragment). Update memory reservations and estimates to be lower to account for sharing of filters between fragment instances. mt_dop plans are modified to show these shared and non-shared resources separately. Enable waiting for runtime filters for kudu scanner with mt_dop. Made min/max filters const-correct. TODO: * Rebase onto KRPC runtime filter change Testing * Added unit tests for Or() methods. * Added some additional e2e test coverage for mt_dop queries * Updated planner tests with new estimates and reservation. * Ran a single node 3-impalad stress test with TPC-H kudu and TPC-DS parquet. * TODO: run tests under ASAN. Perf * Did a single-node perf run on TPC-H with default settings. No perf change. * Single-node perf run with mt_dop=8 showed significant speedups: +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(30) | parquet / none / none | 10.07 | -5.96% | 5.07 | -10.80%| +--+---+-++++ +--+--+---++-++---++---++-+-+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval| +--+--+---++-++---++---++-+-+ | TPCH(30) | TPCH-Q7 | parquet / none / none | 37.49 | 36.33 | +3.18% | 6.34% | 4.85%| 20| +1.90% | 3.96| 1.75| | TPCH(30) | TPCH-Q15 | parquet / none / none | 3.77 | 3.75| +0.61% | 1.20% | 1.03%| 20| +0.74% | 1.50| 1.72| | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.32 | 2.32| +0.05% | 1.62% | 2.14%| 20| -0.01% | -0.04 | 0.09| | TPCH(30) | TPCH-Q19 | parquet / none / none | 5.17 | 5.18| -0.20% | 1.56% | 1.63%| 20| -0.09% | -0.89 | -0.39 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.27 | 4.28| -0.29% | 1.09% | 1.80%| 20| -0.05% | -0.74 | -0.61 | | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.24 | 1.25| -0.35% | 3.47% | 2.95%| 20| -0.19% | -0.92 | -0.35 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 9.73 | 9.87| -1.38% | 1.22% | 1.05%| 20| -1.34% | -3.26 | -3.87 | | TPCH(30) | TPCH-Q16 | parquet / none / none | 2.49 | 2.54| -1.97% | 2.91% | 2.41%| 20| -2.07% | -2.09 | -2.36 | | TPCH(30) | TPCH-Q2 | parquet / none / none | 1.97 | 2.01| -1.91% | 2.14% | 2.57%| 20| -2.21% | -2.76 | -2.58 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 80.59 | 82.48 | -2.29% | 6.61% | 3.34%| 20| -3.67% | -3.17 | -1.41 | | TPCH(30) | TPCH-Q10 | parquet / none / none | 5.12 | 5.43| I -5.70% | 0.82% | 1.62%| 20| I -5.72% | -5.27 | -14.22 | | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.50 | 26.20 | I -6.49% | 0.47% | 0.43%| 20| I -7.00% | -5.27 | -47.60 | | TPCH(30) | TPCH-Q18 | parquet / none / none | 8.77 | 9.48| I -7.55% | 0.83% | 0.79%| 20| I -8.06% | -5.27 | -30.59 | | TPCH(30) | TPCH-Q3 | parquet / none / none | 6.05 | 6.61
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14538 ) Change subject: IMPALA-4400: aggregate runtime filters locally .. Patch Set 9: I'm planning to merge this after IMPALA-7984, but it's ready for review so thought I'd publish now. -- To view, visit http://gerrit.cloudera.org:8080/14538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc Gerrit-Change-Number: 14538 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 18:05:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Hello Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14538 to look at the new patch set (#9). Change subject: IMPALA-4400: aggregate runtime filters locally .. IMPALA-4400: aggregate runtime filters locally Move RuntimeFilterBank to QueryState(). Implement fine-grained locking for each filter to mitigate any increased lock contention from the change. Make RuntimeFilterBank handle multiple producers of the same filter, e.g. multiple instances of a partitioned join. It computes the expected number of filters upfront then sends the filter to the coordinator once all the local instances have been merged together. The merging can done in parallel locally to improve latency of filter propagation. Add Or() methods to MinMaxFilter and BloomFilter, since we now need to merge those, not just the thrift versions. Update coordinator filter routing to expect only one instance of a filter from each producer backend and to only send one instance to each consumer backend (instead of sending one per fragment). Update memory reservations and estimates to be lower to account for sharing of filters between fragment instances. mt_dop plans are modified to show these shared and non-shared resources separately. Enable waiting for runtime filters for kudu scanner with mt_dop. Made min/max filters const-correct. TODO: * Rebase onto KRPC runtime filter change Testing * Added unit tests for Or() methods. * Added some additional e2e test coverage for mt_dop queries * Updated planner tests with new estimates and reservation. * Ran a single node 3-impalad stress test with TPC-H kudu and TPC-DS parquet. * TODO: run tests under ASAN. Perf * Did a single-node perf run on TPC-H with default settings. No perf change. * Single-node perf run with mt_dop=8 showed significant speedups: +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(30) | parquet / none / none | 10.07 | -5.96% | 5.07 | -10.80%| +--+---+-++++ +--+--+---++-++---++---++-+-+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval| +--+--+---++-++---++---++-+-+ | TPCH(30) | TPCH-Q7 | parquet / none / none | 37.49 | 36.33 | +3.18% | 6.34% | 4.85%| 20| +1.90% | 3.96| 1.75| | TPCH(30) | TPCH-Q15 | parquet / none / none | 3.77 | 3.75| +0.61% | 1.20% | 1.03%| 20| +0.74% | 1.50| 1.72| | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.32 | 2.32| +0.05% | 1.62% | 2.14%| 20| -0.01% | -0.04 | 0.09| | TPCH(30) | TPCH-Q19 | parquet / none / none | 5.17 | 5.18| -0.20% | 1.56% | 1.63%| 20| -0.09% | -0.89 | -0.39 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.27 | 4.28| -0.29% | 1.09% | 1.80%| 20| -0.05% | -0.74 | -0.61 | | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.24 | 1.25| -0.35% | 3.47% | 2.95%| 20| -0.19% | -0.92 | -0.35 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 9.73 | 9.87| -1.38% | 1.22% | 1.05%| 20| -1.34% | -3.26 | -3.87 | | TPCH(30) | TPCH-Q16 | parquet / none / none | 2.49 | 2.54| -1.97% | 2.91% | 2.41%| 20| -2.07% | -2.09 | -2.36 | | TPCH(30) | TPCH-Q2 | parquet / none / none | 1.97 | 2.01| -1.91% | 2.14% | 2.57%| 20| -2.21% | -2.76 | -2.58 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 80.59 | 82.48 | -2.29% | 6.61% | 3.34%| 20| -3.67% | -3.17 | -1.41 | | TPCH(30) | TPCH-Q10 | parquet / none / none | 5.12 | 5.43| I -5.70% | 0.82% | 1.62%| 20| I -5.72% | -5.27 | -14.22 | | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.50 | 26.20 | I -6.49% | 0.47% | 0.43%| 20| I -7.00% | -5.27 | -47.60 | | TPCH(30) | TPCH-Q18 | parquet / none / none | 8.77 | 9.48| I -7.55% | 0.83% | 0.79%| 20| I -8.06% | -5.27 | -30.59 | | TPCH(30) | TPCH-Q3 | parquet / none / none | 6.05 | 6.61| I -8.51% |
[Impala-ASF-CR] IMPALA-4400: aggregate runtime filters locally
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14538 ) Change subject: IMPALA-4400: aggregate runtime filters locally .. Patch Set 9: (3 comments) http://gerrit.cloudera.org:8080/#/c/14538/9/be/src/runtime/runtime-filter-bank.h File be/src/runtime/runtime-filter-bank.h: http://gerrit.cloudera.org:8080/#/c/14538/9/be/src/runtime/runtime-filter-bank.h@193 PS9, Line 193: /// Contains references to all the min-max filters generated. Used in Close() to safely line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/14538/9/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/14538/9/fe/src/main/java/org/apache/impala/planner/Planner.java@444 PS9, Line 444: totalRuntimeFilterMemBytes += fragment.getProducedRuntimeFiltersMemReservationBytes(); line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/14538/9/tests/custom_cluster/test_mt_dop.py File tests/custom_cluster/test_mt_dop.py: http://gerrit.cloudera.org:8080/#/c/14538/9/tests/custom_cluster/test_mt_dop.py@90 PS9, Line 90: flake8: W391 blank line at end of file -- To view, visit http://gerrit.cloudera.org:8080/14538 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iabeeab5eec869ff2197250ad41c1eb5551704acc Gerrit-Change-Number: 14538 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Oct 2019 17:56:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 2: Code-Review+2 This looks good to me. -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 17:53:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
Greg Rahn has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13794 ) Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function .. IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function This patch adds new built-in functions to calculate restricted Damerau-Levenshtein edit distance (optimal string alignment). Implmented as dle_dst() and damerau_levenshtein(). If either value is NULL or both values are NULL returns NULL which differs from Netezza's dle_dst() which returns the length of the not NULL value or 0 if both values are NULL. The NULL behavior matches the existing levenshtein() function. Also cleans up levenshtein tests. Testing: - Added unit tests to expr-test.cc - Manual testing on over 1400 string pairs from http://marvin.cs.uidaho.edu/misspell.html and results match Netezza Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d --- M be/src/exprs/expr-test.cc M be/src/exprs/string-functions-ir.cc M be/src/exprs/string-functions.h M common/function-registry/impala_functions.py 4 files changed, 142 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/13794/7 -- To view, visit http://gerrit.cloudera.org:8080/13794 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d Gerrit-Change-Number: 13794 Gerrit-PatchSet: 7 Gerrit-Owner: Greg Rahn Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9047: Bump CDP BUILD NUMBER to 1471450
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/14576 ) Change subject: IMPALA-9047: Bump CDP_BUILD_NUMBER to 1471450 .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/14576 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I962423cf202ad632b5817669500b3e3479f1a454 Gerrit-Change-Number: 14576 Gerrit-PatchSet: 2 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Oct 2019 14:43:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/14291 )
Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
..
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14291/12/be/src/runtime/datetime-iso-sql-format-tokenizer.cc
File be/src/runtime/datetime-iso-sql-format-tokenizer.cc:
http://gerrit.cloudera.org:8080/#/c/14291/12/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@124
PS12, Line 124: if (token->second.type == FX_MODIFIER) {
: if (used_tokens_.size() > 0 || dt_ctx_->fx_modifier) {
: return MISPLACED_FX_MODIFIER_ERROR;
: }
: dt_ctx_->fx_modifier = true;
: *current_pos += curr_token_size;
: return SUCCESS;
: }
'FMFX-MM-DD' is another format string that should be rejected but currently
it is not.
--
To view, visit http://gerrit.cloudera.org:8080/14291
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855
Gerrit-Change-Number: 14291
Gerrit-PatchSet: 12
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 30 Oct 2019 14:22:39 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/14291 )
Change subject: IMPALA-8704: ISO:SQL:2016 datetime patterns - Milestone 2
..
Patch Set 12:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/14291/12//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14291/12//COMMIT_MSG@24
PS12, Line 24: Using this the value of a token can be
: shorter than the max length if followed by a separator.
Please clarify that this is about datetime to string conversion.
http://gerrit.cloudera.org:8080/#/c/14291/12/be/src/runtime/datetime-iso-sql-format-parser.h
File be/src/runtime/datetime-iso-sql-format-parser.h:
http://gerrit.cloudera.org:8080/#/c/14291/12/be/src/runtime/datetime-iso-sql-format-parser.h@81
PS12, Line 81: a
nit: the
http://gerrit.cloudera.org:8080/#/c/14291/12/be/src/runtime/datetime-iso-sql-format-parser.cc
File be/src/runtime/datetime-iso-sql-format-parser.cc:
http://gerrit.cloudera.org:8080/#/c/14291/12/be/src/runtime/datetime-iso-sql-format-parser.cc@238
PS12, Line 238: DCHECK(current_tok_idx != nullptr && *current_tok_idx <
dt_ctx.toks.size());
Add
&& dt_ctx.toks[*current_tok_idx].type == SEPARATOR
http://gerrit.cloudera.org:8080/#/c/14291/12/be/src/runtime/datetime-iso-sql-format-tokenizer.cc
File be/src/runtime/datetime-iso-sql-format-tokenizer.cc:
http://gerrit.cloudera.org:8080/#/c/14291/12/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@113
PS12, Line 113: if (IsStartOfTextToken(*current_pos)) {
: return ProcessTextToken(current_pos, str_begin, str_end);
: }
What if text token is preceded by an FM modifier?
I did some testing and it looks like in a format string like
'FX-MM-FM"text"DD' the FM-modifier applies to DD token, instead of the
"text" token.
I think we need to do here something like L141-144.
ProcessSeaprators() has a similar problem. In 'FX-MM-FM-DD' FM applies to
DD token instead of the '-' separator.
http://gerrit.cloudera.org:8080/#/c/14291/12/be/src/runtime/datetime-iso-sql-format-tokenizer.cc@124
PS12, Line 124: if (token->second.type == FX_MODIFIER) {
: if (used_tokens_.size() > 0 || dt_ctx_->fx_modifier) {
: return MISPLACED_FX_MODIFIER_ERROR;
: }
: dt_ctx_->fx_modifier = true;
: *current_pos += curr_token_size;
: return SUCCESS;
: }
This still allows format strings like:
'--FX-MM-DD'
or
'"text"FX-MM-DD'
Maybe it would be easier to parse the optional FX modifier before calling
ProcessNextToken() in a loop.
--
To view, visit http://gerrit.cloudera.org:8080/14291
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I30d2f6656054371476aaa8bd0d51f572b9369855
Gerrit-Change-Number: 14291
Gerrit-PatchSet: 12
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 30 Oct 2019 14:06:30 +
Gerrit-HasComments: Yes
