[Impala-ASF-CR] IMPALA-8138: Remove FAULT INJECTION RPC DELAY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13060 ) Change subject: IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4260/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Gerrit-Change-Number: 13060 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 15 May 2019 05:43:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8121: part 3: invalidate on memory pressure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13302 ) Change subject: IMPALA-8121: part 3: invalidate on memory pressure .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13302 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d55ef0058abcf70f75b10ae9d89a0274859969 Gerrit-Change-Number: 13302 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 05:21:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8121: part 3: invalidate on memory pressure
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13302 ) Change subject: IMPALA-8121: part 3: invalidate on memory pressure .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4259/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13302 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d55ef0058abcf70f75b10ae9d89a0274859969 Gerrit-Change-Number: 13302 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 05:21:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Remove FAULT INJECTION RPC DELAY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13060 ) Change subject: IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3234/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Gerrit-Change-Number: 13060 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 15 May 2019 05:20:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8545: Test Ldap authentication
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13337 ) Change subject: IMPALA-8545: Test Ldap authentication .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3233/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I92b5e60860c60209c1bd8afe5b3ea201fb7a7513 Gerrit-Change-Number: 13337 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 04:43:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Remove FAULT INJECTION RPC DELAY
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13060 ) Change subject: IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY .. Patch Set 3: Code-Review+2 carrying forward -- To view, visit http://gerrit.cloudera.org:8080/13060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Gerrit-Change-Number: 13060 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 15 May 2019 04:25:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-6903: Download profile from WebUI in text format .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3232/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 15 May 2019 04:26:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Remove FAULT INJECTION RPC DELAY
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13060 to look at the new patch set (#3). Change subject: IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY .. IMPALA-8138: Remove FAULT_INJECTION_RPC_DELAY This patch removes the FAULT_INJECTION_RPC_DELAY macro and replaces its uses with DebugAction which is more flexible. For example, it supports JITTER which injects random delays. Every backend rpc has a debug action of the form RPC_NAME_DELAY. DebugAction has previously always been used via query options. However, for the rpcs considered here there is not always a query with an accessible TQUeryOptions available (for example, we do not send any query info with the RemoteShutdown rpc), so this patch introduces a flag, '--debug_actions', which is used to control these rpc delay debug actions. Testing: - Updated existing tests to use the new mechanism. Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b --- M be/src/common/global-flags.cc M be/src/service/control-service.cc M be/src/service/data-stream-service.cc M be/src/service/impala-internal-service.cc M be/src/testutil/fault-injection-util.cc M be/src/testutil/fault-injection-util.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M tests/custom_cluster/test_rpc_timeout.py 9 files changed, 54 insertions(+), 83 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/13060/3 -- To view, visit http://gerrit.cloudera.org:8080/13060 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I712b188e0cdf91f431c9b94052501e5411af407b Gerrit-Change-Number: 13060 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-6903: Download profile from WebUI in text format .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3231/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 15 May 2019 04:17:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8545: Test Ldap authentication
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13337 to look at the new patch set (#2). Change subject: IMPALA-8545: Test Ldap authentication .. IMPALA-8545: Test Ldap authentication Currently, Impala does not have any automated tests for LDAP auth functionality, due to the challenge of setting up an LDAP server for use by the minicluster. This patch adds LDAP tests by utilizing the ApacheDS project's unit testing functionality, which works with JUnit to setup up a local LDAP server for the duration of a test suite. Change-Id: I92b5e60860c60209c1bd8afe5b3ea201fb7a7513 --- M bin/rat_exclude_files.txt M fe/pom.xml M fe/src/test/java/org/apache/impala/service/JdbcTest.java A fe/src/test/java/org/apache/impala/service/JdbcTestBase.java A fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java A fe/src/test/java/org/apache/impala/testutil/CustomClusterRunner.java M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java A fe/src/test/resources/users.ldif 8 files changed, 297 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/13337/2 -- To view, visit http://gerrit.cloudera.org:8080/13337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I92b5e60860c60209c1bd8afe5b3ea201fb7a7513 Gerrit-Change-Number: 13337 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Yongzhi Chen has posted comments on this change. (
http://gerrit.cloudera.org:8080/1 )
Change subject: IMPALA-6903: Download profile from WebUI in text format
..
Patch Set 3:
(10 comments)
Patch 3 should address most issues. I will figure out how to attach screenshot
tomorrow. Can we guarantee the test queries always go to impalad with port
25000?
http://gerrit.cloudera.org:8080/#/c/1/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/1/1//COMMIT_MSG@9
PS1, Line 9: Add a link called "Download Text Profile" to profile tab.
> Suggest adding a few more things; mention that the profile tab now includes
Done
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@546
PS1, Line 546:
> should we add another test similar to test_query_profile_encoded_unknown_qu
Add to the end of the test.
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@549
PS1, Line 549: query = "select count(*) from functional.alltypes"
> is there a race condition if this is executed asynchronously? what if the r
Done
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@553
PS1, Line 553:
> don't mix + and %s python formatting. I think we should use "{0}query_profi
Done
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@554
PS1, Line 554: id={1}"
> use IMPALAD_TEST_PORT
Done
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@555
PS1, Line 555: self.ROOT_URL, query_h
> I'm no Javascript expert, so if this approach is getting too complex. I thi
Done
http://gerrit.cloudera.org:8080/#/c/1/1/www/query_profile.tmpl
File www/query_profile.tmpl:
http://gerrit.cloudera.org:8080/#/c/1/1/www/query_profile.tmpl@34
PS1, Line 34:
> is download the name of the file that will be downloaded? if so, I suggest
Done
http://gerrit.cloudera.org:8080/#/c/1/1/www/query_profile.tmpl@43
PS1, Line 43:
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Hello Lars Volker, Fredy Wijaya, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/1 to look at the new patch set (#2). Change subject: IMPALA-6903: Download profile from WebUI in text format .. IMPALA-6903: Download profile from WebUI in text format Add a link called "Download Text Profile" to profile tab. The link allows users to download runtime profiles as UTF-8 encoded file. Get text profile from backend by making http request. Tests: Manually tested to check downloaded files. Add test_download_text_profile to test_web_pages.py Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b --- M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M tests/webserver/test_web_pages.py M www/query_profile.tmpl 4 files changed, 70 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/1/2 -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Hello Lars Volker, Fredy Wijaya, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/1 to look at the new patch set (#3). Change subject: IMPALA-6903: Download profile from WebUI in text format .. IMPALA-6903: Download profile from WebUI in text format Add a link called "Download Text Profile" to profile tab. The link allows users to download runtime profiles as UTF-8 encoded file. Get text profile from backend by making http request. Tests: Manually tested to check downloaded files. Add test_download_text_profile to test_web_pages.py Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b --- M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M tests/webserver/test_web_pages.py M www/query_profile.tmpl 4 files changed, 70 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/1/3 -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-6903: Download profile from WebUI in text format .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/1/2/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/1/2/tests/webserver/test_web_pages.py@567 PS2, Line 567: r flake8: F841 local variable 'response' is assigned to but never used -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 15 May 2019 03:21:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369 (part 3): Hive 3: fix test permanent udfs.py for Hive 3 support
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13236 ) Change subject: IMPALA-8369 (part 3): Hive 3: fix test_permanent_udfs.py for Hive 3 support .. Patch Set 7: Code-Review+2 Forwarding +2 after rebases -- To view, visit http://gerrit.cloudera.org:8080/13236 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f50845c7d4769d8843cad87988498e165902169 Gerrit-Change-Number: 13236 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 15 May 2019 02:11:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 (part 3): Hive 3: fix test permanent udfs.py for Hive 3 support
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13236 ) Change subject: IMPALA-8369 (part 3): Hive 3: fix test_permanent_udfs.py for Hive 3 support .. IMPALA-8369 (part 3): Hive 3: fix test_permanent_udfs.py for Hive 3 support This fixes two issues in test_permanent_udfs.py: - two of Hive's built-ins were ported to the new GenericUDF interface which Impala can't execute. These UDFs are now excluded from the test when running with Hive 3. - The 'hive' commandline is deprecated nowadays, so the test now uses the standard HS2 approach to run Hive queries. Hive 2+ caches UDFs, so now that we are connecting to an already-running HS2 rather than starting a new standalone 'hive' command, we need to explicitly invalidate that cache by using 'RELOAD FUNCTION' after making changes to UDFs in Impala. Change-Id: I7f50845c7d4769d8843cad87988498e165902169 Reviewed-on: http://gerrit.cloudera.org:8080/13236 Tested-by: Impala Public Jenkins Reviewed-by: Todd Lipcon --- M tests/common/impala_test_suite.py M tests/custom_cluster/test_permanent_udfs.py 2 files changed, 27 insertions(+), 34 deletions(-) Approvals: Impala Public Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13236 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7f50845c7d4769d8843cad87988498e165902169 Gerrit-Change-Number: 13236 Gerrit-PatchSet: 8 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8121: part 3: invalidate on memory pressure
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13302 ) Change subject: IMPALA-8121: part 3: invalidate on memory pressure .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13302 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d55ef0058abcf70f75b10ae9d89a0274859969 Gerrit-Change-Number: 13302 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 02:10:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8400: Implement Ranger audit event handler
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13309 ) Change subject: IMPALA-8400: Implement Ranger audit event handler .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3229/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13309 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife93c3e708875ef5fc0117153ad8ee225a88518b Gerrit-Change-Number: 13309 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 02:08:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Skip test owner privileges test when running against Hive-3
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13339 ) Change subject: IMPALA-8369 : Skip test_owner_privileges test when running against Hive-3 .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f904446f50b5095443bf27b3092a2e3665b76d3 Gerrit-Change-Number: 13339 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 02:13:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 : Skip test owner privileges test when running against Hive-3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13339 ) Change subject: IMPALA-8369 : Skip test_owner_privileges test when running against Hive-3 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3230/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9f904446f50b5095443bf27b3092a2e3665b76d3 Gerrit-Change-Number: 13339 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 15 May 2019 02:05:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8545 (part 2): Add tests for JDBC + LDAP + SSL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13338 ) Change subject: IMPALA-8545 (part 2): Add tests for JDBC + LDAP + SSL .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3228/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2af89cd374f8b8783fc29fc91ceac5513d857cb5 Gerrit-Change-Number: 13338 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 01:54:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13326 ) Change subject: IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ecc06ddfad414091bd50f683b767b61abdcc4 Gerrit-Change-Number: 13326 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 15 May 2019 01:57:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 12 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Wed, 15 May 2019 02:03:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. IMPALA-966: Attribute type error to the right expression in an insert statement Currently, if an insert statement contains multiple expressions that are incompatible with the column type, the error message returned attributes the error to the wrong expression. This patch makes sure the right expression is blamed. If there are multiple incompatible type values for the target column, then the error is attributed to the first widest (highest precision) incompatible type expression. Testing: - Added tests to AnalyzeStmtsTest.java Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Reviewed-on: http://gerrit.cloudera.org:8080/13050 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 7 files changed, 86 insertions(+), 23 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 13 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers
[Impala-ASF-CR] IMPALA-8545: Test Ldap authentication
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13337 ) Change subject: IMPALA-8545: Test Ldap authentication .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3227/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I92b5e60860c60209c1bd8afe5b3ea201fb7a7513 Gerrit-Change-Number: 13337 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 15 May 2019 01:58:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13326 ) Change subject: IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load .. IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load Whenever closing a FileGroup, the TmpFileMgr::scratch_bytes_used_metric_ was incorrectly being decremented by the total scratch space bytes across the entire FileGroup (i.e FileGroup::scratch_space_bytes_used_counter_), for every File in the FileGroup. This was resulting in the -ive value for the current scratch space bytes. The fix is to decrement the TmpFileMgr::scratch_bytes_used_metric_ by the FileGroup::scratch_space_bytes_used_counter_, only once when the FileGroup is closed. Testing Done: - Added checks for expected current value and HWM of the scratch space bytes in some of the existing test units in tmp-file-mgr-test.cc. - Added a new scenario in tmp-file-mgr-test.cc which mimics concurrent spilling queries and checks for propper current and HWM value for the scratch space bytes. - Ad-hoc tests forcing multiple scratch space dirs/files and running concurrent spilling queries while making sure the current value is never -ive. Change-Id: I338ecc06ddfad414091bd50f683b767b61abdcc4 Reviewed-on: http://gerrit.cloudera.org:8080/13326 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/util/metrics.h 3 files changed, 86 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I338ecc06ddfad414091bd50f683b767b61abdcc4 Gerrit-Change-Number: 13326 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8369 : Skip test owner privileges test when running against Hive-3
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13339 Change subject: IMPALA-8369 : Skip test_owner_privileges test when running against Hive-3 .. IMPALA-8369 : Skip test_owner_privileges test when running against Hive-3 Currently, when running with USE_CDP_HIVE=true, Sentry service's sync with HMS is very slow. This is most likely due to the fact that in HMS-3 the notification events are generated using the JSONMessageFactory provided by Metastore, unlike in case of HMS-2 setup. When running against HMS-2, Sentry provides its own MessageFactory implementation which has its limitations and cannot be used in HMS-3. In order to fix this Sentry should add support for the out-of-box message factory available in Hive-3 (See SENTRY-2518). Due to these additional delays from Sentry test_owner_privileges fails due to race conditions between the cached information in catalog and Sentry server (See IMPALA-8550). This patch disables this test when running against HMS-3 until we fix the issues both on the Sentry and Impala side. Testing done: 1. Confirmed the test is skipped when using USE_CDP_HIVE=true 2. Confirmed the test is not skipped when using USE_CDP_HIVE=false Change-Id: I9f904446f50b5095443bf27b3092a2e3665b76d3 --- M tests/authorization/test_owner_privileges.py M tests/common/skip.py 2 files changed, 8 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/13339/2 -- To view, visit http://gerrit.cloudera.org:8080/13339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9f904446f50b5095443bf27b3092a2e3665b76d3 Gerrit-Change-Number: 13339 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8400: Implement Ranger audit event handler
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13309 )
Change subject: IMPALA-8400: Implement Ranger audit event handler
..
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13309/4/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/13309/4/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@435
PS4, Line 435: authzChecker.authorize(analysisResult_, catalog_,
authzCtxConsumer);
: } catch (AuthorizationException e) {
: authException = e;
: }
:
: // AuthorizationExceptions take precedence over
AnalysisExceptions so as not
: // to reveal the existence/absence of objects the user is
not authorized to see.
: if (authException != null) throw authException;
: if (analysisException != null) throw analysisException;
: r
> Had a discussion with Fredy because I didn't understand the purpose of havi
Done. Rename the overloaded authorize to authorizeResource().
I removed the pre/postAutohrize(), but the interface is a bit leaky now with
Consumer since the authorize() is the one that handles
the creation of AuthorizationContext and creating the AuthorizationContext now
becomes a specific implementation of AuthorizationChecker
(BaseAuthorizationChecker in this case). Please take a look at the latest CR
and let me know what you think.
--
To view, visit http://gerrit.cloudera.org:8080/13309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife93c3e708875ef5fc0117153ad8ee225a88518b
Gerrit-Change-Number: 13309
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Wed, 15 May 2019 01:19:41 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8400: Implement Ranger audit event handler
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13309 ) Change subject: IMPALA-8400: Implement Ranger audit event handler .. IMPALA-8400: Implement Ranger audit event handler This patch implements Ranger audit event handler to behave similarly to the Hive/Ranger audit event handler, most notably: - Buffer the audit events during authorization and only flush them once the authorization is complete. - The audit will only add the event for the first failure. Testing: - Added test cases in RangerAuditLogTest - Ran FE tests - Ran all E2E authorization tests Change-Id: Ife93c3e708875ef5fc0117153ad8ee225a88518b --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/AuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/BaseAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/NoopAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerCatalogdAuthorizationManager.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaAuditHandler.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java A fe/src/test/java/org/apache/impala/authorization/AuthorizationTestBase.java A fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 16 files changed, 1,448 insertions(+), 826 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/09/13309/6 -- To view, visit http://gerrit.cloudera.org:8080/13309 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ife93c3e708875ef5fc0117153ad8ee225a88518b Gerrit-Change-Number: 13309 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8545 (part 2): Add tests for JDBC + LDAP + SSL
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13338 )
Change subject: IMPALA-8545 (part 2): Add tests for JDBC + LDAP + SSL
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13338/1/fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java
File fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java:
http://gerrit.cloudera.org:8080/#/c/13338/1/fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java@79
PS1, Line 79: args += String.format("
--ssl_client_ca_certificate=%1$s/be/src/testutil/server-cert.pem "
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/13338/1/fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java@81
PS1, Line 81: +
"--ssl_private_key=%1$s/be/src/testutil/server-key.pem --hostname=localhost",
System.getenv("IMPALA_HOME"));
line too long (120 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13338
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2af89cd374f8b8783fc29fc91ceac5513d857cb5
Gerrit-Change-Number: 13338
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 15 May 2019 01:01:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8545: Test Ldap authentication
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13337 Change subject: IMPALA-8545: Test Ldap authentication .. IMPALA-8545: Test Ldap authentication Currently, Impala does not have any automated tests for LDAP auth functionality, due to the challenge of setting up an LDAP server for use by the minicluster. This patch adds LDAP tests by utilizing the ApacheDS project's unit testing functionality, which works with JUnit to setup up a local LDAP server for the duration of a test suite. Change-Id: I92b5e60860c60209c1bd8afe5b3ea201fb7a7513 --- M fe/pom.xml M fe/src/test/java/org/apache/impala/service/JdbcTest.java A fe/src/test/java/org/apache/impala/service/JdbcTestBase.java A fe/src/test/java/org/apache/impala/service/LdapJdbcTest.java A fe/src/test/java/org/apache/impala/testutil/CustomClusterRunner.java M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java A fe/src/test/resources/users.ldif 7 files changed, 296 insertions(+), 85 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/13337/1 -- To view, visit http://gerrit.cloudera.org:8080/13337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I92b5e60860c60209c1bd8afe5b3ea201fb7a7513 Gerrit-Change-Number: 13337 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-6903: Download profile from WebUI in text format .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@555 PS1, Line 555: assert len(responses) == 1 > The href is dynamic, I can not find a way to get real download link for the I'm no Javascript expert, so if this approach is getting too complex. I think it might be better just to duplicate what we do for the "query_profile_encoded". The advantage of your current approach is that you save requesting the runtime-profile from the web server again, but I don't think its that big of a benefit. Ideally, the new URL works exactly like the "Download Thrift Profile" link; you can copy the URL and curl it, you can download the file, you can open it in a new tab. -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 15 May 2019 00:24:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. IMPALA-8369: Fixing some core tests in Hive environment Fixes: impala_test_suite.py: DROP PARTITIONS in the SETUP section of test files did not work with Hive 3, because 'max_parts' argument of hive_client.get_partition_names() was 0, while it should be -1 to return all partitions. The issue broke sevaral 'insert' tests. Hive 2 used to return all partitions with argument 0 too but Hive 3 changed this to be more consistent, see HIVE-18567. load_nested.py: query/test_mt_dop.py:test_parquet_filtering amd several planner tests were broken because Hive 3 generates different number of files for tpch_nested_parquet.customer than Hive 2. The fix is to split the loading of this table to two inserts on Hive 3 in order to produce an extra file. Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Reviewed-on: http://gerrit.cloudera.org:8080/13283 Tested-by: Impala Public Jenkins Reviewed-by: Vihang Karajgaonkar --- M testdata/bin/load_nested.py M tests/common/environ.py M tests/common/impala_test_suite.py 3 files changed, 38 insertions(+), 14 deletions(-) Approvals: Impala Public Jenkins: Verified Vihang Karajgaonkar: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 5 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-6903: Download profile from WebUI in text format .. Patch Set 1: (1 comment) I would like to have a better test, but do not know how to. http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@555 PS1, Line 555: assert len(responses) == 1 > should we have more assert instead of just asserting the len? The href is dynamic, I can not find a way to get real download link for the file in the html page, so can not verify more than that. In the page, it is initial value (javascript:void(0)) Any idea on how to get the real link with the response? -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 14 May 2019 23:53:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 14 May 2019 23:44:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13251 ) Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4252/ -- To view, visit http://gerrit.cloudera.org:8080/13251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Gerrit-Change-Number: 13251 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 14 May 2019 23:42:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8400: Implement Ranger audit event handler
Austin Nobis has posted comments on this change. (
http://gerrit.cloudera.org:8080/13309 )
Change subject: IMPALA-8400: Implement Ranger audit event handler
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13309/4/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/13309/4/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@435
PS4, Line 435: try {
: authzCtx = authzChecker.preAuthorize(analysisResult_,
catalog_);
: authzChecker.authorize(authzCtx, analysisResult_,
catalog_);
: } catch (AuthorizationException e) {
: authException = e;
: } finally {
: if (authzCtx != null) {
: authzChecker.postAuthorize(authzCtx, analysisResult_,
catalog_);
: }
: }
Had a discussion with Fredy because I didn't understand the purpose of having
`preAuthorize`, `authorize`, and then `postAuthorize` called sequentially.
The `BaseAuthorizationChecker` class has 2 `authorize` methods with different
signatures. The `abstract` one is the one that is being `override`'d. This is
why these 3 methods can't be merged into a single `authorize`.
Fredy and I came to an agreement that we should rename the `abstract authorize`
method to `authorizeResource` as it is an authorization that occurs per
resources, whereas the `authorize` being called here is the authorization for
the entire SQL statement.
The logic flow is as follows:
RangerAuthorizationChecker#preAuthorize
BaseAuthorizationChecker#authorize
foreach resource: RangerAuthorizationChecker#authorize (authorizeResource)
RangerAuthorizationChecker#postAuthorize
I still think the `preAuthorize`, `authorize`, `postAuthorize` sequence is a
little strange as the methods execute sequentially.
--
To view, visit http://gerrit.cloudera.org:8080/13309
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife93c3e708875ef5fc0117153ad8ee225a88518b
Gerrit-Change-Number: 13309
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Tue, 14 May 2019 22:44:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/1 )
Change subject: IMPALA-6903: Download profile from WebUI in text format
..
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/1/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/1/1//COMMIT_MSG@9
PS1, Line 9: Get query profile string and create URL to download
Suggest adding a few more things; mention that the profile tab now includes a
link called "Download Profile" that allows users to download runtime profiles
as UTF-8 encoded strings.
I would document the technical approach a bit more here as well. Reading
through the code, it looks like rather than making a separate HTTP request to
the backend for the profile, this change just takes the runtime profile that
has already been loaded onto the web-page, and just allows users to download it
as a text file.
I would suggest adding a few screenshots to the JIRA of the changes as well.
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@546
PS1, Line 546: def test_download_textprofile(self):
should we add another test similar to
test_query_profile_encoded_unknown_query_id? or update
test_query_profile_encoded_unknown_query_id to do the same thing for text
profiles
nit: test_download_textprofile --> test_download_text_profile
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@549
PS1, Line 549: query_handle = self.client.execute_async(query)
is there a race condition if this is executed asynchronously? what if the
runtime profile is not ready by the time it is requested below?
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@555
PS1, Line 555: assert len(responses) == 1
> should we have more assert instead of just asserting the len?
+1
I think we could add some asserts on the response data to make sure we are
getting back a valid text profile for the given query. I think
test_admission_controller.py has some examples of parsing runtime profiles.
http://gerrit.cloudera.org:8080/#/c/1/1/www/query_profile.tmpl
File www/query_profile.tmpl:
http://gerrit.cloudera.org:8080/#/c/1/1/www/query_profile.tmpl@34
PS1, Line 34:
is download the name of the file that will be downloaded? if so, I suggest
renaming the file to 'profile_{{query_id}}' and changing the thrift encoded
version to 'thrift_profile_{{query_id}}'. Not sure how others feel, but I think
that makes more sense.
http://gerrit.cloudera.org:8080/#/c/1/1/www/query_profile.tmpl@43
PS1, Line 43: var contents =
document.getElementsByTagName("pre")[0].innerText;
I think it would make more sense to use getElementById instead of
getElementsByTagName, similar to what you have done for the txtprofile
http://gerrit.cloudera.org:8080/#/c/1/1/www/query_profile.tmpl@49
PS1, Line 49: var dl = document.getElementById("txtprofile");
nit: txtprofile -> text_profile
--
To view, visit http://gerrit.cloudera.org:8080/1
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b
Gerrit-Change-Number: 1
Gerrit-PatchSet: 1
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Tue, 14 May 2019 22:38:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8072: addendum: don't require fe rebuild for config
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13288 ) Change subject: IMPALA-8072: addendum: don't require fe rebuild for config .. IMPALA-8072: addendum: don't require fe rebuild for config Previously config changes wouldn't be picked up by containers until maven copied the files from fe/src/test/resources to fe/target/test-classes. This makes it more convenient - after running ./bin/create-test-configuration.sh new configs are picked up by any newly-run containers. Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733 Reviewed-on: http://gerrit.cloudera.org:8080/13288 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/start-impala-cluster.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733 Gerrit-Change-Number: 13288 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8072: addendum: don't require fe rebuild for config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13288 ) Change subject: IMPALA-8072: addendum: don't require fe rebuild for config .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733 Gerrit-Change-Number: 13288 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 14 May 2019 22:23:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12797 ) Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3226/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11 Gerrit-Change-Number: 12797 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 May 2019 22:23:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13215 ) Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3225/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 14 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 14 May 2019 22:20:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12797 ) Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4256/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11 Gerrit-Change-Number: 12797 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 May 2019 21:48:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12797 ) Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. Patch Set 16: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4256/ -- To view, visit http://gerrit.cloudera.org:8080/12797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11 Gerrit-Change-Number: 12797 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 May 2019 21:53:26 + Gerrit-HasComments: No
[Impala-ASF-CR] [acid] Predicate to test if dir must be included.
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13091 ) Change subject: [acid] Predicate to test if dir must be included. .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13091/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13091/3//COMMIT_MSG@7 PS3, Line 7: [acid] Predicate to test if dir must be included. Just passing by, but I think this CR should warrant a JIRA. -- To view, visit http://gerrit.cloudera.org:8080/13091 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0e88281d277127c9499d37b95fbba55dcc7761c Gerrit-Change-Number: 13091 Gerrit-PatchSet: 3 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 14 May 2019 22:02:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8435: Prohibit write operations on ACID table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435: Prohibit write operations on ACID table. .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3223/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 5 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 14 May 2019 22:02:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8400: Implement Ranger audit event handler
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13309 ) Change subject: IMPALA-8400: Implement Ranger audit event handler .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3224/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13309 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife93c3e708875ef5fc0117153ad8ee225a88518b Gerrit-Change-Number: 13309 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 14 May 2019 22:02:33 + Gerrit-HasComments: No
[Impala-ASF-CR] [acid] Predicate to test if dir must be included.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13091 ) Change subject: [acid] Predicate to test if dir must be included. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3222/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13091 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0e88281d277127c9499d37b95fbba55dcc7761c Gerrit-Change-Number: 13091 Gerrit-PatchSet: 3 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 14 May 2019 21:58:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-6903: Download profile from WebUI in text format .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3221/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Tue, 14 May 2019 21:57:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8497: dealing with query ends with '\n'
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13293 )
Change subject: IMPALA-8497: dealing with query ends with '\n'
..
Patch Set 5:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/13293/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13293/5//COMMIT_MSG@7
PS5, Line 7: dealing with query ends with '\n'
nit: Fix ArrayIndexOutOfBoundsException for queries that end with '\n'
http://gerrit.cloudera.org:8080/#/c/13293/5//COMMIT_MSG@9
PS5, Line 9: sql
nit: the query
http://gerrit.cloudera.org:8080/#/c/13293/5//COMMIT_MSG@10
PS5, Line 10: syntax error
nit: a syntax error
http://gerrit.cloudera.org:8080/#/c/13293/5//COMMIT_MSG@11
PS5, Line 11: This situation usually occured when sql submit by hive jdbc in
code,
: user sometimes have their own BI tools,and may submit sql ends
: with '\n' to impala server.
nit: The bug only affects the queries submitted directly to Impala outside
Impala shell.
http://gerrit.cloudera.org:8080/#/c/13293/5/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/13293/5/fe/src/main/cup/sql-parser.cup@182
PS5, Line 182: int errorIndex = stmt.endsWith("\n") ? errorToken_.left - 2 :
errorToken_.left - 1;
i don't think this code is quite correct. I think the code will throw
ArrayIndexOutOfBoundsException when there are multiple \n at the end?
http://gerrit.cloudera.org:8080/#/c/13293/5/fe/src/test/java/org/apache/impala/analysis/ParserTest.java
File fe/src/test/java/org/apache/impala/analysis/ParserTest.java:
http://gerrit.cloudera.org:8080/#/c/13293/5/fe/src/test/java/org/apache/impala/analysis/ParserTest.java@3440
PS5, Line 3440: "SELECT\n"
can we have another test case for "SELECT\n\n"?
http://gerrit.cloudera.org:8080/#/c/13293/5/fe/src/test/java/org/apache/impala/analysis/ParserTest.java@3445
PS5, Line 3445: Expected: ALL, CASE, CAST, DEFAULT, DISTINCT, EXISTS, FALSE, "
you missed the DATE, i.e.
Expected: ALL, CASE, CAST, DATE, ALL, CASE, CAST, DATE,
--
To view, visit http://gerrit.cloudera.org:8080/13293
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3f034b351d0468a3f6482e27ddef818b34d8
Gerrit-Change-Number: 13293
Gerrit-PatchSet: 5
Gerrit-Owner: wangsheng
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Tue, 14 May 2019 21:55:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Tim Armstrong has removed a vote on this change. Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/12797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11 Gerrit-Change-Number: 12797 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12797 ) Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. Patch Set 16: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4257/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11 Gerrit-Change-Number: 12797 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 May 2019 21:52:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12797 ) Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs .. Patch Set 16: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11 Gerrit-Change-Number: 12797 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 May 2019 21:48:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/12797 )
Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
..
Patch Set 15: Code-Review+2
(3 comments)
carry
http://gerrit.cloudera.org:8080/#/c/12797/12/be/src/exprs/scalar-fn-call.cc
File be/src/exprs/scalar-fn-call.cc:
http://gerrit.cloudera.org:8080/#/c/12797/12/be/src/exprs/scalar-fn-call.cc@143
PS12, Line 143: // this function is invoked. staging_input_vals is
preallocated here
: // so they can be reused across calls. If we have a codegen'd
entry point
: // for this expression, allocating these input values may be
unnecessary,
: // but they only add a small constant overhead on top of the
ScalarExpr tree, so
: // we always allocate them for simplicity.
: vector* input_vals =
fn_ctx->impl()->staging_input_vals();
: for (int i = 0; i < NumFixedArgs(); ++i) {
: AnyVal* input_val;
: RETURN_IF_ERROR(AllocateAnyVal(state,
eval->expr_perm_pool(), children_[i]->type(),
:
> I suppose an alternative is to do:
Updated the comment (it was a bit stale anyway).
http://gerrit.cloudera.org:8080/#/c/12797/13/be/src/exprs/scalar-fn-call.cc
File be/src/exprs/scalar-fn-call.cc:
http://gerrit.cloudera.org:8080/#/c/12797/13/be/src/exprs/scalar-fn-call.cc@142
PS13, Line 142: Prepare staging_input_vals in case the inte
> Prepare input_vals in case the interpreted path is invoked during initializ
Done
http://gerrit.cloudera.org:8080/#/c/12797/13/be/src/exprs/scalar-fn-call.cc@144
PS13, Line 144: a
> nit: extra .
Done
--
To view, visit http://gerrit.cloudera.org:8080/12797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11
Gerrit-Change-Number: 12797
Gerrit-PatchSet: 15
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 14 May 2019 21:48:26 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8438: Store WriteId and ValidWriteId list for table and partition
Hello Vihang Karajgaonkar, Sudhanshu Arora, Zoltan Borok-Nagy, Todd Lipcon, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13215 to look at the new patch set (#14). Change subject: IMPALA-8438: Store WriteId and ValidWriteId list for table and partition .. IMPALA-8438: Store WriteId and ValidWriteId list for table and partition This happens when tables load metadata from HMS. Add MetastoreShim functions to support HMS3 only functions. Add validwriteIdlists to query profile through timeline. Tests: Manually tests HMS2 and HMS3, using log files to check Unit tests against HMS3 ToDo: WriteId and valid writeIds can be fetched in other time, need more study on that. Profile example: Query Compilation: 5s057ms - Metadata load started: 63.006ms (63.006ms) - Metadata load finished. loaded-tables=2/2...: 4s801ms (4s738ms) - Loaded ValidWriteIdLists: acid.insert_only_no_partitions:6:9223372036854775807:: acid.insert_only_with_partitions:3:9223372036854775807:: : 4s921ms (120.580ms) - Analysis finished: 4s929ms (8.013ms) Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 --- M common/thrift/CatalogObjects.thrift A fe/src/compat-hive-2/java/org/apache/hadoop/hive/common/ValidWriteIdList.java M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsPartition.java M fe/src/main/java/org/apache/impala/catalog/FeTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsPartition.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java 15 files changed, 346 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/13215/14 -- To view, visit http://gerrit.cloudera.org:8080/13215 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6edbd64424edf0ba88af110ab8b958a1966b8b54 Gerrit-Change-Number: 13215 Gerrit-PatchSet: 14 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12797 )
Change subject: IMPALA-4356,IMPALA-7331: codegen all ScalarExprs
..
Patch Set 13: Code-Review+2
(3 comments)
http://gerrit.cloudera.org:8080/#/c/12797/12/be/src/exprs/scalar-fn-call.cc
File be/src/exprs/scalar-fn-call.cc:
http://gerrit.cloudera.org:8080/#/c/12797/12/be/src/exprs/scalar-fn-call.cc@143
PS12, Line 143: // staging_input_vals, which will be reused across calls to
scalar_fn_ on the
: // interpreted code path..
: vector* input_vals =
fn_ctx->impl()->staging_input_vals();
: for (int i = 0; i < NumFixedArgs(); ++i) {
: AnyVal* input_val;
: RETURN_IF_ERROR(AllocateAnyVal(state,
eval->expr_perm_pool(), children_[i]->type(),
: "Could not allocate expression value", _val));
: input_vals->push_back(input_val);
: }
:
> Yes, the problem I ran into is that code like the below code to GetConstVal
I suppose an alternative is to do:
bool is_interpreted = codegend_compute_fn_ == nullptr;
Of course, this makes assumption about the order in which codegend_compute_fn_
is populated and when this function is called so it may add unnecessary
complication. So, I think it's okay to leave it as-is but would be nice to
briefly document the reasoning behind it.
http://gerrit.cloudera.org:8080/#/c/12797/13/be/src/exprs/scalar-fn-call.cc
File be/src/exprs/scalar-fn-call.cc:
http://gerrit.cloudera.org:8080/#/c/12797/13/be/src/exprs/scalar-fn-call.cc@142
PS13, Line 142: We're in the interpreted path (i.e. no JIT)
Prepare input_vals in case the interpreted path is invoked during
initialization (e.g. GetConstantValue() below).
http://gerrit.cloudera.org:8080/#/c/12797/13/be/src/exprs/scalar-fn-call.cc@144
PS13, Line 144: .
nit: extra .
--
To view, visit http://gerrit.cloudera.org:8080/12797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I839d7a3a2f5e1309c33a1f66013ef11628c5dc11
Gerrit-Change-Number: 12797
Gerrit-PatchSet: 13
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 14 May 2019 21:30:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/1 )
Change subject: IMPALA-6903: Download profile from WebUI in text format
..
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@553
PS1, Line 553: self.ROOT_URL + "query_profile?query_id=%s"
don't mix + and %s python formatting. I think we should use
"{0}query_profile?query_id={1}" instead.
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@554
PS1, Line 554: [25000]
use IMPALAD_TEST_PORT
http://gerrit.cloudera.org:8080/#/c/1/1/tests/webserver/test_web_pages.py@555
PS1, Line 555: assert len(responses) == 1
should we have more assert instead of just asserting the len?
http://gerrit.cloudera.org:8080/#/c/1/1/www/query_profile.tmpl
File www/query_profile.tmpl:
http://gerrit.cloudera.org:8080/#/c/1/1/www/query_profile.tmpl@47
PS1, Line 47: );
nit: join this to L46
--
To view, visit http://gerrit.cloudera.org:8080/1
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b
Gerrit-Change-Number: 1
Gerrit-PatchSet: 1
Gerrit-Owner: Yongzhi Chen
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Tue, 14 May 2019 21:26:29 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8435: Prohibit write operations on ACID table.
Sudhanshu Arora has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13311 Change subject: IMPALA-8435: Prohibit write operations on ACID table. .. IMPALA-8435: Prohibit write operations on ACID table. This change will be reverted once we add write support for insert only table. Testing Done: - Added a new test for insert statement. Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java 6 files changed, 35 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13311/5 -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 5 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8424: [DOCS] Support for column-level permissions on views
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13310 ) Change subject: IMPALA-8424: [DOCS] Support for column-level permissions on views .. Patch Set 1: (4 comments) Do we have a doc somewhere that says column-level permissions/privileges on views aren't supported? If yes, I think we should update that doc by removing that info. If not, I guess the users can assume column-level permissions are supported on both tables and views. http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_grant.xml File docs/topics/impala_grant.xml: http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_grant.xml@62 PS1, Line 62: | VIEW This is incorrect, the syntax is still "TABLE" even for view. http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_grant.xml@71 PS1, Line 71: The object_type of VIEW is supported : for the SELECT(column_name) privilege. : Remove this. This is incorrect. http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_revoke.xml File docs/topics/impala_revoke.xml: http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_revoke.xml@63 PS1, Line 63: | VIEW similarly, remove this. http://gerrit.cloudera.org:8080/#/c/13310/1/docs/topics/impala_revoke.xml@65 PS1, Line 65: : The object_type of VIEW is supported : for the SELECT(column_name) privilege. : : remove this as well. -- To view, visit http://gerrit.cloudera.org:8080/13310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f95b37891b618b71460cd2ad444a422371455a6 Gerrit-Change-Number: 13310 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 14 May 2019 21:00:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [acid] Predicate to test if dir must be included.
Sudhanshu Arora has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13091 ) Change subject: [acid] Predicate to test if dir must be included. .. [acid] Predicate to test if dir must be included. The predicate is currently not set anywhere but the plan is to modify HdfsTable and use the validWriteIdList there in the predicate implementation and remove the directories that should not be used. Testing Done: - mvn test -Dtest=FileMetadataLoaderTest Change-Id: If0e88281d277127c9499d37b95fbba55dcc7761c --- M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java 6 files changed, 66 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/13091/3 -- To view, visit http://gerrit.cloudera.org:8080/13091 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If0e88281d277127c9499d37b95fbba55dcc7761c Gerrit-Change-Number: 13091 Gerrit-PatchSet: 3 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Yongzhi Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/1 Change subject: IMPALA-6903: Download profile from WebUI in text format .. IMPALA-6903: Download profile from WebUI in text format Get query profile string and create URL to download it as UTF-8 text file. Tests: Manually tested to check downloaded files. Add test_download_textprofile to test_web_pages.py Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b --- M tests/webserver/test_web_pages.py M www/query_profile.tmpl 2 files changed, 31 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/1/1 -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen
[Impala-ASF-CR] [acid] Disallow any operation on full acid table.
Sudhanshu Arora has abandoned this change. ( http://gerrit.cloudera.org:8080/13047 ) Change subject: [acid] Disallow any operation on full acid table. .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/13047 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ifb92e5b691bf192980f2cc7d68c491bb1e8455ac Gerrit-Change-Number: 13047 Gerrit-PatchSet: 2 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 12 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Tue, 14 May 2019 20:45:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4255/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 12 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Tue, 14 May 2019 20:45:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 11 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Tue, 14 May 2019 20:45:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13326 ) Change subject: IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ecc06ddfad414091bd50f683b767b61abdcc4 Gerrit-Change-Number: 13326 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 May 2019 20:34:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13326 ) Change subject: IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4254/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ecc06ddfad414091bd50f683b767b61abdcc4 Gerrit-Change-Number: 13326 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 May 2019 20:34:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13326 ) Change subject: IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I338ecc06ddfad414091bd50f683b767b61abdcc4 Gerrit-Change-Number: 13326 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 May 2019 20:34:17 + Gerrit-HasComments: No
[native-toolchain-CR] Update cmake to 3.14.3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13286 ) Change subject: Update cmake to 3.14.3 .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ad46a7b820bbb727ce47219f03f1d4a36301ab4 Gerrit-Change-Number: 13286 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 14 May 2019 20:32:15 + Gerrit-HasComments: No
[native-toolchain-CR] Update cmake to 3.14.3
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13286 ) Change subject: Update cmake to 3.14.3 .. Update cmake to 3.14.3 This updates to the latest cmake, which includes a few improvements that may be helpful. In particular, the Ninja generator avoids generating some false dependencies[1] Instead of carrying over the patch to change one of the #defines in the CMake source, this uses a newly-provided variable when building CMake to accomplish the same. I tested building Impala with this version of CMake on el7 and it worked fine. [1] https://gitlab.kitware.com/cmake/cmake/merge_requests/430 Change-Id: I1ad46a7b820bbb727ce47219f03f1d4a36301ab4 Reviewed-on: http://gerrit.cloudera.org:8080/13286 Reviewed-by: Laszlo Gaal Tested-by: Tim Armstrong --- M init.sh M source/cmake/build.sh 2 files changed, 7 insertions(+), 2 deletions(-) Approvals: Laszlo Gaal: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/13286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1ad46a7b820bbb727ce47219f03f1d4a36301ab4 Gerrit-Change-Number: 13286 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[native-toolchain-CR] Update cmake to 3.14.3
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/13286 ) Change subject: Update cmake to 3.14.3 .. Patch Set 2: Code-Review+2 The last patch set passed our internal automated build. The two (remarkably similar) symbol names look more like bad design than a bug, with the goal to distinguish two different roles: - KWSYS_PROCESS_USE_SELECT gates whether you can pass in a preference for using select() calls at all. - KWSYSPE_USE_SELECT is the actual preprocessor symbol that disables some part of the code. But it doesn't really matter; at the end of the day you still need to define both (and both with a value) to get the desired result. -- To view, visit http://gerrit.cloudera.org:8080/13286 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ad46a7b820bbb727ce47219f03f1d4a36301ab4 Gerrit-Change-Number: 13286 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 14 May 2019 20:11:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1653: Don't close hiveserver2 session when connection is closed
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13306 ) Change subject: IMPALA-1653: Don't close hiveserver2 session when connection is closed .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG@28 PS3, Line 28: timeout : is off by default May be worth considering setting it to some large values instead of being "off". http://gerrit.cloudera.org:8080/#/c/13306/3//COMMIT_MSG@30 PS3, Line 30: We may want to consider changing : some of this behavior. We probably should consider removing states of expired sessions or Impala may risk being attacked indirectly by clients which open many sessions and never close them. Previously, a client will need to keep a connection opened to DoS Impala but with this change, it doesn't even have to. -- To view, visit http://gerrit.cloudera.org:8080/13306 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia4555cd9b73db5b4dde92cd4fac4f9bfa3664d78 Gerrit-Change-Number: 13306 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Tue, 14 May 2019 20:14:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13300 ) Change subject: IMPALA-8447: [DOCS] INSERT event is supported in automatic invalidation .. Patch Set 1: (10 comments) http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml File docs/topics/impala_metadata.xml: http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@135 PS1, Line 135: notifications should it be "notification events"? http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@146 PS1, Line 146: ADD, or DROP their : partitions After IMPALA-7973, the add, drop and alter partition events trigger a refresh table action. Alter table is still invalidates the table http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@161 PS1, Line 161: Refreshes I think may be we should also mention that if the table is not loaded, the event processor does not refresh the table. This is done to avoid loading the table unnecessarily in the catalog in case the table is unlikely to be used in the future. http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@198 PS1, Line 198: non-zero may be say positive value to be accurate. Non-zero implies negative values are okay too. http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@208 PS1, Line 208: such as adding new data to existing : tables/partitions from Spark I think this is a very generic statement which may cause users to avoid using this feature. We should try to be more specific on exact cases where it may not work when using spark. We can do this in a separate patch if you like. http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@221 PS1, Line 221: dump data directly I think a better way to say this would be ".. and add or remove data into table by adding files directly on the filesystem". We should also recommend using Hive's LOAD DATA command to do the data load in such cases, so that event processor can act on the events generated by the LOAD command. http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@233 PS1, Line 233: Inserts into tables from Hive are ignored. This can be misinterpreted as insert from Hive is not supported. I think insert from hive works from the end-user's perspective. Whether it works because of the underlying invalidate or refresh is a implementation detail which they will most likely not care about. So may be just skip this line? http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@237 PS1, Line 237: invalidate invalidated. Also, I think this is a implementation detail which do not need to be documented. http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@242 PS1, Line 242: Unlike other types of events, inserts by Impala triggers table refresh This can be removed. I think its implementation detail and users may not get any additional value by knowing this. http://gerrit.cloudera.org:8080/#/c/13300/1/docs/topics/impala_metadata.xml@267 PS1, Line 267: org.apache.hive.hcatalog.listener.DbNotificationListener After IMPALA-7971, we also need hive.metastore.dml.events set to true -- To view, visit http://gerrit.cloudera.org:8080/13300 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I68133b0beeb15cacc73829b8a8b0838fc7f4b7d8 Gerrit-Change-Number: 13300 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 14 May 2019 19:18:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Enable data cache by default for all S3 builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13330 ) Change subject: Enable data cache by default for all S3 builds .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3220/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13330 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09117ab289c2355408212a5fc6493ab751f4853b Gerrit-Change-Number: 13330 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 14 May 2019 18:34:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13251 ) Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3219/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Gerrit-Change-Number: 13251 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 14 May 2019 18:38:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8344: Add support for running the minicluster with S3Guard
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13020 ) Change subject: IMPALA-8344: Add support for running the minicluster with S3Guard .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13020 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c748529a494bb6e70fec96dc031523ff79bf61d Gerrit-Change-Number: 13020 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Tue, 14 May 2019 18:31:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13251 ) Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading .. Patch Set 5: Code-Review+2 This makes sense to me. -- To view, visit http://gerrit.cloudera.org:8080/13251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Gerrit-Change-Number: 13251 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 14 May 2019 18:21:06 + Gerrit-HasComments: No
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13185 ) Change subject: fe: set classpath using maven dependency resolution .. fe: set classpath using maven dependency resolution This changes the FE pom to generate a build classpath file in the target/ directory. Then, bin/set-classpath.sh uses this file to generate the classpath to start the cluster. This replaces the former approach of including all of the jars found in target/dependency/ The advantage of this is that a clean build is no longer required when switching artifact versions. Prior to this patch, if you changed an artifact version and rebuilt, both the old and new artifact would be left in the target/dependency/ directory and pollute the classpath. This doesn't fully remove the target/dependency/ directory, because its existence is likely important for downstream packaging of Impala. We can likely assume that such packaging always does a clean build. This also changes the set-classpath script to no longer load jars from testdata/target/dependency/ since it appears that directory doesn't actually get created during the build. Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Reviewed-on: http://gerrit.cloudera.org:8080/13185 Tested-by: Impala Public Jenkins Reviewed-by: Todd Lipcon --- M bin/set-classpath.sh M docker/setup_build_context.py M fe/pom.xml 3 files changed, 36 insertions(+), 16 deletions(-) Approvals: Impala Public Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 7 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] fe: set classpath using maven dependency resolution
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13185 ) Change subject: fe: set classpath using maven dependency resolution .. Patch Set 6: Code-Review+2 Forwarding +2 -- To view, visit http://gerrit.cloudera.org:8080/13185 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I103a1da10a54c7525ba7fb584d942ba1cb9fcb94 Gerrit-Change-Number: 13185 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 14 May 2019 18:23:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13251 ) Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Gerrit-Change-Number: 13251 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 14 May 2019 18:22:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13251 ) Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4252/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Gerrit-Change-Number: 13251 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 14 May 2019 18:22:32 + Gerrit-HasComments: No
[Impala-ASF-CR] Enable data cache by default for all S3 builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13330 ) Change subject: Enable data cache by default for all S3 builds .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3218/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13330 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I09117ab289c2355408212a5fc6493ab751f4853b Gerrit-Change-Number: 13330 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Tue, 14 May 2019 18:17:36 + Gerrit-HasComments: No
[Impala-ASF-CR] Enable data cache by default for all S3 builds
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13330 to look at the new patch set (#2). Change subject: Enable data cache by default for all S3 builds .. Enable data cache by default for all S3 builds Change-Id: I09117ab289c2355408212a5fc6493ab751f4853b --- M bin/run-all-tests.sh 1 file changed, 11 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/13330/2 -- To view, visit http://gerrit.cloudera.org:8080/13330 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I09117ab289c2355408212a5fc6493ab751f4853b Gerrit-Change-Number: 13330 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
Hello Vihang Karajgaonkar, Zoltan Borok-Nagy, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13251 to look at the new patch set (#5). Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading .. IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading This fixes three issues for functional dataset loading: - works around HIVE-21675, a bug in which 'CREATE VIEW IF NOT EXISTS' does not function correctly in our current Hive build. This has been fixed already, but the workaround is pretty simple, and actually the 'drop and recreate' pattern is used more widely for data-loading than the 'create if not exists' one. - Moves the creation of the 'hive_index' table from load-dependent-tables.sql to a new load-dependent-tables-hive2.sql file which is only executed on Hive 2. - Moving from MR to Tez execution changed the behavior of data loading by disabling the auto-merging of small files. With Hive-on-MR, this behavior defaulted to true, but with Hive-on-Tez it defaults false. The change is likely motivated by the fact that Tez automatically groups small splits on the _input_ side and thus is less likely to produce lots of small files. However, that grouping functionality doesn't work properly in localhost clusters (TEZ-3310) so we aren't seeing the benefit. So, this patch enables the post-process merging of small files. Prior to this change, the 'alltypesaggmultifilesnopart' test table was getting 40+ files inside it, which broke various planner tests. With the change, it gets the expected 4 files. Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd --- M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/resources/hive-site.xml.py M testdata/bin/create-load-data.sh A testdata/bin/load-dependent-tables-hive2.sql M testdata/bin/load-dependent-tables.sql M testdata/datasets/functional/functional_schema_template.sql 6 files changed, 60 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/13251/5 -- To view, visit http://gerrit.cloudera.org:8080/13251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Gerrit-Change-Number: 13251 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13329 ) Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4250/ -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 14 May 2019 18:12:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369: Fixing some core tests in Hive environment
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13283 ) Change subject: IMPALA-8369: Fixing some core tests in Hive environment .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45d9b9312c6c77f436ab020ae68c15f3c7c737de Gerrit-Change-Number: 13283 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 14 May 2019 18:01:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Alice Fan has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. Patch Set 11: Checked failed tests in verified build. Fixed the relevant test 'AnalyzeExprsTest.TestDecimalCasts' by converting expression of error message to all upper case. 'metadata.test_compute_stats.TestComputeStats.test_compute_stats_compression_codec' and 'metadata.test_refresh_partition.TestRefreshPartition.test_refresh_partition_num_rows' are irrelevant to the change. Tested the patch 11 at private-parameterized build and these two tests are both passed. Link to patch 11's private-parameterized build: https://master-02.jenkins.cloudera.com/job/impala-private-parameterized/5035/ -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 11 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Tue, 14 May 2019 17:54:59 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/13307 ) Change subject: WIP IMPALA-8536: Add Scalable Pool Configuration to Admission Controller. .. Patch Set 2: I have realized that there is a big mistake in this code. I had been basing AdmisisionController's idea of the cluster size on the number of hosts specified in the QuerySchedule. This might be a bit out-of-date but it seemed a reasonable proxy. But I forgot that some queries run on the coordinator only and so these queries 'see' a cluster size of 1. I'm now going to get the cluster size directly from the Scheduler. At present I am making GetExecutorsConfig() and SetExecutorsConfig() public, but when IMPALA-8460 is done then I can use ClusterMembershipMgr instead. In GetMaxToDequeue() there is no QuerySchedule so I was estimating the cluster size by looking at the size of the RemoteStatsMap. This was maybe OK but I will change this to also get the cluster size from the Scheduler. -- To view, visit http://gerrit.cloudera.org:8080/13307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188 Gerrit-Change-Number: 13307 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 14 May 2019 17:45:34 + Gerrit-HasComments: No
[Impala-ASF-CR] Enable data cache by default for all S3 builds
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13330 Change subject: Enable data cache by default for all S3 builds .. Enable data cache by default for all S3 builds Change-Id: I09117ab289c2355408212a5fc6493ab751f4853b --- M bin/run-all-tests.sh 1 file changed, 11 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/13330/1 -- To view, visit http://gerrit.cloudera.org:8080/13330 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I09117ab289c2355408212a5fc6493ab751f4853b Gerrit-Change-Number: 13330 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] IMPALA-8072: addendum: don't require fe rebuild for config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13288 ) Change subject: IMPALA-8072: addendum: don't require fe rebuild for config .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4251/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733 Gerrit-Change-Number: 13288 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 14 May 2019 17:19:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8072: addendum: don't require fe rebuild for config
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13288 ) Change subject: IMPALA-8072: addendum: don't require fe rebuild for config .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I18f9f90667b1d16cf97d3e3f9fac400980d5b733 Gerrit-Change-Number: 13288 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 14 May 2019 17:19:35 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/13329 )
Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from
ParquetColumnReader
..
Patch Set 2:
(8 comments)
Did a first pass, will take another round tomorrow.
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-column-readers.cc
File be/src/exec/parquet/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-column-readers.cc@1124
PS2, Line 1124: if (data_size == 0) {
: return CreateDictionaryDecoder(nullptr, 0, _decoder);
: }
nit: fits single-line
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-column-readers.cc@1178
PS2, Line 1178: if (eos) {
: return HandleTooEarlyEos();
: }
nit: single line
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-page-reader.h
File be/src/exec/parquet/parquet-page-reader.h:
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-page-reader.h@39
PS2, Line 39: /// Moved to implementation to be able to forward declare class
in scoped_ptr.
nit: Please rephrase, I can't parse this sentence.
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-page-reader.h@83
PS2, Line 83: &
nit: output parameters usually have pointer types
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-page-reader.h@91
PS2, Line 91: slot_desc
has_slot_desc
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-page-reader.cc
File be/src/exec/parquet/parquet-page-reader.cc:
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-page-reader.cc@60
PS2, Line 60: ParquetColumnReader
ParquetPageReader
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-page-reader.cc@382
PS2, Line 382:
nit: indent
http://gerrit.cloudera.org:8080/#/c/13329/2/be/src/exec/parquet/parquet-page-reader.cc@413
PS2, Line 413:
nit: indent
--
To view, visit http://gerrit.cloudera.org:8080/13329
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12
Gerrit-Change-Number: 13329
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 14 May 2019 17:00:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13050 ) Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3217/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 11 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Tue, 14 May 2019 16:58:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-966: Attribute type error to the right expression in an insert statement
Hello Bharath Vissapragada, Paul Rogers, xiaom...@cloudera.com, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13050 to look at the new patch set (#11). Change subject: IMPALA-966: Attribute type error to the right expression in an insert statement .. IMPALA-966: Attribute type error to the right expression in an insert statement Currently, if an insert statement contains multiple expressions that are incompatible with the column type, the error message returned attributes the error to the wrong expression. This patch makes sure the right expression is blamed. If there are multiple incompatible type values for the target column, then the error is attributed to the first widest (highest precision) incompatible type expression. Testing: - Added tests to AnalyzeStmtsTest.java Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 7 files changed, 86 insertions(+), 23 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13050/11 -- To view, visit http://gerrit.cloudera.org:8080/13050 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I88718fc2cbe1a492165435a542fd2d91d8693a39 Gerrit-Change-Number: 13050 Gerrit-PatchSet: 11 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13329 ) Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4250/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 14 May 2019 15:29:12 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13329 ) Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3216/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 14 May 2019 14:57:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13251 ) Change subject: IMPALA-8369 (part 4): Hive 3: fixes for functional dataset loading .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/13251/4/testdata/datasets/functional/functional_schema_template.sql File testdata/datasets/functional/functional_schema_template.sql: http://gerrit.cloudera.org:8080/#/c/13251/4/testdata/datasets/functional/functional_schema_template.sql@2578 PS4, Line 2578: CREATE_HIVE > Yeah, that seems fine. We literally can't use hive_index for anything, so i I looked at the build failure and the problem is that generate-schema-statements.py assumes that all the entries in this file are tables and it wants to generate INVALIDATE/REFRESH statements for each. There are several ways to workaround this. What do you think about adding a new section called 'CREATE_HIVE2_INDEX' and use it in the entry that creates the 'alltypes' table? And the contents of this section would be ignored in case of Hive3? -- To view, visit http://gerrit.cloudera.org:8080/13251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34930dc064da3136dde4e01a011d14db6a74ecd Gerrit-Change-Number: 13251 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 14 May 2019 14:22:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13329 to look at the new patch set (#2). Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader Moved some responsibilities from parquet-column-readers.cc to a new class 'ParquetPageReader': - reading pages from ScanRange - decompress data if needed The main motivation is to make the implementation of V2 data page reading simpler by moving most parts that will differ between V1 and V2 into a class with manageable complexity. TODOs: - The current implementation tries to change the existing logic as little as possible. The interface of ParquetPageReader could be simplified by changing the logic a bit e.g. reorder handling of errors. - The comments could be extended / polished once the interface is more or less final. Testing: - ran parquet related scanner tests Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 --- M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-column-readers.h A be/src/exec/parquet/parquet-page-reader.cc A be/src/exec/parquet/parquet-page-reader.h 6 files changed, 674 insertions(+), 393 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/13329/2 -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13329 ) Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3215/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 14 May 2019 13:49:26 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13329 Change subject: WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader .. WIP: IMPALA-6433: Part 1: Extract page reading logic from ParquetColumnReader Moved some responsibilities from parquet-column-readers.cc to a new class 'ParquetPageReader': - reading pages from ScanRange - decompress data if needed The main motivation is to make the implementation of V2 data page reading simpler by moving most parts that will differ between V1 and V2 into a class with manageable complexity. TODOs: - The current implementation tries to change the existing logic as little as possible. The interface of ParquetPageReader could be simplified by changing the logic a bit e.g. reorder handling of errors. - The comments could be extended / polished once the interface is more or less final. Testing: - ran parquet related scanner tests Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 --- M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-column-readers.h A be/src/exec/parquet/parquet-page-reader.cc A be/src/exec/parquet/parquet-page-reader.h 6 files changed, 674 insertions(+), 393 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/13329/1 -- To view, visit http://gerrit.cloudera.org:8080/13329 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic0289394adcb97a3529313030930c9c5b85aaa12 Gerrit-Change-Number: 13329 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer
