[Impala-ASF-CR] IMPALA-8960: Fix test owner privileges::test drop if exists on S3
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14334 ) Change subject: IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3 .. IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3 hdfs:// URIs do not work on Impala's S3 tests. The fix is to drop the hdfs:// part of the URI. Our tests run against the default filesystem, so dropping the hdfs:// works on both HDFS test setups and S3 test setups. Testing: - Ran tests on S3 - Ran test_owner_privileges.py on HDFS locally Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332 Reviewed-on: http://gerrit.cloudera.org:8080/14334 Reviewed-by: Michael Ho Tested-by: Impala Public Jenkins --- M tests/authorization/test_owner_privileges.py M tests/common/sentry_cache_test_suite.py 2 files changed, 6 insertions(+), 5 deletions(-) Approvals: Michael Ho: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332 Gerrit-Change-Number: 14334 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14311 ) Change subject: IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp .. Patch Set 6: Code-Review+2 This looks good to me. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/14311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 Gerrit-Change-Number: 14311 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 03 Oct 2019 04:37:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4710/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 15 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 04:34:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8960: Fix test owner privileges::test drop if exists on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14334 ) Change subject: IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3 .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332 Gerrit-Change-Number: 14334 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Thu, 03 Oct 2019 04:03:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14059 to look at the new patch set (#15). Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. IMPALA-8863: Add support to run tests over HTTP/HS2 This change adds support to run backend tests over HTTP using a new version of Impyla. It also adds a test that exercises authentication over HTTP. Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 --- M bin/start-impala-cluster.py M fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java A fe/src/test/java/org/apache/impala/customcluster/LdapImpylaHttpTest.java A fe/src/test/java/org/apache/impala/customcluster/RunShellCommand.java M infra/python/deps/compiled-requirements.txt M tests/common/impala_connection.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/custom_cluster/test_client_ssl.py M tests/custom_cluster/test_hs2.py M tests/custom_cluster/test_session_expiration.py M tests/custom_cluster/test_shell_interactive.py M tests/custom_cluster/test_shell_interactive_reconnect.py M tests/query_test/test_cast_with_format.py M tests/query_test/test_chars.py M tests/query_test/test_date_queries.py M tests/query_test/test_decimal_queries.py M tests/query_test/test_queries.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py A tests/util/run_impyla_http_query.py 21 files changed, 370 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/14059/15 -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 15 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8863: Add support to run tests over HTTP/HS2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14059 ) Change subject: IMPALA-8863: Add support to run tests over HTTP/HS2 .. Patch Set 15: (2 comments) http://gerrit.cloudera.org:8080/#/c/14059/15/tests/common/test_dimensions.py File tests/common/test_dimensions.py: http://gerrit.cloudera.org:8080/#/c/14059/15/tests/common/test_dimensions.py@116 PS15, Line 116: def create_client_protocol_dimension(): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/14059/15/tests/common/test_dimensions.py@117 PS15, Line 117: t flake8: E501 line too long (99 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/14059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7156558071781378fcb9c8941c0f4dd82eb0d018 Gerrit-Change-Number: 14059 Gerrit-PatchSet: 15 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 03:54:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14304 ) Change subject: IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue .. IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue Automated tests were failing while creating Kudu tables with openssl1.1.1 installed on the machine This issue was previously reported under KUDU-2871 and resolved by disabling TLS 1.3 in Kudu source. This change bumps the Kudu version in impala-config.sh to pull in the fix for KUDU-2871. Testing: Create kudu table via automated tests passes Change-Id: Ic34520b4b260e9fe610bc6a5a0e8acf8066dfa69 Reviewed-on: http://gerrit.cloudera.org:8080/14304 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 4 insertions(+), 4 deletions(-) Approvals: Lars Volker: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic34520b4b260e9fe610bc6a5a0e8acf8066dfa69 Gerrit-Change-Number: 14304 Gerrit-PatchSet: 10 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14304 ) Change subject: IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34520b4b260e9fe610bc6a5a0e8acf8066dfa69 Gerrit-Change-Number: 14304 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 03:41:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4709/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 01:26:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 ) Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4708/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 Gerrit-Change-Number: 14332 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 00:59:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14344 ) Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/14344/7/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/14344/7/fe/src/main/java/org/apache/impala/planner/Planner.java@130 PS7, Line 130: // specified. Explain the singleNodeExec() bit http://gerrit.cloudera.org:8080/#/c/14344/7/fe/src/main/java/org/apache/impala/planner/Planner.java@132 PS7, Line 132: ctx_.getQueryOptions().isSetMt_dop( This check is not needed since it's always set at this point in planning. http://gerrit.cloudera.org:8080/#/c/14344/7/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/14344/7/fe/src/main/java/org/apache/impala/service/Frontend.java@1169 PS7, Line 1169: Don't need to add blank line. -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 00:40:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8997: auto fallback to mt dop=0
Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14344 to look at the new patch set (#8). Change subject: IMPALA-8997: auto fallback to mt_dop=0 .. IMPALA-8997: auto fallback to mt_dop=0 Add a temporary --mt_dop_auto_fallback to allow a graceful transition to using mt_dop for workloads. When this flag is set, DML queries and joins that would otherwise fail with an error when run with mt_dop > 0 fall back to running with mt_dop = 0. This means that a user can set mt_dop for their queries and it will only take effect when supported. The behaviour generally does not change when this flag is not set, with a couple of exceptions: * I made mt_dop automatic for compute stats on all file formats * mt_dop is allowed for single node plans with inserts. The quirky validatePlan() logic previously disallowed this but allowed joins in single node plans. The checks added by this patch can be removed safely once mt_dop is supported by default for all queries. This includes some cleanup: * isDmlStmt() was stale and incorrectly implemented. * Various TreeNode methods did not return instances of subclasses of the requested class, which was strange. This fix is required to make 'contains(JoinNode.class)' work correctly. I checked the callsites of the fixed functions and none of them would be affected by this change because they specified a terminal class without any subclasses. I didn't actually use this fix in the end (I had to write a custom tree traversal in hasUnsupportedMtDopJoin()), but figured I would leave the improvement in here. Testing: Add some basic functional tests ensuring that the fallback takes effect. Run basic join and insert tests with this flag enabled. Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/common/TreeNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test A testdata/workloads/functional-query/queries/QueryTest/mt-dop-auto-fallback.test M tests/custom_cluster/test_mt_dop.py 15 files changed, 162 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/14344/8 -- To view, visit http://gerrit.cloudera.org:8080/14344 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie0d73d8744059874293697c8e104891a10dba04d Gerrit-Change-Number: 14344 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue
Lars Volker has uploaded a new patch set (#9) to the change originally created by Kurt Deschler. ( http://gerrit.cloudera.org:8080/14304 ) Change subject: IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue .. IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue Automated tests were failing while creating Kudu tables with openssl1.1.1 installed on the machine This issue was previously reported under KUDU-2871 and resolved by disabling TLS 1.3 in Kudu source. This change bumps the Kudu version in impala-config.sh to pull in the fix for KUDU-2871. Testing: Create kudu table via automated tests passes Change-Id: Ic34520b4b260e9fe610bc6a5a0e8acf8066dfa69 --- M bin/impala-config.sh 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/14304/9 -- To view, visit http://gerrit.cloudera.org:8080/14304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic34520b4b260e9fe610bc6a5a0e8acf8066dfa69 Gerrit-Change-Number: 14304 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8587: show grant does not produce correct privileges
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14356 ) Change subject: IMPALA-8587: show grant does not produce correct privileges .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4707/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8511656fe386a37a66d20e07ce1b875190bc4b65 Gerrit-Change-Number: 14356 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 00:28:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8587: show grant does not produce correct privileges
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14356 ) Change subject: IMPALA-8587: show grant does not produce correct privileges .. Patch Set 1: > Patch Set 1: > > > Patch Set 1: > > > > Can you please add/update tests demonstrating the new functionality? I'm > > not sure this solves the issue outlined in IMPALA-8587. > > Thanks Austin! I will update test_ranger.py accordingly. -- To view, visit http://gerrit.cloudera.org:8080/14356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8511656fe386a37a66d20e07ce1b875190bc4b65 Gerrit-Change-Number: 14356 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 00:19:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Hello Thomas Tauber-Marshall, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14332 to look at the new patch set (#4). Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available .. IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available IMPALA-7312 added the query option FETCH_ROWS_TIMEOUT_MS, but it only applies to fetch requests against a query that has already transitioned to the 'FINISHED' state. This patch changes the timeout so that it applies to queries in the 'RUNNING' state as well. Before this patch, fetch requests issued while a query was 'RUNNING' blocked until the query transitioned to the 'FINISHED' state, and then it fetched results and returned them. After this patch, fetch requests against queries in the 'RUNNING' state will block for 'FETCH_ROWS_TIMEOUT_MS' and then return. For HS2 clients, fetch requests that return while a query is 'RUNNING' set their TStatusCode to STILL_EXECUTING_STATUS. For Beeswax clients, fetch requests that return while a query is 'RUNNING' set the 'ready' flag to false. For both clients, hasMoreRows is set to true. If the following sequence of events occurs: * A fetch request is issued and blocks on a 'RUNNING' query * The query transitions to the 'FINISHED' state * The fetch request attempts to read multiple batches Then the time spent waiting for the query to finish is deducted from the timeout used when waiting for rows to be produced by the Coordinator fragment. Fixed a bug in the current usage of FETCH_ROWS_TIMEOUT_MS where the time units for FETCH_ROWS_TIMEOUT_MS and MonotonicStopWatch were not being converted properly. Tests: * Moved existing fetch timeout tests from hs2/test_fetch.py into a new test file hs2/test_fetch_timeout.py. * Added several new tests to hs2/test_fetch_timeout.py to validate that the timeout is applied to 'RUNNING' queries and that the timeout applies across a 'RUNNING' and 'FINISHED' query. * Added new tests to query_test/test_fetch.py to validate the timeout while using the Beeswax protocol. Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 --- M be/src/common/status.cc M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/plan-root-sink.cc M be/src/exec/plan-root-sink.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/util/error-util.cc M common/thrift/ImpalaService.thrift M tests/hs2/hs2_test_suite.py M tests/hs2/test_fetch.py A tests/hs2/test_fetch_timeout.py M tests/query_test/test_fetch.py 19 files changed, 560 insertions(+), 188 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14332/4 -- To view, visit http://gerrit.cloudera.org:8080/14332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 Gerrit-Change-Number: 14332 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 ) Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available .. Patch Set 3: (2 comments) Addressed comments and made a few cosmetic changes. http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/runtime/coordinator.cc@687 PS3, Line 687: parent_request_state_->fetch_rows_timeout_us() : - parent_request_state_->block_on_wait_time_us > If by coincidence block_on_wait_time_us_ is the same as fetch_rows_timeout_ whoops, yeah that should be a max(1, ...) instead of max(0, ...). There was also a bug where timeout_us might not be set to 0, even if FETCH_ROWS_TIMEOUT_MS was 0. Fixed both issues. http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/service/client-request-state.h@353 PS3, Line 353: uint64_t block_on_wait_time_us_ = 0; > I think storing this per-query is incorrect since there could be multiple c Done -- To view, visit http://gerrit.cloudera.org:8080/14332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 Gerrit-Change-Number: 14332 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 00:19:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8587: show grant does not produce correct privileges
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/14356 ) Change subject: IMPALA-8587: show grant does not produce correct privileges .. Patch Set 1: > Patch Set 1: > > Can you please add/update tests demonstrating the new functionality? I'm not > sure this solves the issue outlined in IMPALA-8587. Thanks Austin! I will update test_ranger.py accordingly. -- To view, visit http://gerrit.cloudera.org:8080/14356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8511656fe386a37a66d20e07ce1b875190bc4b65 Gerrit-Change-Number: 14356 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 00:18:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8587: show grant does not produce correct privileges
Fang-Yu Rao has uploaded this change for review. (
http://gerrit.cloudera.org:8080/14356
Change subject: IMPALA-8587: show grant does not produce correct privileges
..
IMPALA-8587: show grant does not produce correct privileges
Currently the show grant command cannot produce correct privileges when the
list of resources in a RangerPolicy contains '*', a wildcard denoting all the
possible resources.
For example, when a user 'admin' having 'all' as one of its allowed operations
in a Ranger policy, if the field of 'database' in this RangerPolicy contains
only '*', given a SQL statement 'show grant user admin on database functional',
the current logic in RangerImpaladAuthorizationManager.java would consider that
the user 'admin' does not have any privilege on the database 'functional',
whereas in fact the user 'admin' should have all privileges for each database
on this server.
This patch resolves the issue by taking into consideration the case in which
the field of resources could contain a wildcard when determining whether or not
the name of the resource of interest ('functional' in the example above) is
contained in the list of resources the user ('admin' in the example above) has
privileges on.
Testing:
Manually tested the patch using the show grant command on resources of different
scopes, i.e., server, database, table, and column.
Change-Id: I8511656fe386a37a66d20e07ce1b875190bc4b65
---
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
1 file changed, 6 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/14356/1
--
To view, visit http://gerrit.cloudera.org:8080/14356
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8511656fe386a37a66d20e07ce1b875190bc4b65
Gerrit-Change-Number: 14356
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14304 ) Change subject: IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4706/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34520b4b260e9fe610bc6a5a0e8acf8066dfa69 Gerrit-Change-Number: 14304 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 03 Oct 2019 00:11:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8587: show grant does not produce correct privileges
Austin Nobis has posted comments on this change. ( http://gerrit.cloudera.org:8080/14356 ) Change subject: IMPALA-8587: show grant does not produce correct privileges .. Patch Set 1: Can you please add/update tests demonstrating the new functionality? I'm not sure this solves the issue outlined in IMPALA-8587. -- To view, visit http://gerrit.cloudera.org:8080/14356 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8511656fe386a37a66d20e07ce1b875190bc4b65 Gerrit-Change-Number: 14356 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 03 Oct 2019 00:14:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py@1337 PS1, Line 1337: sleep(30) # Wait for the 'lineitem' scan to complete > Not sure if there is any misunderstanding here. I understand the purpose of Yeah, maybe discussing in person is best, will find you tomorrow. I get that waiting for for all F01 instances to transition to "FINISHED" is a prerequisite for NumCompletedBackends == 1, I guess my question is what is the benefit of doing that check? How would it reduce flakiness? You would end up with code like: assert_eventually(1, 30, lambda: [method-to-check-F01-fragments-finished]) assert_eventually(1, 30, lamdba: "NumCompletedBackends: 1 (1)" in self.client.get_runtime_profile(handle))) I don't see how that is better than just doing: assert_eventually(1, 30, lamdba: "NumCompletedBackends: 1 (1)" in self.client.get_runtime_profile(handle))) Given that NumCompletedBackends should be set to 1 at roughly the same time all F01 fragments finish. -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 02 Oct 2019 23:53:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9001: Fix SPNEGO for requests with no 'Authorization'
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14352 ) Change subject: IMPALA-9001: Fix SPNEGO for requests with no 'Authorization' .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4705/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id9b6ac99b799324ec22e95fd1eb022d5ad6f54bd Gerrit-Change-Number: 14352 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 02 Oct 2019 23:59:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8960: Fix test owner privileges::test drop if exists on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14334 ) Change subject: IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5044/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332 Gerrit-Change-Number: 14334 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 02 Oct 2019 23:44:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8960: Fix test owner privileges::test drop if exists on S3
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14334 ) Change subject: IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332 Gerrit-Change-Number: 14334 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 02 Oct 2019 23:38:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py@1337 PS1, Line 1337: sleep(30) # Wait for the 'lineitem' scan to complete > The test itself is timing dependent, so I don't think we can get out of usi Not sure if there is any misunderstanding here. I understand the purpose of the test is to validate that NumCompletedBackends == 1 and I am not suggesting you to remove that check. Instead, my suggestion is that the test should wait on a logical condition in which NumCompletedBackends is expected to be 1 before checking NumCompletedBackends == 1 in the query profile. My understanding is that this condition should happen once all instances of F01 reach the "FINISHED" state. Anyhow, this is just a minor point but I think timing dependent tests are usually prone to flakiness so avoiding that if possible would be great. Please feel free to grab me and discuss this offline if it's still not clear. -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 02 Oct 2019 23:38:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14304 ) Change subject: IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5043/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34520b4b260e9fe610bc6a5a0e8acf8066dfa69 Gerrit-Change-Number: 14304 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Oct 2019 23:30:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/14304 ) Change subject: IMPALA-8973: Update Kudu version to fix openssl1.1.1 compatibility issue .. Patch Set 9: Code-Review+2 PS 8 rebases, PS 9 puts the right Parquet version back. Carrying Tim's +2 -- To view, visit http://gerrit.cloudera.org:8080/14304 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic34520b4b260e9fe610bc6a5a0e8acf8066dfa69 Gerrit-Change-Number: 14304 Gerrit-PatchSet: 9 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Oct 2019 23:30:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9001: Fix SPNEGO for requests with no 'Authorization'
Thomas Tauber-Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14352 Change subject: IMPALA-9001: Fix SPNEGO for requests with no 'Authorization' .. IMPALA-9001: Fix SPNEGO for requests with no 'Authorization' When SPNEGO was first implemented for both hs2 and the webui, the way we handled requests that did not include an "Authorization" header was to pass an empty string to gss-api and then return a "WWW-Authenticate: Negotiate " where was whatever was returned by gss-api. This works with some clients, but appears to fail with others. This patch modifies the behavior to not send the with the initial WWW-Authenticate, which works with all tested clients. Testing: - Tested with curl, Knox, and Java's HttpURLConnection API. Change-Id: Id9b6ac99b799324ec22e95fd1eb022d5ad6f54bd --- M be/src/rpc/authentication.cc M be/src/util/webserver.cc 2 files changed, 10 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/14352/1 -- To view, visit http://gerrit.cloudera.org:8080/14352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id9b6ac99b799324ec22e95fd1eb022d5ad6f54bd Gerrit-Change-Number: 14352 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-8960: Fix test owner privileges::test drop if exists on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14334 ) Change subject: IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4704/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332 Gerrit-Change-Number: 14334 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 02 Oct 2019 22:57:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8899: Add cookie support to the webui
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14339 ) Change subject: IMPALA-8899: Add cookie support to the webui .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4703/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I30788e0539627ee6154ad8183b124947c5da8ef4 Gerrit-Change-Number: 14339 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Oct 2019 22:53:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8960: Fix test owner privileges::test drop if exists on S3
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14334 to look at the new patch set (#2). Change subject: IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3 .. IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on S3 hdfs:// URIs do not work on Impala's S3 tests. The fix is to drop the hdfs:// part of the URI. Our tests run against the default filesystem, so dropping the hdfs:// works on both HDFS test setups and S3 test setups. Testing: - Ran tests on S3 - Ran test_owner_privileges.py on HDFS locally Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332 --- M tests/authorization/test_owner_privileges.py M tests/common/sentry_cache_test_suite.py 2 files changed, 6 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14334/2 -- To view, visit http://gerrit.cloudera.org:8080/14334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332 Gerrit-Change-Number: 14334 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8960: Fix test owner privileges::test drop if exists on S3
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/14334 )
Change subject: IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on
S3
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14334/1/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:
http://gerrit.cloudera.org:8080/#/c/14334/1/tests/authorization/test_owner_privileges.py@172
PS1, Line 172: self.execute_query("grant all on uri
'/test-warehouse/libTestUdfs.so' to"
> I could be missing some details here but why not use FILESYSTEM_PREFIX inst
Good point, switched to use FILESYSTEM_PREFIX. (My understanding is that
FILESYSTEM_PREFIX only impacts local filesystem these days, but it is still
good to use it.)
--
To view, visit http://gerrit.cloudera.org:8080/14334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332
Gerrit-Change-Number: 14334
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Wed, 02 Oct 2019 22:17:03 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8899: Add cookie support to the webui
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14339 to look at the new patch set (#4). Change subject: IMPALA-8899: Add cookie support to the webui .. IMPALA-8899: Add cookie support to the webui This patches takes the machinery for generating and checking cookies for authentication that was added in IMPALA-8584 and applies to the webui. It also fixes an issue where some clients may return the cookie value surrounded by "". It adds metrics for both SPNEGO auth success/failure and cookie auth success/failure to the webserver. This patch also fixes IMPALA-8898 by returning an empty cookie with a Max-Age of 0 on requests where an invalid cookie was provided to indicate to the client that the cookie should be deleted. Testing: - Added a test that uses curl to access the webserver with SPNEGO enabled while storing and using cookies. This test only runs when curl is present, which is generally not the case in our automated testing runs. Change-Id: I30788e0539627ee6154ad8183b124947c5da8ef4 --- M be/src/catalog/catalogd-main.cc M be/src/rpc/authentication.cc M be/src/rpc/cookie-util.cc M be/src/rpc/cookie-util.h M be/src/runtime/exec-env.cc M be/src/statestore/statestored-main.cc M be/src/util/webserver-test.cc M be/src/util/webserver.cc M be/src/util/webserver.h M common/thrift/metrics.json M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java 11 files changed, 284 insertions(+), 144 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/14339/4 -- To view, visit http://gerrit.cloudera.org:8080/14339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I30788e0539627ee6154ad8183b124947c5da8ef4 Gerrit-Change-Number: 14339 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14311 ) Change subject: IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4702/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 Gerrit-Change-Number: 14311 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 02 Oct 2019 21:32:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14311 ) Change subject: IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4701/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 Gerrit-Change-Number: 14311 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 02 Oct 2019 21:30:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py@1337 PS1, Line 1337: sleep(30) # Wait for the 'lineitem' scan to complete > assert_eventually still has a timeout itself so it will suffer the same tim The test itself is timing dependent, so I don't think we can get out of using timeouts. The test runs a query where some fragments block indefinitely, and others Even checking the status of the in-flight fragments will have to be done with a timeout. The reason I want to use NumCompletedBackends, is that this feature should increment NumCompletedBackends whenever a backend completes. If it doesn't, then that is a bug. NumCompletedBackends was added as part of this feature. Using assert_eventually should be more robust than the previous approach of just sleeping for a fixed amount of time and then asserting that NumCompletedBackends == 1. assert_eventually polls the runtime profile at a fixed interval and validates that the value of NumCompletedBackends eventually reaches 1, within a timeout of 30 seconds. The previous approach just wanted for 10 seconds, and then asserted that NumCompletedBackends == 1. We could even increase the timeout to a higher value, like 60 if necessary. -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 02 Oct 2019 21:07:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp
Hello Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14311 to look at the new patch set (#6). Change subject: IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp .. IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp Add the -d option and -f option to the following commands: `hdfs dfs -copyFromLocal URI` `hdfs dfs -put [ - | .. ]. ` `hdfs dfs -cp URI [URI ...] ` The -d option "Skip[s] creation of temporary file with the suffix ._COPYING_." which improves performance of these commands on S3 since S3 does not support metadata only renames. The -f option "Overwrites the destination if it already exists" combined with HADOOP-13884 this improves issues seen with S3 consistency issues by avoiding a HEAD request to check if the destination file exists or not. Added the method 'copy_from_local' to the BaseFilesystem class. Re-factored most usages of the aforementioned HDFS commands to use the filesystem_client. Some usages were not appropriate / worth refactoring, so occasionally this patch just adds the '-d' and '-f' options explicitly. All calls to '-put' were replaced with 'copyFromLocal' because they both copy files from the local fs to a HDFS compatible target fs. Since WebHDFS does not have good support for copying files, this patch removes the copy functionality from the PyWebHdfsClientWithChmod. Re-factored the hdfs_client so that it uses a DelegatingHdfsClient that delegates to either the HadoopFsCommandLineClient or PyWebHdfsClientWithChmod. Testing: * Ran core tests on HDFS and S3 Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 --- M testdata/workloads/functional-query/queries/QueryTest/parquet-resolution-by-name.test M tests/common/impala_test_suite.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_udf_concurrency.py M tests/metadata/test_hidden_files.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_stale_metadata.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_file_mods.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_multiple_filesystems.py M tests/query_test/test_nested_types.py M tests/query_test/test_scanners.py M tests/query_test/test_scanners_fuzz.py M tests/query_test/test_udfs.py M tests/util/adls_util.py M tests/util/filesystem_base.py M tests/util/hdfs_util.py 21 files changed, 202 insertions(+), 109 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/14311/6 -- To view, visit http://gerrit.cloudera.org:8080/14311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 Gerrit-Change-Number: 14311 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14311 ) Change subject: IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/14311/5/tests/custom_cluster/test_insert_behaviour.py File tests/custom_cluster/test_insert_behaviour.py: http://gerrit.cloudera.org:8080/#/c/14311/5/tests/custom_cluster/test_insert_behaviour.py@24 PS5, Line 24: C > flake8: E126 continuation line over-indented for hanging indent Done -- To view, visit http://gerrit.cloudera.org:8080/14311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 Gerrit-Change-Number: 14311 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 02 Oct 2019 20:52:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14311 ) Change subject: IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp .. Patch Set 4: (2 comments) Addressed comments + fixed a bunch of issues that popped up when I ran exhaustive tests. http://gerrit.cloudera.org:8080/#/c/14311/4/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/14311/4/tests/common/impala_test_suite.py@179 PS4, Line 179: # There are multiple clients for interacting with the underlying storage service. : # : # There are two main types of clients: HTTP clients and CLI clients. CLI clients all : # use the 'hdfs dfs' CLI to execute operations againt a target filesystem. HTTP : # clients issue HTTP requests to execute operations and are filesystem specific. For : # HDFS, the HTTP client uses WebHDFS. : # : # 'hdfs_client' is a wrapper around a HTTP client and CLI client for interacting : # with HDFS. The 'hdfs_client' delegates to the HTTP client when possible, and for : # operations not supported by the HTTP client, it delegates to the CLI client. The : # 'hdfs_client' is specific to HDFS and always points to the local HDFS cluster. : # : # 'filesystem_client' is set depending on the value of the 'TARGET_FILESYSTEM'. For : # HDFS, it is the same as the 'hdfs_client'. For S3 and and ABFS, the client is a : # HadoopFsCommandLineClient which is a simple wrapper around 'hdfs dfs' commands. : # For ADLS, the 'filesystem_client' is an ADLSClient. > These are mostly nits, but here goes: Done. Attempted to incorporate everything you wrote above + some of the existing text. http://gerrit.cloudera.org:8080/#/c/14311/4/tests/util/hdfs_util.py File tests/util/hdfs_util.py: http://gerrit.cloudera.org:8080/#/c/14311/4/tests/util/hdfs_util.py@58 PS4, Line 58: DelegatingHdfsFilesystem > Nit: The other clients all have "Client" in their name rather than Filesyst Done -- To view, visit http://gerrit.cloudera.org:8080/14311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 Gerrit-Change-Number: 14311 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 02 Oct 2019 20:51:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14311 ) Change subject: IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/14311/5/tests/custom_cluster/test_insert_behaviour.py File tests/custom_cluster/test_insert_behaviour.py: http://gerrit.cloudera.org:8080/#/c/14311/5/tests/custom_cluster/test_insert_behaviour.py@24 PS5, Line 24: H flake8: E126 continuation line over-indented for hanging indent -- To view, visit http://gerrit.cloudera.org:8080/14311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 Gerrit-Change-Number: 14311 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 02 Oct 2019 20:51:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp
Hello Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14311 to look at the new patch set (#5). Change subject: IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp .. IMPALA-8950: Add -d, -f options to hdfs copyFromLocal, put, cp Add the -d option and -f option to the following commands: `hdfs dfs -copyFromLocal URI` `hdfs dfs -put [ - | .. ]. ` `hdfs dfs -cp URI [URI ...] ` The -d option "Skip[s] creation of temporary file with the suffix ._COPYING_." which improves performance of these commands on S3 since S3 does not support metadata only renames. The -f option "Overwrites the destination if it already exists" combined with HADOOP-13884 this improves issues seen with S3 consistency issues by avoiding a HEAD request to check if the destination file exists or not. Added the method 'copy_from_local' to the BaseFilesystem class. Re-factored most usages of the aforementioned HDFS commands to use the filesystem_client. Some usages were not appropriate / worth refactoring, so occasionally this patch just adds the '-d' and '-f' options explicitly. All calls to '-put' were replaced with 'copyFromLocal' because they both copy files from the local fs to a HDFS compatible target fs. Since WebHDFS does not have good support for copying files, this patch removes the copy functionality from the PyWebHdfsClientWithChmod. Re-factored the hdfs_client so that it uses a DelegatingHdfsClient that delegates to either the HadoopFsCommandLineClient or PyWebHdfsClientWithChmod. Testing: * Ran core tests on HDFS and S3 Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 --- M testdata/workloads/functional-query/queries/QueryTest/parquet-resolution-by-name.test M tests/common/impala_test_suite.py M tests/custom_cluster/test_coordinators.py M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py M tests/custom_cluster/test_insert_behaviour.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_udf_concurrency.py M tests/metadata/test_hidden_files.py M tests/metadata/test_refresh_partition.py M tests/metadata/test_stale_metadata.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_hdfs_file_mods.py M tests/query_test/test_insert_parquet.py M tests/query_test/test_multiple_filesystems.py M tests/query_test/test_nested_types.py M tests/query_test/test_scanners.py M tests/query_test/test_scanners_fuzz.py M tests/query_test/test_udfs.py M tests/util/adls_util.py M tests/util/filesystem_base.py M tests/util/hdfs_util.py 21 files changed, 202 insertions(+), 109 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/14311/5 -- To view, visit http://gerrit.cloudera.org:8080/14311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0d45db1c00554e6fb6bcc0b552596d86d4e30144 Gerrit-Change-Number: 14311 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 ) Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available .. Patch Set 3: (2 comments) Nice tests! http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/runtime/coordinator.cc@687 PS3, Line 687: parent_request_state_->fetch_rows_timeout_us() : - parent_request_state_->block_on_wait_time_us If by coincidence block_on_wait_time_us_ is the same as fetch_rows_timeout_us, can't this work out to 0 and incorrectly block indefinitely? http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/service/client-request-state.h@353 PS3, Line 353: uint64_t block_on_wait_time_us_ = 0; I think storing this per-query is incorrect since there could be multiple concurrent calls to BlockOnWait() and one thread might pick up the other thread's value. I think that might be an unimportant edge case and we could probably live with that but I think it still makes the logic unnecessarily confusing because this value really fits with the RPC request, not the query. I could see either threading it through the function calls (probably the most obvious and explicit way) or using a thread-local variable. -- To view, visit http://gerrit.cloudera.org:8080/14332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4 Gerrit-Change-Number: 14332 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Oct 2019 20:13:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8742: Switch to ScanRange::bytes to read() instead of len()
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14348 ) Change subject: IMPALA-8742: Switch to ScanRange::bytes_to_read() instead of len() .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie896db3f4b5f3e2272d81c2d360049af09c41d9c Gerrit-Change-Number: 14348 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Oct 2019 19:58:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8960: Fix test owner privileges::test drop if exists on S3
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/14334 )
Change subject: IMPALA-8960: Fix test_owner_privileges::test_drop_if_exists on
S3
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14334/1/tests/authorization/test_owner_privileges.py
File tests/authorization/test_owner_privileges.py:
http://gerrit.cloudera.org:8080/#/c/14334/1/tests/authorization/test_owner_privileges.py@172
PS1, Line 172: self.execute_query("grant all on uri
'/test-warehouse/libTestUdfs.so' to"
I could be missing some details here but why not use FILESYSTEM_PREFIX instead ?
--
To view, visit http://gerrit.cloudera.org:8080/14334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibfe15ac2d5ba0d8a6d4383be8d01395c74d67332
Gerrit-Change-Number: 14334
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Wed, 02 Oct 2019 19:43:18 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14340 ) Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/502/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 02 Oct 2019 18:54:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14340 ) Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/502/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 02 Oct 2019 18:34:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Hello Attila Jeges, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14340 to look at the new patch set (#2). Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. IMPALA-8988: [DOCS] DATE type is supported AVRO tables Change-Id: I95f37accddadcba436676498d5cbb34cda281846 --- M docs/topics/impala_avro.xml M docs/topics/impala_date.xml 2 files changed, 102 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/14340/2 -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14340 ) Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14340/1/docs/topics/impala_date.xml File docs/topics/impala_date.xml: http://gerrit.cloudera.org:8080/#/c/14340/1/docs/topics/impala_date.xml@53 PS1, Line 53: -01-01 > Note that we are in the midst of changing the valid range for DATE values t I will update this now, assuming that patch will be approved. http://gerrit.cloudera.org:8080/#/c/14340/1/docs/topics/impala_date.xml@69 PS1, Line 69: Parquet uses DATE logical type for dates. The : DATE logical type annotates an INT32 : that stores the number of days from the Unix epoch, January 1, 1970. This : representation introduces a Parquet interoperability issue between Impala : and older versions of Hive: : : If Hive versions lower than 3.1 wrote dates earlier than 1582-10-15 to a : Parquet table, those dates would be read back incorrectly by Impala and : vice versa. In Hive 3.1 and higher, this is no longer an issue. > Avro has the same issue since avro too stores DATE logical type as a number Done -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 02 Oct 2019 18:33:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14337 ) Change subject: IMPALA-8926, IMPALA-8989: Fix flaky result spooling tests .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py File tests/custom_cluster/test_admission_controller.py: http://gerrit.cloudera.org:8080/#/c/14337/1/tests/custom_cluster/test_admission_controller.py@1337 PS1, Line 1337: sleep(30) # Wait for the 'lineitem' scan to complete > The goal of the test is to actually validate that NumCompletedBackends is i assert_eventually still has a timeout itself so it will suffer the same timing issue in the worst case. Given that this test checks that NumCompletedBackends is incremented by one, isn't it more precise to check that all F01 reached "FINISHED" states by parsing the json output from the debug webpage ? http://:25000/query_finstances?query_id= A proxy for the above is to check the number of in-flight fragments on all nodes. This of course bakes a lot of assumption into the test about the generated plan for the given query. -- To view, visit http://gerrit.cloudera.org:8080/14337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ea6bf3d84f174745c8a0b1e0f2b55ce05ee618b Gerrit-Change-Number: 14337 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 02 Oct 2019 17:45:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8988: [DOCS] DATE type is supported AVRO tables
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/14340 ) Change subject: IMPALA-8988: [DOCS] DATE type is supported AVRO tables .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14340/1/docs/topics/impala_date.xml File docs/topics/impala_date.xml: http://gerrit.cloudera.org:8080/#/c/14340/1/docs/topics/impala_date.xml@53 PS1, Line 53: -01-01 Note that we are in the midst of changing the valid range for DATE values to 0001-01-01 to -12-31,. We are changing the lower end of the valid range from year 0 to year 1 to comply with the ANSI SQL standard. http://gerrit.cloudera.org:8080/#/c/14340/1/docs/topics/impala_date.xml@69 PS1, Line 69: Parquet uses DATE logical type for dates. The : DATE logical type annotates an INT32 : that stores the number of days from the Unix epoch, January 1, 1970. This : representation introduces a Parquet interoperability issue between Impala : and older versions of Hive: : : If Hive versions lower than 3.1 wrote dates earlier than 1582-10-15 to a : Parquet table, those dates would be read back incorrectly by Impala and : vice versa. In Hive 3.1 and higher, this is no longer an issue. Avro has the same issue since avro too stores DATE logical type as a number of days since epoch. -- To view, visit http://gerrit.cloudera.org:8080/14340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95f37accddadcba436676498d5cbb34cda281846 Gerrit-Change-Number: 14340 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 02 Oct 2019 13:46:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7368: Change supported year range for DATE values to 1..9999
Attila Jeges has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14349 Change subject: IMPALA-7368: Change supported year range for DATE values to 1.. .. IMPALA-7368: Change supported year range for DATE values to 1.. Before this patch the supported year range for DATE type started with year 0. This contradicts the ANSI SQL standard that defines the valid DATE value range to be 0001-01-01 to -12-31. Change-Id: Iefdf1c036834763f52d44d0c39a25a1f04e41e07 --- M be/src/exec/hdfs-avro-scanner-test.cc M be/src/exprs/expr-test.cc M be/src/runtime/date-test.cc M be/src/runtime/date-value.cc M be/src/runtime/date-value.h M be/src/runtime/raw-value-test.cc M be/src/udf/udf.h M be/src/util/string-parser-test.cc M common/thrift/generate_error_codes.py M docs/topics/impala_date.xml M fe/src/main/java/org/apache/impala/catalog/ColumnStats.java M testdata/data/README M testdata/workloads/functional-query/queries/DataErrorsTest/avro-errors.test M testdata/workloads/functional-query/queries/QueryTest/avro_date.test M testdata/workloads/functional-query/queries/QueryTest/date-fileformat-support.test M testdata/workloads/functional-query/queries/QueryTest/date.test M testdata/workloads/functional-query/queries/QueryTest/out-of-range-date.test 17 files changed, 239 insertions(+), 199 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/14349/1 -- To view, visit http://gerrit.cloudera.org:8080/14349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iefdf1c036834763f52d44d0c39a25a1f04e41e07 Gerrit-Change-Number: 14349 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges
[Impala-ASF-CR] IMPALA-8742: Switch to ScanRange::bytes to read() instead of len()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14348 ) Change subject: IMPALA-8742: Switch to ScanRange::bytes_to_read() instead of len() .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4700/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie896db3f4b5f3e2272d81c2d360049af09c41d9c Gerrit-Change-Number: 14348 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Oct 2019 13:27:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8742: Switch to ScanRange::bytes to read() instead of len()
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14348 Change subject: IMPALA-8742: Switch to ScanRange::bytes_to_read() instead of len() .. IMPALA-8742: Switch to ScanRange::bytes_to_read() instead of len() IMPALA-7543 introduced sub-ranges in scan ranges. These are smaller parts of the scan ranges that actually need to be read, other parts of the scan range can be skipped. Currently sub-ranges are only used in the Parquet scanner during page filtering. With sub-ranges the scan range has a new field 'bytes_to_read_', that is the sum of the lengths of the sub-ranges. Or, if there are no sub-ranges, 'bytes_to_read_' equals to field 'len_' which is the length of the whole scan range. At some parts of Impala ScanRange::len() is being used instead of ScanRange::bytes_to_read(). It doesn't cause a bug because only the Parquet scanner uses sub-ranges, i.e. bytes_to_read() usually equals to len(). The Parquet scanner also doesn't hit the bug because it tracks which pages it reads. However, it can be a potential source of bugs in the future to leave the invocations of len() instead of bytes_to_read(). Also, the scanners might allocate more memory than needed. At couple of places we still need to invoke len(), e.g. when we test scan-range containment (for local splits), or when we test whether a scan range contains the mid-point of a Parquet row group. Testing: Added a scanner reservation test. Ran the exhaustive tests. Change-Id: Ie896db3f4b5f3e2272d81c2d360049af09c41d9c --- M be/src/exec/base-sequence-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/scanner-context.h M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/request-context.cc M be/src/runtime/io/scan-range.cc M testdata/workloads/functional-query/queries/QueryTest/scanner-reservation.test 8 files changed, 24 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/14348/1 -- To view, visit http://gerrit.cloudera.org:8080/14348 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie896db3f4b5f3e2272d81c2d360049af09c41d9c Gerrit-Change-Number: 14348 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] Remove hard-coded localhost from impala test suite.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14318 ) Change subject: Remove hard-coded localhost from impala_test_suite.py .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5041/ -- To view, visit http://gerrit.cloudera.org:8080/14318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c52d94d9876661faada42d6570ef5005960e74d Gerrit-Change-Number: 14318 Gerrit-PatchSet: 3 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 02 Oct 2019 06:57:07 + Gerrit-HasComments: No
