[Impala-ASF-CR] IMPALA-10294: Improvement to test skew reporting in runtime profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16662 ) Change subject: IMPALA-10294: Improvement to test_skew_reporting_in_runtime_profile .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7a36551f2507d724891707d26b7394fbe3a5657b Gerrit-Change-Number: 16662 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 30 Oct 2020 05:34:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10294: Improvement to test skew reporting in runtime profile
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16662 ) Change subject: IMPALA-10294: Improvement to test_skew_reporting_in_runtime_profile .. IMPALA-10294: Improvement to test_skew_reporting_in_runtime_profile This fix improved the skew reporting test by lowering the threshold to 0 and by taking care of the extreme case of no skews. Testing: 1. Unit testing. Change-Id: I7a36551f2507d724891707d26b7394fbe3a5657b Reviewed-on: http://gerrit.cloudera.org:8080/16662 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_observability.py 1 file changed, 10 insertions(+), 10 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7a36551f2507d724891707d26b7394fbe3a5657b Gerrit-Change-Number: 16662 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10304: Fix log level and format for pytests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16679 ) Change subject: IMPALA-10304: Fix log level and format for pytests .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7588/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d91b7f910b3606c50bcba4579179a0bc8c20588 Gerrit-Change-Number: 16679 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Oct 2020 05:08:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10304: Fix log level and format for pytests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16679 ) Change subject: IMPALA-10304: Fix log level and format for pytests .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/16679/1/bin/start-impala-cluster.py File bin/start-impala-cluster.py: http://gerrit.cloudera.org:8080/#/c/16679/1/bin/start-impala-cluster.py@719 PS1, Line 719: " flake8: E501 line too long (92 > 90 characters) http://gerrit.cloudera.org:8080/#/c/16679/1/tests/stress/query_retries_stress_runner.py File tests/stress/query_retries_stress_runner.py: http://gerrit.cloudera.org:8080/#/c/16679/1/tests/stress/query_retries_stress_runner.py@295 PS1, Line 295: ) flake8: E501 line too long (91 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/16679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d91b7f910b3606c50bcba4579179a0bc8c20588 Gerrit-Change-Number: 16679 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Oct 2020 04:49:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10304: Fix log level and format for pytests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16679 ) Change subject: IMPALA-10304: Fix log level and format for pytests .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6624/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5d91b7f910b3606c50bcba4579179a0bc8c20588 Gerrit-Change-Number: 16679 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Oct 2020 04:48:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10304: Fix log level and format for pytests
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16679 Change subject: IMPALA-10304: Fix log level and format for pytests .. IMPALA-10304: Fix log level and format for pytests Recent testing showed that the pytests are not respecting the log level and format set in conftest.py's configure_logging(). It is using the default log level of WARNING and the default formatter. The issue is that logging.basicConfig() is only effective the first time it is called. The code in lib/python/impala_py_lib/helpers.py does a call to logging.basicConfig() at the global level, and conftest.py imports that file. This renders the call in configure_logging() ineffective. To avoid this type of confusion, logging.basicConfig() should only be called from the main() functions for libraries. This removes the call in lib/python/impala_py_lib (as it is not needed for a library without a main function). It also fixes up various other locations to move the logging.basicConfig() call to the main() function. Testing: - Ran the end to end tests and custom cluster tests - Confirmed the logging format - Added an assert in configure_logging() to test that the INFO log level is applied to the root logger. Change-Id: I5d91b7f910b3606c50bcba4579179a0bc8c20588 --- M bin/dump_breakpad_symbols.py M bin/run-workload.py M bin/start-impala-cluster.py M lib/python/impala_py_lib/helpers.py M tests/conftest.py M tests/shell/util.py M tests/stress/query_retries_stress_runner.py 7 files changed, 14 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/16679/1 -- To view, visit http://gerrit.cloudera.org:8080/16679 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5d91b7f910b3606c50bcba4579179a0bc8c20588 Gerrit-Change-Number: 16679 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode To compliant with FIPS requirement, we should use OpenSSL libraries for cryptographic hash functions, instead of own hash functions. This patch replace MD5 and SHA1 functions in Squeasel Web server with OpenSSL APIs. It also force to turn off Digest Authorization for Web server in FIPS approved mode since Digest Authorization use MD5 hash and it doesn't comply with FIPS 140-2. Testing: - Passed webserver-test. - Passed exhaustive tests. - Manually verified HTTP Digest Authorization could not be enabled by setting webserver_password_file on a FIPS enabled cluster. Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Reviewed-on: http://gerrit.cloudera.org:8080/16630 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins --- M be/src/thirdparty/squeasel/squeasel.c M be/src/util/webserver-test.cc M be/src/util/webserver.cc 3 files changed, 38 insertions(+), 338 deletions(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 9 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 30 Oct 2020 04:33:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py This changes test_scanners_fuzz.py to set the logging level to INFO. By default, it is WARNING, so it was missing some useful INFO log messages like the random seed used. This also fixes formatting on one of the log lines. Testing: - Ran test_scanners_fuzz.py locally and checked to make sure the INFO messages were present Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Reviewed-on: http://gerrit.cloudera.org:8080/16677 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_scanners_fuzz.py 1 file changed, 6 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 30 Oct 2020 03:25:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10166 (part 1): ALTER TABLE for Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16606 ) Change subject: IMPALA-10166 (part 1): ALTER TABLE for Iceberg tables .. Patch Set 9: (4 comments) Thanks for review! http://gerrit.cloudera.org:8080/#/c/16606/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16606/8//COMMIT_MSG@9 PS8, Line 9: implement > nit: implements Done http://gerrit.cloudera.org:8080/#/c/16606/8/fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java File fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java: http://gerrit.cloudera.org:8080/#/c/16606/8/fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java@25 PS8, Line 25: import org.apache.impala.catalog.FeKuduTable; > Is this import needed? Done http://gerrit.cloudera.org:8080/#/c/16606/8/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/16606/8/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java@118 PS8, Line 118: support > nit: supports Done http://gerrit.cloudera.org:8080/#/c/16606/8/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test File testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test: http://gerrit.cloudera.org:8080/#/c/16606/8/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test@117 PS8, Line 117: ALTER TABLE iceberg_test3 RENAME TO iceberg_test3_new; > Instead of creating a new table here, could you re-use any of the ones abov I can use 'iceberg_test3' to replace 'iceberg_test9', since 'iceberg_test3' is a normal 'hadoop.tables' table. But iceberg_test10 is necessary here, since we don't create a normal 'hadoop.catalog' table above. So I replace 'iceberg_test9' by 'iceberg_test3', and rename 'iceberg_test10' to 'iceberg_test9'. Shall we can reuse these table name, such as 'iceberg_test1' to replace 'iceberg_test10'? -- To view, visit http://gerrit.cloudera.org:8080/16606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5104cc47c7b42dacdb52983f503cd263135d6bfc Gerrit-Change-Number: 16606 Gerrit-PatchSet: 9 Gerrit-Owner: wangsheng Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 30 Oct 2020 03:00:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10166 (part 1): ALTER TABLE for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16606 ) Change subject: IMPALA-10166 (part 1): ALTER TABLE for Iceberg tables .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7587/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5104cc47c7b42dacdb52983f503cd263135d6bfc Gerrit-Change-Number: 16606 Gerrit-PatchSet: 9 Gerrit-Owner: wangsheng Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 30 Oct 2020 02:58:11 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Rewrite ImpalaHttpClient.py to use the requests package
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16678 ) Change subject: [WIP] Rewrite ImpalaHttpClient.py to use the requests package .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6620/ -- To view, visit http://gerrit.cloudera.org:8080/16678 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1030fd7b74b5c1cfaeed8a57fbcbb7a0703dd56 Gerrit-Change-Number: 16678 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Oct 2020 02:56:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10166 (part 1): ALTER TABLE for Iceberg tables
wangsheng has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/16606 ) Change subject: IMPALA-10166 (part 1): ALTER TABLE for Iceberg tables .. IMPALA-10166 (part 1): ALTER TABLE for Iceberg tables This patch mainly implements ALTER TABLE for Iceberg tables, we currently support these statements: * ADD COLUMNS * RENAME TABLE * SET TBL_PROPERTIES * SET OWNER We forbid DROP COLUMN/REPLACE COLUMNS/ALTER COLUMN in this patch, since these statemens may make Iceberg tables unreadable. We may support column resolution by field id in the near future, after that, we will support COLUMN/REPLACE COLUMNS/ALTER COLUMN for Iceberg tables. Here something we still need to pay attention: 1.RENAME TABLE is not supported for HadoopCatalog/HadoopTables, even if we already implement 'RENAME TABLE' statement, so we only rename the table in the Hive Metastore for external table. 2.We cannot ADD/DROP PARTITION now since there is no API for that in Iceberg, but related work is already in progess in Iceberg. Testing: - Iceberg table alter test in test_iceberg.py - Iceberg table negative test in test_scanners.py Change-Id: I5104cc47c7b42dacdb52983f503cd263135d6bfc --- M fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableRecoverPartitionsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetRowFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCatalog.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopCatalog.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergHadoopTables.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java A testdata/workloads/functional-query/queries/QueryTest/iceberg-alter.test M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test M tests/query_test/test_iceberg.py 18 files changed, 486 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/16606/9 -- To view, visit http://gerrit.cloudera.org:8080/16606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5104cc47c7b42dacdb52983f503cd263135d6bfc Gerrit-Change-Number: 16606 Gerrit-PatchSet: 9 Gerrit-Owner: wangsheng Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10303: Fix warnings from impala-shell with --quiet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16673 ) Change subject: IMPALA-10303: Fix warnings from impala-shell with --quiet .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e94c9445ffba159725bacd6f6bc36f7c91b88fe Gerrit-Change-Number: 16673 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 30 Oct 2020 02:17:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10303: Fix warnings from impala-shell with --quiet
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16673 ) Change subject: IMPALA-10303: Fix warnings from impala-shell with --quiet .. IMPALA-10303: Fix warnings from impala-shell with --quiet When the --quiet flag is used with impala-shell, the intention is that if the query is successful then only the query results should be printed. This patch fixes two cases where --quiet was not being respected: - When using the HTTP transport and --client_connect_timeout_ms is set, a warning is printed that the timeout is not applied. - When running in non-interactive mode, a warning is printed that --live_progress is automatically disabled. This warning is now also only printed if --live_progress is actually set. Testing: - Added a test that runs a simple query with --quiet and confirms the output is as expected. Change-Id: I1e94c9445ffba159725bacd6f6bc36f7c91b88fe Reviewed-on: http://gerrit.cloudera.org:8080/16673 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M shell/impala_client.py M shell/impala_shell.py M tests/shell/test_shell_commandline.py 3 files changed, 14 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1e94c9445ffba159725bacd6f6bc36f7c91b88fe Gerrit-Change-Number: 16673 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10298: Change column mask hash as SHA512 in FIPS mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16671 ) Change subject: IMPALA-10298: Change column mask hash as SHA512 in FIPS mode .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I422d4b11b31c3e6eb7963260a1da730579c4ca74 Gerrit-Change-Number: 16671 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 30 Oct 2020 01:16:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10298: Change column mask hash as SHA512 in FIPS mode
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16671 ) Change subject: IMPALA-10298: Change column mask hash as SHA512 in FIPS mode .. IMPALA-10298: Change column mask hash as SHA512 in FIPS mode Column masking API is called by Ranger during policy evaluation. Ranger team requires to change the column mask hash as SHA-512 in FIPS mode without changing API. This patch changes the MaskFunctions::MaskHash() for string type to use SHA-512 in FIPS mode. Testing: - Passed exhaustive tests. - Manually test the API. Change-Id: I422d4b11b31c3e6eb7963260a1da730579c4ca74 Reviewed-on: http://gerrit.cloudera.org:8080/16671 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exprs/expr-test.cc M be/src/exprs/mask-functions-ir.cc 2 files changed, 34 insertions(+), 11 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I422d4b11b31c3e6eb7963260a1da730579c4ca74 Gerrit-Change-Number: 16671 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10295: fix analytic limit pushdown with no predicates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16663 ) Change subject: IMPALA-10295: fix analytic limit pushdown with no predicates .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7586/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I254e85edd5ea6b6e76d20cbdf27fd88059a98a21 Gerrit-Change-Number: 16663 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 30 Oct 2020 00:35:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10295: fix analytic limit pushdown with no predicates
Hello Aman Sinha, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16663 to look at the new patch set (#4). Change subject: IMPALA-10295: fix analytic limit pushdown with no predicates .. IMPALA-10295: fix analytic limit pushdown with no predicates This handles the first case where analytic limit pushdown could be applied incorrectly: when there are no predicates applied to the output of the analytic. If no rows are filtered out between the pre-analytic sort and the place where the top-N will be inserted, and the order matches exactly, we can push down the limit safely because the limit below the analytic will filter exactly the same rows as the limit above the analytic would. We add a helper to check if the sort order matches exactly and then handle the case with no select node correctly. We leave the other cases where there is a special predicate to be handled in the next patch of the series, as the logic there is a bit more subtle. Tests: Added regression planner and query tests that demonstrate the problem. Ran core tests. Change-Id: I254e85edd5ea6b6e76d20cbdf27fd88059a98a21 --- M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M testdata/workloads/functional-planner/queries/PlannerTest/limit-pushdown-analytic.test M testdata/workloads/tpch/queries/limit-pushdown-analytic.test 3 files changed, 238 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/16663/4 -- To view, visit http://gerrit.cloudera.org:8080/16663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I254e85edd5ea6b6e76d20cbdf27fd88059a98a21 Gerrit-Change-Number: 16663 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10295: fix analytic limit pushdown with no predicates
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16663 ) Change subject: IMPALA-10295: fix analytic limit pushdown with no predicates .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16663/3/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java File fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java: http://gerrit.cloudera.org:8080/#/c/16663/3/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@22 PS3, Line 22: import java.util.logging.Logger; > nit: for some reason this changed from slf4j to the java logger. Done -- To view, visit http://gerrit.cloudera.org:8080/16663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I254e85edd5ea6b6e76d20cbdf27fd88059a98a21 Gerrit-Change-Number: 16663 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 30 Oct 2020 00:10:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10294: Improvement to test skew reporting in runtime profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16662 ) Change subject: IMPALA-10294: Improvement to test_skew_reporting_in_runtime_profile .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7a36551f2507d724891707d26b7394fbe3a5657b Gerrit-Change-Number: 16662 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 30 Oct 2020 00:07:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10294: Improvement to test skew reporting in runtime profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16662 ) Change subject: IMPALA-10294: Improvement to test_skew_reporting_in_runtime_profile .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6623/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7a36551f2507d724891707d26b7394fbe3a5657b Gerrit-Change-Number: 16662 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 30 Oct 2020 00:07:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7585/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 23:22:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6622/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 23:09:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 23:09:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Wenzhe Zhou has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode To compliant with FIPS requirement, we should use OpenSSL libraries for cryptographic hash functions, instead of own hash functions. This patch replace MD5 and SHA1 functions in Squeasel Web server with OpenSSL APIs. It also force to turn off Digest Authorization for Web server in FIPS approved mode since Digest Authorization use MD5 hash and it doesn't comply with FIPS 140-2. Testing: - Passed webserver-test. - Passed exhaustive tests. - Manually verified HTTP Digest Authorization could not be enabled by setting webserver_password_file on a FIPS enabled cluster. Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f --- M be/src/thirdparty/squeasel/squeasel.c M be/src/util/webserver-test.cc M be/src/util/webserver.cc 3 files changed, 38 insertions(+), 338 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/16630/8 -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 8 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/16630/7/be/src/util/webserver.cc File be/src/util/webserver.cc: http://gerrit.cloudera.org:8080/#/c/16630/7/be/src/util/webserver.cc@401 PS7, Line 401: stringstream ss; > This is unnecessary, you can just pass the string directly into the Status Right, fixed it. -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 23:04:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/16630/7/be/src/util/webserver.cc File be/src/util/webserver.cc: http://gerrit.cloudera.org:8080/#/c/16630/7/be/src/util/webserver.cc@401 PS7, Line 401: stringstream ss; This is unnecessary, you can just pass the string directly into the Status constructor. -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 22:38:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7584/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 22:15:48 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Rewrite ImpalaHttpClient.py to use the requests package
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16678 ) Change subject: [WIP] Rewrite ImpalaHttpClient.py to use the requests package .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7583/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16678 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1030fd7b74b5c1cfaeed8a57fbcbb7a0703dd56 Gerrit-Change-Number: 16678 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 22:13:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10295: fix analytic limit pushdown with no predicates
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16663 ) Change subject: IMPALA-10295: fix analytic limit pushdown with no predicates .. Patch Set 3: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/16663/2/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java File fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java: http://gerrit.cloudera.org:8080/#/c/16663/2/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@478 PS2, Line 478: if (!analyticSortExprs.get(i).equals(sortExprs.get(i))) return false; > It returns true in some circumstances. That particular example didn't consi Done http://gerrit.cloudera.org:8080/#/c/16663/3/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java File fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java: http://gerrit.cloudera.org:8080/#/c/16663/3/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@22 PS3, Line 22: import java.util.logging.Logger; nit: for some reason this changed from slf4j to the java logger. -- To view, visit http://gerrit.cloudera.org:8080/16663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I254e85edd5ea6b6e76d20cbdf27fd88059a98a21 Gerrit-Change-Number: 16663 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Oct 2020 22:07:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. Patch Set 2: Test failure is a dataload issue unrelated to this change. Retrying -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Oct 2020 22:01:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6621/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Oct 2020 22:01:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6617/ -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Oct 2020 21:58:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Wenzhe Zhou has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode To compliant with FIPS requirement, we should use OpenSSL libraries for cryptographic hash functions, instead of own hash functions. This patch replace MD5 and SHA1 functions in Squeasel Web server with OpenSSL APIs. It also force to turn off Digest Authorization for Web server in FIPS approved mode since Digest Authorization use MD5 hash and it doesn't comply with FIPS 140-2. Testing: - Passed webserver-test. - Passed exhaustive tests. - Manually verified HTTP Digest Authorization could not be enabled by setting webserver_password_file on a FIPS enabled cluster. Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f --- M be/src/thirdparty/squeasel/squeasel.c M be/src/util/webserver-test.cc M be/src/util/webserver.cc 3 files changed, 38 insertions(+), 336 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/16630/7 -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 7 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] [WIP] Rewrite ImpalaHttpClient.py to use the requests package
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16678 ) Change subject: [WIP] Rewrite ImpalaHttpClient.py to use the requests package .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6620/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16678 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1030fd7b74b5c1cfaeed8a57fbcbb7a0703dd56 Gerrit-Change-Number: 16678 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 21:53:00 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Rewrite ImpalaHttpClient.py to use the requests package
Thomas Tauber-Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16678 Change subject: [WIP] Rewrite ImpalaHttpClient.py to use the requests package .. [WIP] Rewrite ImpalaHttpClient.py to use the requests package Change-Id: Ib1030fd7b74b5c1cfaeed8a57fbcbb7a0703dd56 --- M fe/src/test/java/org/apache/impala/customcluster/LdapImpalaShellTest.java M shell/ImpalaHttpClient.py M shell/impala_client.py 3 files changed, 70 insertions(+), 128 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/16678/1 -- To view, visit http://gerrit.cloudera.org:8080/16678 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib1030fd7b74b5c1cfaeed8a57fbcbb7a0703dd56 Gerrit-Change-Number: 16678 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-10303: Fix warnings from impala-shell with --quiet
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/16673 ) Change subject: IMPALA-10303: Fix warnings from impala-shell with --quiet .. Patch Set 2: Code-Review+2 LGTM -- To view, visit http://gerrit.cloudera.org:8080/16673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e94c9445ffba159725bacd6f6bc36f7c91b88fe Gerrit-Change-Number: 16673 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 20:57:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10303: Fix warnings from impala-shell with --quiet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16673 ) Change subject: IMPALA-10303: Fix warnings from impala-shell with --quiet .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e94c9445ffba159725bacd6f6bc36f7c91b88fe Gerrit-Change-Number: 16673 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 20:58:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10303: Fix warnings from impala-shell with --quiet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16673 ) Change subject: IMPALA-10303: Fix warnings from impala-shell with --quiet .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6619/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e94c9445ffba159725bacd6f6bc36f7c91b88fe Gerrit-Change-Number: 16673 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 20:58:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/16630/6/be/src/util/webserver-test.cc File be/src/util/webserver-test.cc: http://gerrit.cloudera.org:8080/#/c/16630/6/be/src/util/webserver-test.cc@439 PS6, Line 439: if (FIPS_mode()) return; Once we're returning an error status (per my other comment), lets have this test check for that, ie. do an if(FIPS_mode()) ASSERT_ERROR(webserver.Start()) http://gerrit.cloudera.org:8080/#/c/16630/6/be/src/util/webserver.cc File be/src/util/webserver.cc: http://gerrit.cloudera.org:8080/#/c/16630/6/be/src/util/webserver.cc@401 PS6, Line 401: LOG(WARNING) << "HTTP digest authorization is not supported in FIPS approved mode."; We should probably return an error status here, just to be sure people don't accidentally misconfigure this. -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 20:47:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10298: Change column mask hash as SHA512 in FIPS mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16671 ) Change subject: IMPALA-10298: Change column mask hash as SHA512 in FIPS mode .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I422d4b11b31c3e6eb7963260a1da730579c4ca74 Gerrit-Change-Number: 16671 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 19:46:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10298: Change column mask hash as SHA512 in FIPS mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16671 ) Change subject: IMPALA-10298: Change column mask hash as SHA512 in FIPS mode .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6618/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I422d4b11b31c3e6eb7963260a1da730579c4ca74 Gerrit-Change-Number: 16671 Gerrit-PatchSet: 2 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 19:46:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10298: Change column mask hash as SHA512 in FIPS mode
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16671 ) Change subject: IMPALA-10298: Change column mask hash as SHA512 in FIPS mode .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I422d4b11b31c3e6eb7963260a1da730579c4ca74 Gerrit-Change-Number: 16671 Gerrit-PatchSet: 1 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 19:46:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7582/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Thu, 29 Oct 2020 19:44:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode
Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16630 ) Change subject: IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode .. IMPALA-10206: Avoid MD5 Digest Authorization in FIPS approved mode To compliant with FIPS requirement, we should use OpenSSL libraries for cryptographic hash functions, instead of own hash functions. This patch replace MD5 and SHA1 functions in Squeasel Web server with OpenSSL APIs. It also force to turn off Digest Authorization for Web server in FIPS approved mode since Digest Authorization use MD5 hash and it doesn't comply with FIPS 140-2. Testing: - Passed exhaustive tests. - Manually verified HTTP Digest Authorization could not be enabled by setting webserver_password_file on a FIPS enabled cluster. Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f --- M be/src/thirdparty/squeasel/squeasel.c M be/src/util/webserver-test.cc M be/src/util/webserver.cc 3 files changed, 31 insertions(+), 333 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/16630/6 -- To view, visit http://gerrit.cloudera.org:8080/16630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie075389b3ab65c612d64ba58e16a10b19bdf4d6f Gerrit-Change-Number: 16630 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10295: fix analytic limit pushdown with no predicates
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16663 ) Change subject: IMPALA-10295: fix analytic limit pushdown with no predicates .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/7581/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I254e85edd5ea6b6e76d20cbdf27fd88059a98a21 Gerrit-Change-Number: 16663 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Oct 2020 18:06:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10295: fix analytic limit pushdown with no predicates
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16663 )
Change subject: IMPALA-10295: fix analytic limit pushdown with no predicates
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/16663/2/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java
File fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java:
http://gerrit.cloudera.org:8080/#/c/16663/2/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@478
PS2, Line 478: if (!analyticSortExprs.get(i).equals(sortExprs.get(i)))
return false;
> Will this comparison return True if for example both the analytic sort expr
It returns true in some circumstances. That particular example didn't consider
them equal because we materialized the order by expression into a slot of the
sort tuple, and one of the exprs referenced that slot and the other referenced
the original slot.
So this is now slightly more general than the previous variant, which only
handled bare SlotRefs, but it doesn't handle all cases where the optimisation
might be valid.
http://gerrit.cloudera.org:8080/#/c/16663/2/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@480
PS2, Line 480: if
(!sortInfo.getIsAscOrder().get(i).equals(analyticSortInfo.getIsAscOrder().get(i)))
{
> line too long (93 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16663/2/fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java@483
PS2, Line 483: if
(!sortInfo.getNullsFirst().get(i).equals(analyticSortInfo.getNullsFirst().get(i)))
{
> line too long (93 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/16663/2/testdata/workloads/functional-planner/queries/PlannerTest/limit-pushdown-analytic.test
File
testdata/workloads/functional-planner/queries/PlannerTest/limit-pushdown-analytic.test:
http://gerrit.cloudera.org:8080/#/c/16663/2/testdata/workloads/functional-planner/queries/PlannerTest/limit-pushdown-analytic.test@213
PS2, Line 213: # Limit pushdown should be applied
> Change this comment according to the new behavior.
Done
--
To view, visit http://gerrit.cloudera.org:8080/16663
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I254e85edd5ea6b6e76d20cbdf27fd88059a98a21
Gerrit-Change-Number: 16663
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Thu, 29 Oct 2020 17:53:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10295: fix analytic limit pushdown with no predicates
Hello Aman Sinha, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16663 to look at the new patch set (#3). Change subject: IMPALA-10295: fix analytic limit pushdown with no predicates .. IMPALA-10295: fix analytic limit pushdown with no predicates This handles the first case where analytic limit pushdown could be applied incorrectly: when there are no predicates applied to the output of the analytic. If no rows are filtered out between the pre-analytic sort and the place where the top-N will be inserted, and the order matches exactly, we can push down the limit safely because the limit below the analytic will filter exactly the same rows as the limit above the analytic would. We add a helper to check if the sort order matches exactly and then handle the case with no select node correctly. We leave the other cases where there is a special predicate to be handled in the next patch of the series, as the logic there is a bit more subtle. Tests: Added regression planner and query tests that demonstrate the problem. Ran core tests. Change-Id: I254e85edd5ea6b6e76d20cbdf27fd88059a98a21 --- M fe/src/main/java/org/apache/impala/planner/AnalyticEvalNode.java M testdata/workloads/functional-planner/queries/PlannerTest/limit-pushdown-analytic.test M testdata/workloads/tpch/queries/limit-pushdown-analytic.test 3 files changed, 238 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/16663/3 -- To view, visit http://gerrit.cloudera.org:8080/16663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I254e85edd5ea6b6e76d20cbdf27fd88059a98a21 Gerrit-Change-Number: 16663 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Oct 2020 17:46:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6617/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Oct 2020 17:46:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10132: Implement ds hll estimate bounds as string() function.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16626 ) Change subject: IMPALA-10132: Implement ds_hll_estimate_bounds_as_string() function. .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46bf8263e8fd3877a087b9cb6f0d1a2392bb9153 Gerrit-Change-Number: 16626 Gerrit-PatchSet: 6 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 17:45:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10132: Implement ds hll estimate bounds as string() function.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16626 ) Change subject: IMPALA-10132: Implement ds_hll_estimate_bounds_as_string() function. .. IMPALA-10132: Implement ds_hll_estimate_bounds_as_string() function. This function receives a string that is a serialized Apache DataSketches HLL sketch and optional kappa that is a number of standard deviations from the mean: 1, 2 or 3 (default 2). Returns estimate and bounds with the values separated with commas. The result is three values: estimate, lower bound and upper bound. ds_hll_estimate_bounds_as_string(sketch [, kappa]) Kappa: 1 represent the 68.3% confidence bounds 2 represent the 95.4% confidence bounds 3 represent the 99.7% confidence bounds Note, ds_hll_estimate_bounds() should return an Array of doubles as the result but with that we have to wait for the complex type support. Until, we provide ds_hll_estimate_bounds_as_string() that can be deprecated once we have array support. Tracking Jira for returning complex types from functions is IMPALA-9520. Example: select ds_hll_estimate_bounds_as_string(ds_hll_sketch(int_col)) from functional_parquet.alltypestiny; +--+ | ds_hll_estimate_bounds_as_string(ds_hll_sketch(int_col)) | +--+ | 2,2,2.0002 | +--+ Change-Id: I46bf8263e8fd3877a087b9cb6f0d1a2392bb9153 Reviewed-on: http://gerrit.cloudera.org:8080/16626 Reviewed-by: Gabor Kaszab Tested-by: Impala Public Jenkins --- M be/src/exprs/datasketches-common.h M be/src/exprs/datasketches-functions-ir.cc M be/src/exprs/datasketches-functions.h M common/function-registry/impala_functions.py M testdata/workloads/functional-query/queries/QueryTest/datasketches-hll.test 5 files changed, 103 insertions(+), 0 deletions(-) Approvals: Gabor Kaszab: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I46bf8263e8fd3877a087b9cb6f0d1a2392bb9153 Gerrit-Change-Number: 16626 Gerrit-PatchSet: 7 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Oct 2020 17:43:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10294: Improvement to test skew reporting in runtime profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16662 ) Change subject: IMPALA-10294: Improvement to test_skew_reporting_in_runtime_profile .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7580/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7a36551f2507d724891707d26b7394fbe3a5657b Gerrit-Change-Number: 16662 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 29 Oct 2020 17:42:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. Patch Set 1: Code-Review+1 Looks good! -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Comment-Date: Thu, 29 Oct 2020 17:26:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10294: Improvement to test skew reporting in runtime profile
Qifan Chen has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16662 ) Change subject: IMPALA-10294: Improvement to test_skew_reporting_in_runtime_profile .. IMPALA-10294: Improvement to test_skew_reporting_in_runtime_profile This fix improved the skew reporting test by lowering the threshold to 0 and by taking care of the extreme case of no skews. Testing: 1. Unit testing. Change-Id: I7a36551f2507d724891707d26b7394fbe3a5657b --- M tests/query_test/test_observability.py 1 file changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/16662/2 -- To view, visit http://gerrit.cloudera.org:8080/16662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7a36551f2507d724891707d26b7394fbe3a5657b Gerrit-Change-Number: 16662 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16677 ) Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7579/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 17:15:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9355: TestExchangeMemUsage.test exchange mem usage scaling doesn't hit the memory limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16676 ) Change subject: IMPALA-9355: TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7578/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16676 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c33e3b254ce75054bf6cee0eff8b7206fa0c122 Gerrit-Change-Number: 16676 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 16:57:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10302: Enable logging at the INFO level for test scanners fuzz.py
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16677 Change subject: IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py .. IMPALA-10302: Enable logging at the INFO level for test_scanners_fuzz.py This changes test_scanners_fuzz.py to set the logging level to INFO. By default, it is WARNING, so it was missing some useful INFO log messages like the random seed used. This also fixes formatting on one of the log lines. Testing: - Ran test_scanners_fuzz.py locally and checked to make sure the INFO messages were present Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 --- M tests/query_test/test_scanners_fuzz.py 1 file changed, 6 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/16677/1 -- To view, visit http://gerrit.cloudera.org:8080/16677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ida4a9cbed6572520998def9618a8b4189c1ba799 Gerrit-Change-Number: 16677 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-9355: TestExchangeMemUsage.test exchange mem usage scaling doesn't hit the memory limit
Qifan Chen has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16676 Change subject: IMPALA-9355: TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit .. IMPALA-9355: TestExchangeMemUsage.test_exchange_mem_usage_scaling doesn't hit the memory limit This fix improved the robustness of the test by lowering the memory limit from 170MB to 168MB. In 10 local tests, the memory allocation shortage at EXCHANGE_NODE was as low as 8KB without exceeding the limit of 170MB. Testing: 1. Unit testing with 200 test iterations. Change-Id: I3c33e3b254ce75054bf6cee0eff8b7206fa0c122 --- M testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/16676/2 -- To view, visit http://gerrit.cloudera.org:8080/16676 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3c33e3b254ce75054bf6cee0eff8b7206fa0c122 Gerrit-Change-Number: 16676 Gerrit-PatchSet: 2 Gerrit-Owner: Qifan Chen
[Impala-ASF-CR] IMPALA-10166 (part 1): ALTER TABLE for Iceberg tables
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16606 ) Change subject: IMPALA-10166 (part 1): ALTER TABLE for Iceberg tables .. Patch Set 8: Code-Review+1 (4 comments) Hey! I did a quick review and only found nits. I'm fine to +2 once they are addressed. http://gerrit.cloudera.org:8080/#/c/16606/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16606/8//COMMIT_MSG@9 PS8, Line 9: implement nit: implements http://gerrit.cloudera.org:8080/#/c/16606/8/fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java File fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java: http://gerrit.cloudera.org:8080/#/c/16606/8/fe/src/main/java/org/apache/impala/analysis/AlterTableReplaceColsStmt.java@25 PS8, Line 25: import org.apache.impala.catalog.FeIcebergTable; Is this import needed? http://gerrit.cloudera.org:8080/#/c/16606/8/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/16606/8/fe/src/main/java/org/apache/impala/service/IcebergCatalogOpExecutor.java@118 PS8, Line 118: support nit: supports http://gerrit.cloudera.org:8080/#/c/16606/8/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test File testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test: http://gerrit.cloudera.org:8080/#/c/16606/8/testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test@117 PS8, Line 117: CREATE TABLE iceberg_test9( Instead of creating a new table here, could you re-use any of the ones above? It might have a good effect on the runtime of the test. (same goes for the other create table below) -- To view, visit http://gerrit.cloudera.org:8080/16606 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5104cc47c7b42dacdb52983f503cd263135d6bfc Gerrit-Change-Number: 16606 Gerrit-PatchSet: 8 Gerrit-Owner: wangsheng Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Thu, 29 Oct 2020 14:34:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10132: Implement ds hll estimate bounds as string() function.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16626 ) Change subject: IMPALA-10132: Implement ds_hll_estimate_bounds_as_string() function. .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6616/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46bf8263e8fd3877a087b9cb6f0d1a2392bb9153 Gerrit-Change-Number: 16626 Gerrit-PatchSet: 6 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 12:24:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10132: Implement ds hll estimate bounds as string() function.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16626 ) Change subject: IMPALA-10132: Implement ds_hll_estimate_bounds_as_string() function. .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46bf8263e8fd3877a087b9cb6f0d1a2392bb9153 Gerrit-Change-Number: 16626 Gerrit-PatchSet: 6 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 12:22:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7097 Print EC info in the query plan and profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16587 ) Change subject: IMPALA-7097 Print EC info in the query plan and profile .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6ea378914624a714fde820d290b3b9c43325c6a1 Gerrit-Change-Number: 16587 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Shant Hovsepian Gerrit-Comment-Date: Thu, 29 Oct 2020 10:35:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7097 Print EC info in the query plan and profile
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16587 ) Change subject: IMPALA-7097 Print EC info in the query plan and profile .. IMPALA-7097 Print EC info in the query plan and profile This fix added the functionality to show the number of erasure coded files and the total size of such files in the scan node in the query plan and profile. Shown below are two examples for the HDFS file system. Non-partitioned table: 00:SCAN HDFS [default.test_show_ec_nonpart, RANDOM] HDFS partitions=1/1 files=2 size=1.65KB erasure coded: files=2 size=1.65KB stored statistics: Partitioned table: 00:SCAN HDFS [default.test_show_ec_part] HDFS partitions=4/4 files=4 size=2.36KB erasure coded: files=3 size=1.77KB row-size=12B cardinality=999 Testing: 1. Unit testing; 2. Ran Core tests successfully. Change-Id: I6ea378914624a714fde820d290b3b9c43325c6a1 Reviewed-on: http://gerrit.cloudera.org:8080/16587 Reviewed-by: Aman Sinha Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java 1 file changed, 28 insertions(+), 2 deletions(-) Approvals: Aman Sinha: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6ea378914624a714fde820d290b3b9c43325c6a1 Gerrit-Change-Number: 16587 Gerrit-PatchSet: 9 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Shant Hovsepian
[Impala-ASF-CR] IMPALA-10132: Implement ds hll estimate bounds as string() function.
Fucun Chu has posted comments on this change. ( http://gerrit.cloudera.org:8080/16626 ) Change subject: IMPALA-10132: Implement ds_hll_estimate_bounds_as_string() function. .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/16626/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16626/4//COMMIT_MSG@10 PS4, Line 10: > nit: please remove the trailing spaces Done http://gerrit.cloudera.org:8080/#/c/16626/4//COMMIT_MSG@11 PS4, Line 11: returns > nit: start with capital letter Done -- To view, visit http://gerrit.cloudera.org:8080/16626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46bf8263e8fd3877a087b9cb6f0d1a2392bb9153 Gerrit-Change-Number: 16626 Gerrit-PatchSet: 4 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 08:58:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10132: Implement ds hll estimate bounds as string() function.
Hello Gabor Kaszab, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16626 to look at the new patch set (#5). Change subject: IMPALA-10132: Implement ds_hll_estimate_bounds_as_string() function. .. IMPALA-10132: Implement ds_hll_estimate_bounds_as_string() function. This function receives a string that is a serialized Apache DataSketches HLL sketch and optional kappa that is a number of standard deviations from the mean: 1, 2 or 3 (default 2). Returns estimate and bounds with the values separated with commas. The result is three values: estimate, lower bound and upper bound. ds_hll_estimate_bounds_as_string(sketch [, kappa]) Kappa: 1 represent the 68.3% confidence bounds 2 represent the 95.4% confidence bounds 3 represent the 99.7% confidence bounds Note, ds_hll_estimate_bounds() should return an Array of doubles as the result but with that we have to wait for the complex type support. Until, we provide ds_hll_estimate_bounds_as_string() that can be deprecated once we have array support. Tracking Jira for returning complex types from functions is IMPALA-9520. Example: select ds_hll_estimate_bounds_as_string(ds_hll_sketch(int_col)) from functional_parquet.alltypestiny; +--+ | ds_hll_estimate_bounds_as_string(ds_hll_sketch(int_col)) | +--+ | 2,2,2.0002 | +--+ Change-Id: I46bf8263e8fd3877a087b9cb6f0d1a2392bb9153 --- M be/src/exprs/datasketches-common.h M be/src/exprs/datasketches-functions-ir.cc M be/src/exprs/datasketches-functions.h M common/function-registry/impala_functions.py M testdata/workloads/functional-query/queries/QueryTest/datasketches-hll.test 5 files changed, 103 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/16626/5 -- To view, visit http://gerrit.cloudera.org:8080/16626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I46bf8263e8fd3877a087b9cb6f0d1a2392bb9153 Gerrit-Change-Number: 16626 Gerrit-PatchSet: 5 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10279: Import CPC functionality from DataSketches
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16645 ) Change subject: IMPALA-10279: Import CPC functionality from DataSketches .. Patch Set 6: (5 comments) http://gerrit.cloudera.org:8080/#/c/16645/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16645/6//COMMIT_MSG@11 PS6, Line 11: First, I updated our existing snapshot of DataSketches to the : following commit:8ae0ab5703783c8c360d48e577a6d450d3514ece : "https://github.com/apache/incubator-datasketches-cpp/pull/172; I see that your patch doesn't change any of the existing files in be/src/thirdparty/datasketches. Does this mean that none of the files in hll/include/ kll/include/ common/include/ directories change in Apache DataCketches between the commits of c1a6f8edb49699520f248d3d02019b87429b4241 and 8ae0ab5703783c8c360d48e577a6d450d3514ece (I just want to be 100% sure that we don't miss any updates on these files) http://gerrit.cloudera.org:8080/#/c/16645/6/be/src/exprs/datasketches-test.cc File be/src/exprs/datasketches-test.cc: http://gerrit.cloudera.org:8080/#/c/16645/6/be/src/exprs/datasketches-test.cc@82 PS6, Line 82: get an estimate for quantiles I think here you don't get the quantiles estimates but the cardinality estimates. http://gerrit.cloudera.org:8080/#/c/16645/6/be/src/exprs/datasketches-test.cc@101 PS6, Line 101: and prints the result This code doesn't print the results. (and shouldn;t. I guess this is a copy-paste comment from the official site) http://gerrit.cloudera.org:8080/#/c/16645/6/be/src/exprs/datasketches-test.cc@114 PS6, Line 114: EXPECT_EQ(149796, (int)sketch.get_estimate()); I'm not 100% familiar with the characteristics of the CPC algorithm. Are you sure that this result is deterministic? http://gerrit.cloudera.org:8080/#/c/16645/6/be/src/thirdparty/datasketches/README.md File be/src/thirdparty/datasketches/README.md: http://gerrit.cloudera.org:8080/#/c/16645/6/be/src/thirdparty/datasketches/README.md@12 PS6, Line 12: The git branch of the snapshot I used as a source for the files: 2.1.0-incubating : The hash: c1a6f8edb49699520f248d3d02019b87429b4241 : : Browse the source files here: : https://github.com/apache/incubator-datasketches-cpp/tree/2.1.0-incubating In case you updated the content of this folder to a newer version of Apache DataSketches then please update this section too. -- To view, visit http://gerrit.cloudera.org:8080/16645 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1e8d0c2e80df95fa84af82c64d493df9bbb34a8c Gerrit-Change-Number: 16645 Gerrit-PatchSet: 6 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 07:35:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10132: Implement ds hll estimate bounds as string() function.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16626 ) Change subject: IMPALA-10132: Implement ds_hll_estimate_bounds_as_string() function. .. Patch Set 4: Code-Review+1 (2 comments) Thanks for the changes! I have some nits but apart from those I'm fine with the patch. Will give a +2 once you address these. http://gerrit.cloudera.org:8080/#/c/16626/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16626/4//COMMIT_MSG@10 PS4, Line 10: nit: please remove the trailing spaces http://gerrit.cloudera.org:8080/#/c/16626/4//COMMIT_MSG@11 PS4, Line 11: returns nit: start with capital letter -- To view, visit http://gerrit.cloudera.org:8080/16626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I46bf8263e8fd3877a087b9cb6f0d1a2392bb9153 Gerrit-Change-Number: 16626 Gerrit-PatchSet: 4 Gerrit-Owner: Fucun Chu Gerrit-Reviewer: Fucun Chu Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 29 Oct 2020 07:03:49 + Gerrit-HasComments: Yes
