[Impala-ASF-CR] IMPALA-10414: fix memory leak when canceling the retried query
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/17735 ) Change subject: IMPALA-10414: fix memory leak when canceling the retried query .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/17735/8/be/src/runtime/query-driver.cc File be/src/runtime/query-driver.cc: http://gerrit.cloudera.org:8080/#/c/17735/8/be/src/runtime/query-driver.cc@368 PS8, Line 368: RETURN_VOID_IF_ERROR(query_handle->Finalize(true, nullptr)); > Agree. When ClientRequestState::Cancel() is called with wait_until_finalize We can add a new API in Coordinator to wait until finalized. In ClientRequestState::Cancel(), we can call this new API instead of Coordinator::Cancel() if both parameter wait_until_finalized and is_cancelled_ are true. -- To view, visit http://gerrit.cloudera.org:8080/17735 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If804ca65da1794c819a6b2e6567ea7651ab5112f Gerrit-Change-Number: 17735 Gerrit-PatchSet: 9 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Tue, 12 Apr 2022 05:52:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. IMPALA-11023: Raise error when delete file is found in an Iceberg table Iceberg V2 DeleteFiles are skipped during scans and the whole content of the DataFiles are returned. This commit adds an extra check to prevent scanning tables that have delete files to avoid unexpected results till merge on read is supported. Metadata operations are allowed on tables with delete files. Testing: - Added e2e test. Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Reviewed-on: http://gerrit.cloudera.org:8080/18383 Reviewed-by: Zoltan Borok-Nagy Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/0-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test 17 files changed, 217 insertions(+), 9 deletions(-) Approvals: Zoltan Borok-Nagy: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 6 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Apr 2022 19:35:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11236: Upgrade ehcache sizeof library to 0.4.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18400 ) Change subject: IMPALA-11236: Upgrade ehcache sizeof library to 0.4.0 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10427/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I853264684aa2ba9c3743f9e53bd52fa2752a45a1 Gerrit-Change-Number: 18400 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Mon, 11 Apr 2022 19:31:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11236: Upgrade ehcache sizeof library to 0.4.0
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/18400 ) Change subject: IMPALA-11236: Upgrade ehcache sizeof library to 0.4.0 .. Patch Set 1: Hi Quanlong and Vincent, please let me know if you have any comment on this patch. Thanks very much for the help! -- To view, visit http://gerrit.cloudera.org:8080/18400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I853264684aa2ba9c3743f9e53bd52fa2752a45a1 Gerrit-Change-Number: 18400 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Mon, 11 Apr 2022 19:12:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11236: Upgrade ehcache sizeof library to 0.4.0
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18400 Change subject: IMPALA-11236: Upgrade ehcache sizeof library to 0.4.0 .. IMPALA-11236: Upgrade ehcache sizeof library to 0.4.0 It was found that the ehcache sizeof (version 0.3.0) library used by CatalogdMetaProvider can fail due to an InaccessibleObjectException during reflection and cause the impalad to stuck during startup indefinitely. This could trigger IMPALA-7168 which affects sync_ddl queries running on unaffected impalads. This patch upgrades the version of the ehcache sizeof library so that the exception could be handled within the ehcache sizeof library after the upgrade. Testing: - Verified that the patch passes Impala's core tests. Change-Id: I853264684aa2ba9c3743f9e53bd52fa2752a45a1 --- M fe/pom.xml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/18400/1 -- To view, visit http://gerrit.cloudera.org:8080/18400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I853264684aa2ba9c3743f9e53bd52fa2752a45a1 Gerrit-Change-Number: 18400 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8035/ -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 11 Apr 2022 18:19:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11232: Do not add some jars to HADOOP CLASSPATH when starting HMS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18398 ) Change subject: IMPALA-11232: Do not add some jars to HADOOP_CLASSPATH when starting HMS .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10426/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifd66fd99a346835b9f81f95b5f046273fcce2590 Gerrit-Change-Number: 18398 Gerrit-PatchSet: 3 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Apr 2022 17:14:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11232: Do not add some jars to HADOOP CLASSPATH when starting HMS
Fang-Yu Rao has uploaded a new patch set (#3). (
http://gerrit.cloudera.org:8080/18398 )
Change subject: IMPALA-11232: Do not add some jars to HADOOP_CLASSPATH when
starting HMS
..
IMPALA-11232: Do not add some jars to HADOOP_CLASSPATH when starting HMS
This patch changes the line that added to HADOOP_CLASSPATH all the jar
files in the folder ${RANGER_HOME}/ews/webapp/WEB-INF/lib to a line that
only includes those jar files with names starting with "ranger-" since
almost all other jar files do not seem to be necessary to run the E2E
test of test_hive_with_ranger_setup.
This way we also avoid adding too many paths to HADOOP_CLASSPATH, which
in turn could result in Hadoop not being able to return its version to
the script that starts HMS due to the error of "Argument list too long".
Testing:
- Verified after this patch, test_hive_with_ranger_setup still
succeeds.
- Verified in a local development environment that the length of
Hadoop's environment variable 'CLASSPATH' logged in
hive-metastore.out decreases from 100,876 characters to 62,634
characters when executing run-hive-server.sh with the flag
'-with_ranger' if $HADOOP_SHELL_SCRIPT_DEBUG is "true" and
$IMPALA_HOME is "/home/fangyurao/Impala_for_FE".
Change-Id: Ifd66fd99a346835b9f81f95b5f046273fcce2590
---
M testdata/bin/run-hive-server.sh
1 file changed, 5 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/18398/3
--
To view, visit http://gerrit.cloudera.org:8080/18398
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifd66fd99a346835b9f81f95b5f046273fcce2590
Gerrit-Change-Number: 18398
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18399 ) Change subject: IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10425/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa Gerrit-Change-Number: 18399 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 11 Apr 2022 16:53:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode
Hello Tamas Mate, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18399 to look at the new patch set (#2). Change subject: IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode .. IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode IcebergScanNode interprets the timestamp literals as UTC timestamps during predicate pushdown to Iceberg. It causes problems when the Iceberg table uses TIMESTAMPTZ (which corresponds to TIMESTAMP WITH LOCAL TIME ZONE in SQL) because in the scanners we assume that the timestamp literals in a query are in local timezone. Hence, if the Iceberg table is partitioned by HOUR(ts), and Impala is running in a different timezone than UTC, then the following query doesn't return any rows: SELECT * from t WHERE ts = ; Because during predicate pushdown the timestamp is interpreted as a UTC timestamp (no conversion from local to UTC), but during query execution the timestamp data in the files are converted to local timezone, then compared to . I.e. in the scanner the assumption is that is in local timezone. On the other hand, when Iceberg type TIMESTAMP (which correcponds to TIMESTAMP WITHOUT TIME ZONE in SQL) is used, then we should just push down the timestamp values without any conversion. In this case there is no conversion in the scanners either. Testing: * added e2e test with TIMESTAMPTZ * added e2e test with TIMESTAMP Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa --- M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M testdata/data/README M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test 6 files changed, 191 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/18399/2 -- To view, visit http://gerrit.cloudera.org:8080/18399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa Gerrit-Change-Number: 18399 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 5: Code-Review+2 Thanks for fixing this! LGTM! -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Apr 2022 16:30:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18399 ) Change subject: IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10424/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa Gerrit-Change-Number: 18399 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 11 Apr 2022 16:17:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10930: Bump the Java artifact versions to 4.1.0-SNAPSHOT
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18390 ) Change subject: IMPALA-10930: Bump the Java artifact versions to 4.1.0-SNAPSHOT .. IMPALA-10930: Bump the Java artifact versions to 4.1.0-SNAPSHOT This changes the Maven pom.xml files to use verison 4.1.0-SNAPSHOT rather than 4.0.0-SNAPSHOT. In the past, these versions were a fixed value, but that changed with IMPALA-10198. This is a new step that needs to happen on each release. Testing: - Ran a build Change-Id: I10a589b4fbc15048199943a0e06d079f57840239 Reviewed-on: http://gerrit.cloudera.org:8080/18390 Reviewed-by: Tamas Mate Tested-by: Impala Public Jenkins --- M fe/pom.xml M java/TableFlattener/pom.xml M java/datagenerator/pom.xml M java/executor-deps/pom.xml M java/ext-data-source/api/pom.xml M java/ext-data-source/pom.xml M java/ext-data-source/sample/pom.xml M java/ext-data-source/test/pom.xml M java/pom.xml M java/query-event-hook-api/pom.xml M java/shaded-deps/hive-exec/pom.xml M java/shaded-deps/s3a-aws-sdk/pom.xml M java/test-hive-udfs/pom.xml M java/yarn-extras/pom.xml 14 files changed, 14 insertions(+), 14 deletions(-) Approvals: Tamas Mate: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18390 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I10a589b4fbc15048199943a0e06d079f57840239 Gerrit-Change-Number: 18390 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18399 Change subject: IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode .. IMPALA-10850: Interpret timestamp predicates in local timezone in IcebergScanNode IcebergScanNode interprets the timestamp literals as UTC timestamps during predicate pushdown to Iceberg. It causes problems when the Iceberg table uses TIMESTAMPTZ (which corresponds to TIMESTAMP WITH LOCAL TIME ZONE in SQL) because in the scanners we assume that the timestamp literals in a query are in local timezone. Hence, if the Iceberg table is partitioned by HOUR(ts), and Impala is running in a different timezone than UTC, then the following query doesn't return any rows: SELECT * from t WHERE ts = ; Because during predicate pushdown the timestamp is interpreted as a UTC timestamp (no conversion from local to UTC), but during query execution the timestamp data in the files are converted to local timezone, then compared to . I.e. in the scanner the assumption is that is in local timezone. On the other hand, when Iceberg type TIMESTAMP (which correcponds to TIMESTAMP WITHOUT TIME ZONE in SQL) is used, then we should just push down the timestamp values without any conversion. In this case there is no conversion in the scanners either. Testing: * added e2e test with TIMESTAMPTZ * added e2e test with TIMESTAMP Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa --- M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-query.test 5 files changed, 184 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/18399/1 -- To view, visit http://gerrit.cloudera.org:8080/18399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I181be5d2fa004f69b457f69ff82dc2f9877f46fa Gerrit-Change-Number: 18399 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8036/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Apr 2022 15:08:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 5: This build hit IMPALA-7864, restarting dry_run. -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Apr 2022 15:08:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8034/ -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Apr 2022 15:03:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10423/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 11 Apr 2022 14:09:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 2: (1 comment) Added a regression test, which turned out to be a bit more complex than I thought, as I found no easy way to check that a regexp does NOT match any line in the profile. http://gerrit.cloudera.org:8080/#/c/18385/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/18385/1//COMMIT_MSG@13 PS1, Line 13: : I have realized that this is not true, we don't report codegen errors here in case of async codegen. -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 11 Apr 2022 13:52:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18385 ) Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8035/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto Gerrit-Comment-Date: Mon, 11 Apr 2022 13:53:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile
Hello Quanlong Huang, Riza Suminto, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/18385 to look at the new patch set (#2). Change subject: IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile .. IMPALA-11200: Avoid redundant "Codegen enabled" messages in profile Before this patch the message was added to the profile in Open(), which can be called multiple times in subplans. Moved it to Close(), which is only called once in the lifetime of a Node/Aggregator. A drawback of this is that this info won't be visible when the Node is still active, but I don't think that it is a very useful info in a still running query. Also added a new feature to test_result_verifier.py: Inside RUNTIME_PROFILE section row_regex can be negated with !, so !row_regex [regex] means that regex is not matched by any line in the profile. Testing: - added a regression test Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f --- M be/src/exec/aggregator.cc M be/src/exec/exec-node.cc M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan-single-node.test M tests/common/test_result_verifier.py 4 files changed, 40 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/18385/2 -- To view, visit http://gerrit.cloudera.org:8080/18385 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iad2e31900ee6d29385cc8adc6bbf067d91f6450f Gerrit-Change-Number: 18385 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Riza Suminto
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8034/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Apr 2022 10:34:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10723: Treat materialized view as a table instead of a view
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17595 ) Change subject: IMPALA-10723: Treat materialized view as a table instead of a view .. Patch Set 14: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8033/ -- To view, visit http://gerrit.cloudera.org:8080/17595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If3108996124c6544a97fb0c34b6aff5e324a6cff Gerrit-Change-Number: 17595 Gerrit-PatchSet: 14 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 11 Apr 2022 09:35:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10422/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 11 Apr 2022 08:35:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11023: Raise error when delete file is found in an Iceberg table
Tamas Mate has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/18383 ) Change subject: IMPALA-11023: Raise error when delete file is found in an Iceberg table .. IMPALA-11023: Raise error when delete file is found in an Iceberg table Iceberg V2 DeleteFiles are skipped during scans and the whole content of the DataFiles are returned. This commit adds an extra check to prevent scanning tables that have delete files to avoid unexpected results till merge on read is supported. Metadata operations are allowed on tables with delete files. Testing: - Added e2e test. Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/planner/IcebergScanNode.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M testdata/data/README A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/0-0-fb178c51-b12a-4c5f-a66e-a8e9375daeba-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/data/00191-4-6e780302-527b-4911-8c6e-88d416adac57-1.parquet A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/0eadf173-0c84-4378-a9d0-5d7f47183978-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/8cbef400-daea-478a-858a-2baf2438f644-m0.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/snap-5725822353600261755-1-0eadf173-0c84-4378-a9d0-5d7f47183978.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/snap-6816997371555012807-1-8cbef400-daea-478a-858a-2baf2438f644.avro A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/v1.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/v2.metadata.json A testdata/data/iceberg_test/hadoop_catalog/ice/iceberg_v2_delete_positional/metadata/version-hint.text M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/queries/QueryTest/iceberg-negative.test 17 files changed, 217 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/18383/5 -- To view, visit http://gerrit.cloudera.org:8080/18383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6e9cbf2424b27157883d551f73e728ab4ec6d21e Gerrit-Change-Number: 18383 Gerrit-PatchSet: 5 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Gergely Fürnstáhl Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
