[Impala-ASF-CR] IMPALA-10361: Supported using field id to resolve columns for Iceberg tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Supported using field id to resolve columns for Iceberg tables .. Patch Set 2: (5 comments) Thanks WangSheng for working on this important task! I did a first pass and the change looks good to me. Looking forward for the tests. http://gerrit.cloudera.org:8080/#/c/16788/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16788/2//COMMIT_MSG@7 PS2, Line 7: Supported nit: Use field id to resolve columns for Iceberg tables http://gerrit.cloudera.org:8080/#/c/16788/2//COMMIT_MSG@11 PS2, Line 11: to choose field id : resolving. I think this should be the default for Iceberg tables. http://gerrit.cloudera.org:8080/#/c/16788/2/common/thrift/ImpalaInternalService.thrift File common/thrift/ImpalaInternalService.thrift: http://gerrit.cloudera.org:8080/#/c/16788/2/common/thrift/ImpalaInternalService.thrift@48 PS2, Line 48: FIELDID nit: FIELD_ID http://gerrit.cloudera.org:8080/#/c/16788/2/fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java File fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java: http://gerrit.cloudera.org:8080/#/c/16788/2/fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java@25 PS2, Line 25: nit: add comment http://gerrit.cloudera.org:8080/#/c/16788/2/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/16788/2/fe/src/main/java/org/apache/impala/catalog/Table.java@439 PS2, Line 439: By nit: From -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 2 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Mon, 30 Nov 2020 12:38:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10361: Supported using field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Supported using field id to resolve columns for Iceberg tables .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7742/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 2 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 27 Nov 2020 09:43:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Supported using field id to resolve columns for Iceberg tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Supported using field id to resolve columns for Iceberg tables .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7741/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 1 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 27 Nov 2020 09:26:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Supported using field id to resolve columns for Iceberg tables
wangsheng has posted comments on this change. ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Supported using field id to resolve columns for Iceberg tables .. Patch Set 2: Hi Zoltan,Gabor. I've already implement a first version for resolving columns by field id. I created a simple test table in my environment for array/struct/map, it worked in normal catalog mode, but return null for struct type in local catalog mode, I'm still working on this patch. Hope you can give me some suggestions, Thanks a lot. -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 2 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Fri, 27 Nov 2020 09:31:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10361: Supported using field id to resolve columns for Iceberg tables
wangsheng has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16788 ) Change subject: IMPALA-10361: Supported using field id to resolve columns for Iceberg tables .. IMPALA-10361: Supported using field id to resolve columns for Iceberg tables We supported resolve column by field id for Iceberg table in this patch. We can use 'set PARQUET_FALLBACK_SCHEMA_RESOLUTION=FIELDID' or 'set PARQUET_FALLBACK_SCHEMA_RESOLUTION=2' to choose field id resolving. But pay attention, if you use this for non-Iceberg table, the result will be NULL. Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 --- M be/src/exec/parquet/parquet-metadata-utils.cc M be/src/exec/parquet/parquet-metadata-utils.h M be/src/runtime/descriptors.cc M be/src/runtime/descriptors.h M be/src/runtime/types.cc M be/src/runtime/types.h M common/thrift/CatalogObjects.thrift M common/thrift/Descriptors.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/Types.thrift M fe/src/main/java/org/apache/impala/catalog/Column.java M fe/src/main/java/org/apache/impala/catalog/IcebergColumn.java A fe/src/main/java/org/apache/impala/catalog/IcebergStructField.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/Type.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java 18 files changed, 183 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/16788/2 -- To view, visit http://gerrit.cloudera.org:8080/16788 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I057bdc6ab2859cc4d40de5ed428d0c20028b8435 Gerrit-Change-Number: 16788 Gerrit-PatchSet: 2 Gerrit-Owner: wangsheng Gerrit-Reviewer: Impala Public Jenkins
