[
https://issues.apache.org/jira/browse/IMPALA-13853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17934637#comment-17934637
]
ASF subversion and git services commented on IMPALA-13853:
----------------------------------------------------------
Commit a49ff618f1137808c75cf81cccdadb980e89c34d in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a49ff618f ]
IMPALA-13853: Don't adjust Iceberg field IDs for data files that don't have
complex types
In migrated Iceberg tables we can have data files with missing field
IDs. We assume that their schema corresponds to the table schema at the
point when the table migration happened. This means during runtime we
can generate the field ids. The logic is more complicated when there are
complex types in the table and the table is partitioned. In such cases
we need to do some adjustments during field ID generation, in which case
we verify that the file schema corresponds to the table schema.
These adjustments are not needed when the table doesn't have complex
types, hence we can be a bit more relaxed and skip schema verification,
because field ID generation for top-level columns are not affected.
This means Impala would still be able to read the table if there were
trivial schema changes before migration.
With this change we allow all data files that have a compatible schema
with the table schema, which was the case before IMPALA-13364. This
behavior is also aligned with Hive.
Testing:
* e2e tests added for both Parquet and ORC files
Change-Id: Ib1f1d0cf36792d0400de346c83e999fa50c0fa67
Reviewed-on: http://gerrit.cloudera.org:8080/22610
Reviewed-by: Daniel Becker <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Don't adjust Iceberg field IDs for Parquet files that don't have complex types
> ------------------------------------------------------------------------------
>
> Key: IMPALA-13853
> URL: https://issues.apache.org/jira/browse/IMPALA-13853
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Zoltán Borók-Nagy
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 5.0.0
>
>
> In migrated Iceberg tables we can have data files with missing field IDs. We
> assume that their schema corresponds to the table schema at the point when
> the table migration happened. This means during runtime we can generate the
> field ids. The logic is more complicated when there are complex types in the
> table and the table is partitioned. In such cases we need to do some
> adjustments during field ID generation, and during that we verify that the
> file schema corresponds to the table schema (during migration).
> This adjustments is not needed when the table doesn't have complex types,
> hence we can be a bit more relaxed and skip schema verification. This means
> Impala would still be able to read the table if there were some trivial
> schema changes before migration.
> Repro:
> {noformat}
> create table mig(i int) partitioned by (p int)
> stored as parquet;
> insert into mig partition(p) values (1, 2);
> alter table mig add column j int;
> alter table mig convert to iceberg;
> select * from mig;
> ERROR: Migrated file
> hdfs://localhost:20500/test-warehouse/mig/p=2/2449e393e29743d0-7ce9249300000000_1145218898_data.0.parq
> has unexpected schema or partitioning.{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
