[
https://issues.apache.org/jira/browse/IMPALA-13273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18080434#comment-18080434
]
ASF subversion and git services commented on IMPALA-13273:
----------------------------------------------------------
Commit 0b2abb171de9ebac669688b72239f144b4bd8bf6 in impala's branch
refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0b2abb171 ]
IMPALA-13273: Disallow writing NULL values in non-nullable columns
Before this patch we didn't enforce NOT NULL constraints during
writing. This caused issues with Iceberg tables with non-nullable
columns. E.g.:
create table t_ice_constr(c1 int not null) stored as iceberg;
insert into t_ice_constr select null;
select c1 from t_ice_constr;
The above select returned a value instead of NULL, because the slot
descriptor associated with column 'c1' was not nullable, so didn't
even have a null indicator bit.
The fix is to forbid writing NULLs in non-nullable columns in the
first place. This is now enforced in the Parquet writer's
FinalizeCurrentPage() function where we have statistics about the
number of NULLs written.
Schema evolution concerns
* Iceberg allows making a required column optional (via
UpdateSchema.makeColumnOptional()) This is a compatible change,
because if a reader expects optional values then it is not a problem
if the values are always there in the data files.
* Iceberg has UpdateSchema.requireColumn(), but it is only allowed
if users call allowIncompatibleChanges() as well, as it can break
reading older data.
* Iceberg also has UpdateSchema.addRequiredColumn() but users should
also set a default value to not break readers. Iceberg only allows
adding new required columns without default values if they explicitly
call allowIncompatibleChanges().
* Iceberg says users should only call allowIncompatibleChanges() if
they have validated that all of their old data files are compatible.
E.g. if a column was optional (nullable), but all old data files
contain values for that column.
Testing
* e2e tests added
Change-Id: I1189da3094beee615a5d4600576febde4be8473d
Reviewed-on: http://gerrit.cloudera.org:8080/24141
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Inconsistent handling of NOT NULL fields in iceberg tables
> ----------------------------------------------------------
>
> Key: IMPALA-13273
> URL: https://issues.apache.org/jira/browse/IMPALA-13273
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.4.0
> Reporter: Aleksey
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> Since [IMPALA-12729|https://issues.apache.org/jira/browse/IMPALA-12729] has
> been introduced, the following scenario occurs:
>
> {code:java}
> -- step 1 - create iceberg table without pk, but with not null
> create table t_ice_constr(c1 int not null) stored as iceberg;
> -- step 2 - insert success, parquet file contains null
> insert into t_ice_constr select null;
> -- step 3 - select returns 0
> select c1 from t_ice_constr;
> -- step 4 - select returns no rows
> select c1 from t_ice_constr where c1 = 0;
> -- step 5 - select return no rows
> select c1 from t_ice_constr where c1 is null;
> {code}
>
> It seems to be an inconsistency here. Expected results - either 1 row on step
> 4, or null in step 3 and 1 row on step 5.
> This behavior is supposedly caused by this change:
> [https://github.com/apache/impala/commit/73171cb7164573349bd53a996a51bb7058b778e0#diff-adcc0990c7301e45752d00f6d8e65b192bfadffe179056037a7e14aff8718856R180]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
