[
https://issues.apache.org/jira/browse/IMPALA-15616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arnab Karmakar resolved IMPALA-15616.
-------------------------------------
Fix Version/s: Impala 5.1.0
Resolution: Fixed
> Iceberg V3: DEFAULT NULL on a nullable column is stored as the string 'NULL'
> or fails in catalogd
> -------------------------------------------------------------------------------------------------
>
> Key: IMPALA-15616
> URL: https://issues.apache.org/jira/browse/IMPALA-15616
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Arnab Karmakar
> Priority: Major
> Labels: impala-iceberg
> Fix For: Impala 5.1.0
>
>
> On Iceberg V3 tables, DEFAULT NULL or DEFAULT CAST(NULL AS <type>) on a
> nullable column does not mean SQL NULL. Impala sends the default to catalogd
> as the text 'NULL', and catalogd parses that text as a value of the column
> type. This affects CREATE TABLE, ALTER TABLE ADD COLUMN(S) and ALTER TABLE
> ALTER COLUMN SET DEFAULT:
> * STRING columns get the 4-character string 'NULL' as their default, with no
> error (wrong results).
> * Other types pass analysis, then fail in catalogd with a raw parse error.
> Repro on master, STRING column:
> {code:sql}
> create table t (i int, s string default null) stored by iceberg
> tblproperties('format-version'='3');
> insert into t (i) values (1);
> insert into t values (2, null);
> select i, s, s is null, length(s) from t order by i;
> {code}
> {noformat}
> 1 NULL false 4
> 2 NULL true NULL
> {noformat}
> Expected for row 1: NULL, true, NULL. The current-schema property in DESCRIBE
> FORMATTED t has the string as the default (quotes unescaped):
> {noformat}
> {"id":2,"name":"s","required":false,"type":"string","initial-default":"NULL","write-default":"NULL"}
> {noformat}
> ALTER TABLE gives the same result:
> {code:sql}
> create table u (i int) stored by iceberg tblproperties('format-version'='3');
> insert into u values (1);
> alter table u add column s string default null;
> alter table u add columns (s2 string default cast(null as string));
> alter table u add column s3 string default 'x';
> alter table u alter column s3 set default null;
> insert into u (i) values (2);
> select i, s, s is null, s2, s2 is null, s3, s3 is null from u order by i;
> {code}
> {noformat}
> 1 NULL false NULL false x false
> 2 NULL false NULL false NULL false
> {noformat}
> Expected: s and s2 are NULL in both rows (IS NULL true), and s3 is 'x' in row
> 1 and NULL in row 2.
> INT, BIGINT, DECIMAL, DOUBLE, BOOLEAN and DATE pass analysis but fail in
> catalogd. For example:
> {code:sql}
> create table t2 (i int default null) stored by iceberg
> tblproperties('format-version'='3');
> {code}
> {noformat}
> ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:
> CAUSED BY: NumberFormatException: Character N is neither a decimal digit
> number, decimal point, nor "e" notation exponential mark.
> {noformat}
> {code:sql}
> alter table u add column b boolean default null;
> {code}
> {noformat}
> ImpalaRuntimeException: Failed to parse default value 'NULL' for column 'b':
> Cannot parse default value 'NULL' as BOOLEAN: invalid BOOLEAN literal: NULL
> {noformat}
> {code:sql}
> alter table u alter column i set default cast(null as int);
> {code}
> {noformat}
> ImpalaRuntimeException: Failed to ALTER table 'u': Character N is neither a
> decimal digit number, decimal point, nor "e" notation exponential mark.
> {noformat}
> Only a NULL default on a NOT NULL column is rejected in analysis:
> {code:sql}
> create table t5 (s string not null default null) stored by iceberg
> tblproperties('format-version'='3');
> {code}
> {noformat}
> AnalysisException: Default value of NULL not allowed on non-nullable column:
> 's'
> {noformat}
> *Analysis*
> ColumnDef.analyze() (in analyzeKuduOptions(), which also handles Iceberg
> defaults) keeps the NullLiteral as the analyzed default and rejects it only
> on NOT NULL or primary key columns. CreateTableStmt.toThrift(),
> AlterTableAddColsStmt.toThrift() and AlterTableAlterColStmt.toThrift() send
> ColumnDef.getIcebergDefaultValueString() to catalogd as
> iceberg_initial_default and iceberg_write_default. For a NullLiteral, this
> returns NullLiteral.getStringValue(), which is the text "NULL". In catalogd,
> IcebergSchemaConverter.parseStringToIcebergLiteral() parses that text with
> LiteralExpr.createFromUnescapedStr(). It is called from
> IcebergSchemaConverter.createIcebergField(),
> IcebergCatalogOpExecutor.addColumns() and
> IcebergCatalogOpExecutor.alterColumn(). For STRING the parse returns
> StringLiteral("NULL"). For the other types it throws. The text form cannot
> tell a NULL default from the string 'NULL'.
> *Suggested fix*
> Handle a NULL default during analysis, so that a NullLiteral is never turned
> into text. There are two options:
> # (preferred) Treat DEFAULT NULL or CAST(NULL AS <type>) on a nullable
> Iceberg column as "no default". CREATE TABLE and ADD COLUMN(S) would then set
> neither iceberg_initial_default nor iceberg_write_default. ALTER COLUMN SET
> DEFAULT NULL would set iceberg_drop_write_default, so it does the same as
> DROP DEFAULT. Returning null from getIcebergDefaultValueString() is not
> enough: with iceberg_write_default unset, SET DEFAULT NULL would do nothing.
> This option matches Iceberg, where a field without initial-default or
> write-default has a null default. It also matches Kudu, which accepts DEFAULT
> NULL on nullable columns (kudu_create.test).
> # Reject a NULL default with an AnalysisException in the Iceberg DDL paths,
> next to the IcebergSchemaConverter.validateIcebergDdlDefaultType() calls.
> *Testing*
> Add FE analysis tests and an EE test for CREATE TABLE, ADD COLUMN(S) and
> ALTER COLUMN SET DEFAULT with NULL and CAST(NULL AS <type>), on a STRING
> column and on a non-STRING column. Check IS NULL on rows written before the
> column was added and on INSERTs that omit the column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
