Zoltán Borók-Nagy created IMPALA-15617:
------------------------------------------
Summary: ALTER COLUMN ... DROP DEFAULT fails on TIMESTAMP columns
(Kudu, Iceberg) and BINARY columns (Iceberg)
Key: IMPALA-15617
URL: https://issues.apache.org/jira/browse/IMPALA-15617
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Zoltán Borók-Nagy
ALTER TABLE ... ALTER COLUMN ... DROP DEFAULT fails during analysis on
TIMESTAMP columns of Kudu and Iceberg V3 tables, and on BINARY columns of
Iceberg V3 tables. It fails whether or not the column has a default, so these
defaults cannot be removed from Impala.
Repro on master:
{code:sql}
create table k (id bigint primary key, ts timestamp default now()) stored as
kudu;
alter table k alter column ts drop default;
{code}
{noformat}
AnalysisException: NULL cannot be cast to a TIMESTAMP literal.
CAUSED BY: InternalException: Error converting timestamp expression:
NullLiteral{}
{noformat}
{code:sql}
create table t (i int, ts timestamp, b binary, s string)
stored by iceberg tblproperties('format-version'='3');
alter table t alter column ts drop default;
alter table t alter column b drop default;
alter table t alter column s drop default;
{code}
{noformat}
AnalysisException: NULL cannot be cast to a TIMESTAMP literal.
CAUSED BY: InternalException: Error converting timestamp expression:
NullLiteral{}
AnalysisException: Default values are not supported for type: BINARY
Column has been altered.
{noformat}
For Kudu, the only way to drop a TIMESTAMP default from Impala is to recreate
the table. {{ALTER COLUMN ts SET DEFAULT NULL}} and {{CREATE TABLE ... (ts
TIMESTAMP DEFAULT NULL)}} fail with the same error.
For Iceberg, Impala cannot define TIMESTAMP or BINARY defaults, but other
engines can. An INSERT that omits such a column fails
(IcebergUtil.resolveWriteDefault()). The user has to specify the value in every
INSERT, because Impala cannot drop the write default.
*Analysis*
AlterTableAlterColStmt.createDropDefaultStmt() represents DROP DEFAULT as a
ColumnDef with {{DEFAULT NullLiteral}}, so ColumnDef.analyze() analyzes NULL as
a default value:
* For TIMESTAMP columns, ColumnDef.analyzeKuduOptions() converts the default to
Unix time micros with ExprUtil.utcTimestampToUnixTimeMicros() and fails for
NULL. This conversion came with IMPALA-5338. The Kudu tests (kudu_create.test
tbl_with_null_defaults, kudu_alter.test DROP DEFAULT) do not cover TIMESTAMP
columns.
* For Iceberg tables, AlterTableAlterColStmt.analyze() calls
IcebergSchemaConverter.validateIcebergDdlDefaultType() whenever
newColDef_.hasDefaultValue() is true, including for DROP DEFAULT. That method
rejects TIMESTAMP, BINARY and complex types (IMPALA-14589).
*Suggested fix*
* ColumnDef: skip the TIMESTAMP conversion for a NULL default, so that DEFAULT
NULL works on nullable TIMESTAMP columns as on other types.
* AlterTableAlterColStmt: call validateIcebergDdlDefaultType() only for SET
DEFAULT, not for DROP DEFAULT.
* Add tests for DROP DEFAULT and SET DEFAULT NULL on Kudu TIMESTAMP columns,
and for DROP DEFAULT on Iceberg TIMESTAMP and BINARY columns whose write
default was set outside Impala.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
