pan3793 commented on PR #25536: URL: https://github.com/apache/spark/pull/25536#issuecomment-1410660133
> Can you split your SQL CTAS to CREATE TABLE and INSERT? Yes, it's a workaround, but it's not convenient since users need to explicitly list all columns. > Or ALTER the table later to change column nullability? I'm afraid not, the CTAS failed because the generated schema violate the restriction. Back to the case mentioned in this PR description, I think table schema evolution is an essential feature of the modern data lake table format, can `INT NOT NULL` safely evolute to `INT NULL`? If yes, I think there is no problem if we reserve the nullable on CTAS/RTAS. BTW, I tested the provided CTAS case on MySQL 8, the result is different. ``` mysql> select version(); +-----------+ | version() | +-----------+ | 8.0.32 | +-----------+ 1 row in set (0.00 sec) mysql> create table t1(i int not null); Query OK, 0 rows affected (0.04 sec) mysql> insert into t1 values (1); Query OK, 1 row affected (0.01 sec) mysql> create table t2 as select i from t1; Query OK, 1 row affected (0.03 sec) Records: 1 Duplicates: 0 Warnings: 0 mysql> desc t1; +-------+------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+------+------+-----+---------+-------+ | i | int | NO | | NULL | | +-------+------+------+-----+---------+-------+ 1 row in set (0.00 sec) mysql> desc t2; +-------+------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+------+------+-----+---------+-------+ | i | int | NO | | NULL | | +-------+------+------+-----+---------+-------+ 1 row in set (0.00 sec) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
