holdenk opened a new issue, #8614:
URL: https://github.com/apache/iceberg/issues/8614
### Apache Iceberg version
1.3.1 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
If you have ever used a column in write sort order (as it's called in the
Spark Iceberg DDL), attempting to drop the column will result in invalid table
metadata.
Repro:
```sql
DROP TABLE IF EXISTS local.udevelopers_sorted;
CREATE TABLE IF NOT EXISTS local.udevelopers_sorted (
username string,
firstname string,
lastname string)
USING ICEBERG;
ALTER TABLE local.udevelopers_sorted WRITE ORDERED BY lastname;
ALTER TABLE local.udevelopers_sorted RENAME COLUMN lastname TO
deprecated_lastname;
SELECT * FROM local.udevelopers_sorted;
ALTER TABLE local.udevelopers_sorted WRITE ORDERED BY username;
ALTER TABLE local.udevelopers_sorted DROP COLUMN lastname;
SELECT * FROM local.udevelopers_sorted;
```
I think the solution is to add similar handling to how Iceberg handles
partition columns & identifier columns (e.g. special drop function etc.)
As a temporary work around for folks you can make it a partition column then
drop the partition column, but this feels ugly to me.
```sql
DROP TABLE IF EXISTS local.udevelopers_sorted;
CREATE TABLE IF NOT EXISTS local.udevelopers_sorted (
username string,
firstname string,
lastname string)
USING ICEBERG;
ALTER TABLE local.udevelopers_sorted WRITE ORDERED BY lastname;
SELECT * FROM local.udevelopers_sorted;
ALTER TABLE local.udevelopers_sorted WRITE ORDERED BY username;
-- Hack, add it to identifier fields so we can do a "partial" drop where it
stays in the schema and we don't
-- corrupt the metadata.
ALTER TABLE local.udevelopers_sorted ADD PARTITION FIELD lastname;
ALTER TABLE local.udevelopers_sorted DROP PARTITION FIELD lastname;
SELECT * FROM local.udevelopers_sorted;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
