ssandona opened a new issue, #13526:
URL: https://github.com/apache/iceberg/issues/13526
### Apache Iceberg version
1.7.1
### Query engine
Spark
### Please describe the bug 🐞
When running a `ALTER TABLE... WRITE LOCALLY ORDERED BY` statement it does
not set the `write.distribution-mode` table property to `none` but it only set
the `sort-order` property.
## Replicate steps
### 1. Create an Iceberg table and check default table properties
```
spark.sql(f"""
CREATE TABLE IF NOT EXISTS {ICEBERG_DB}.{ICEBERG_TABLE_SORT} (
drone_id STRING,
current_lat DOUBLE,
current_lon DOUBLE,
current_height INT,
current_order_id STRING,
direction INT,
loaded INT,
package_delivered INT,
timestamp TIMESTAMP,
warehouse_id STRING
)
USING ICEBERG
PARTITIONED BY (warehouse_id)
LOCATION '{ICEBERG_DB_LOCATION}{ICEBERG_TABLE_SORT}'
""")
```
Check table properties
```
spark.sql(f"SHOW TBLPROPERTIES
{ICEBERG_DB}.{ICEBERG_TABLE_SORT}").show(truncate=False)
+-------------------------------+---------------+
|key |value |
+-------------------------------+---------------+
|current-snapshot-id |none |
|format |iceberg/parquet|
|format-version |2 |
|write.parquet.compression-codec|zstd |
+-------------------------------+---------------+
```
### 2. Change sort order with WRITE LOCALLY ORDERED BY
```
spark.sql(f"""
ALTER TABLE {ICEBERG_DB}.{ICEBERG_TABLE_SORT} WRITE LOCALLY ORDERED BY
drone_id
"""
)
```
Check table properties
```
spark.sql(f"SHOW TBLPROPERTIES
{ICEBERG_DB}.{ICEBERG_TABLE_SORT}").show(truncate=False)
+-------------------------------+------------------------+
|key |value |
+-------------------------------+------------------------+
|current-snapshot-id |none |
|format |iceberg/parquet |
|format-version |2 |
|sort-order |drone_id ASC NULLS FIRST|
|write.parquet.compression-codec|zstd |
+-------------------------------+------------------------+
```
My expectation was to see a property called `write.distribution-mode` to be
added with a value of `none`. Instead only the `sort-order` property was added.
### 3. Change sort order with WRITE DISTRIBUTED BY PARTITION LOCALLY ORDERED
BY
```
spark.sql(f"""
ALTER TABLE {ICEBERG_DB}.{ICEBERG_TABLE_SORT} WRITE DISTRIBUTED BY PARTITION
LOCALLY ORDERED BY drone_id
"""
)
```
Check table properties
```
spark.sql(f"SHOW TBLPROPERTIES
{ICEBERG_DB}.{ICEBERG_TABLE_SORT}").show(truncate=False)
+-------------------------------+------------------------+
|key |value |
+-------------------------------+------------------------+
|current-snapshot-id |none |
|format |iceberg/parquet |
|format-version |2 |
|sort-order |drone_id ASC NULLS FIRST|
|write.distribution-mode |hash |
|write.parquet.compression-codec|zstd |
+-------------------------------+------------------------+
```
I see as expected `write.distribution-mode` set to `hash` and the proper
`sort-order`.
### 4. Change sort order with WRITE ORDERED BY
```
spark.sql(f"""
ALTER TABLE {ICEBERG_DB}.{ICEBERG_TABLE_SORT} WRITE ORDERED BY drone_id
"""
)
```
Check table properties
```
spark.sql(f"SHOW TBLPROPERTIES
{ICEBERG_DB}.{ICEBERG_TABLE_SORT}").show(truncate=False)
+-------------------------------+------------------------+
|key |value |
+-------------------------------+------------------------+
|current-snapshot-id |none |
|format |iceberg/parquet |
|format-version |2 |
|sort-order |drone_id ASC NULLS FIRST|
|write.distribution-mode |range |
|write.parquet.compression-codec|zstd |
+-------------------------------+------------------------+
```
I see as expected `write.distribution-mode` set to `range` and the proper
`sort-order`.
### 4. Try again to change sort order with WRITE LOCALLY ORDERED BY
```
spark.sql(f"""
ALTER TABLE {ICEBERG_DB}.{ICEBERG_TABLE_SORT} WRITE LOCALLY ORDERED BY
drone_id
"""
)
```
Check table properties
```
spark.sql(f"SHOW TBLPROPERTIES
{ICEBERG_DB}.{ICEBERG_TABLE_SORT}").show(truncate=False)
+-------------------------------+------------------------+
|key |value |
+-------------------------------+------------------------+
|current-snapshot-id |none |
|format |iceberg/parquet |
|format-version |2 |
|sort-order |drone_id ASC NULLS FIRST|
|write.distribution-mode |range |
|write.parquet.compression-codec|zstd |
+-------------------------------+------------------------+
```
I see the `write.distribution-mode` remains as `range` and is not set to
`none`.
So to fix we also have to run this
```
spark.sql(f"""
ALTER TABLE {ICEBERG_DB}.{ICEBERG_TABLE_SORT} SET TBLPROPERTIES (
'write.distribution-mode' = 'none'
);
""")
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
