leaves12138 commented on PR #7921:
URL: https://github.com/apache/paimon/pull/7921#issuecomment-4509136688
Thanks for working on this. I think this still has a blocking issue: the
newly added engine scenarios do not pass locally.
I ran the added Flink/Spark tests on the PR head (fe99faae62c3) with Java 8:
```bash
mvn -pl paimon-flink/paimon-flink-common -DskipITs=false -Dcheckstyle.skip
-Drat.skip=true -Dspotless.check.skip=true
-Dtest=SchemaChangeITCase#testAlterAddBlobColumn test
mvn -pl paimon-spark/paimon-spark-ut -DskipITs=false -Dcheckstyle.skip
-Drat.skip=true -Dspotless.check.skip=true
-Dtest=SparkSchemaEvolutionITCase#testAlterAddBlobColumn test
```
Both fail with the same validation error:
```
java.lang.IllegalArgumentException: Field 'picture' in 'blob-field' must be
a BLOB field in table schema.
at
org.apache.paimon.schema.SchemaValidation.validateBlobFields(SchemaValidation.java:807)
```
So the intended workflow in the tests:
```sql
ALTER TABLE ... SET ('blob-field' = 'picture');
ALTER TABLE ... ADD picture BYTES/BINARY;
```
still ends up committing a normal binary column while the table option says
that `picture` is a blob field, and schema validation rejects the table.
Please fix this before merge. The engine-side ALTER ADD COLUMN path needs to
resolve the new column type using the effective table options after the
previous/current option changes, not only the options carried by the ADD COLUMN
request. In particular, Flink's `alterTable(..., newTable, tableChanges, ...)`
should probably compute blob type fields from old table options merged with the
incoming option changes, similar to what the Spark path is trying to do, and
the Spark path also needs to make sure the separate SET-then-ADD workflow sees
the updated `blob-field` option before resolving the added column type.
After the fix, please make sure the added Flink and Spark tests pass for
both `blob-field` and `blob-descriptor-field`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
