LuciferYang opened a new pull request, #11050:
URL: https://github.com/apache/paimon/pull/11050
### Purpose
A SQL function created in a Paimon catalog (`CREATE FUNCTION ... RETURN` on
Spark 4.0 and 4.1) loses an explicit collation on its parameter and return
types. `SparkTypeUtils` maps every Spark `StringType` to `VARCHAR(MAX)`, so
`STRING COLLATE UTF8_LCASE` is read back as plain `STRING` and the function
compares under UTF8_BINARY. For `CREATE FUNCTION f(p STRING COLLATE UTF8_LCASE)
RETURNS BOOLEAN RETURN p = 'A'`, `SELECT f('a')` returns `false` from Paimon
and `true` from the same function created as `TEMPORARY`. A collated `RETURNS`
type is lost the same way, and so, on 4.1, is a return type inferred from the
body.
Paimon has nowhere to keep a function's string collation, so
`SQLFunctionConverter.toPaimonFunction` now rejects a parameter or return type
containing a string collated other than UTF8_BINARY, including one nested in
`ARRAY`, `MAP` or `STRUCT`. The check runs before anything is persisted or
dropped. Plain `STRING` and an explicit `COLLATE UTF8_BINARY` still work.
`SparkTypeUtils` is unchanged because table schemas go through it too.
The check lives in its own `SQLFunctionTypes` object so that
version-specific forks of `SQLFunctionConverter` can call it. #9265 adds such
forks for Spark 4.0 and 4.1.
### Tests
New cases in `PaimonSQLFunctionTestBase`, which runs in `paimon-spark-4.0`
and `paimon-spark-4.1`:
- a collated parameter type, as `STRING`, inside `ARRAY`, as a `MAP` key or
value, and inside `STRUCT`, is rejected and leaves no function behind;
- a collated `RETURNS` type is rejected, and on 4.1 so is an inferred
collated return type (4.0 requires `RETURNS`);
- `COLLATE UTF8_BINARY` is still accepted and compares as before;
- a rejected `CREATE OR REPLACE` keeps the existing function.
Without the fix the rejection cases fail with `Expected exception
java.lang.Exception to be thrown, but no exception was thrown`. Moving
`dropV1Function` ahead of the conversion in the 4.0 command fails the `CREATE
OR REPLACE` case with `FunctionNotExistException`.
### API and Format
no
### Documentation
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]