LuciferYang opened a new pull request, #11050:
URL: https://github.com/apache/paimon/pull/11050

   ### Purpose
   
   A SQL function created in a Paimon catalog (`CREATE FUNCTION ... RETURN` on 
Spark 4.0 and 4.1) loses an explicit collation on its parameter and return 
types. `SparkTypeUtils` maps every Spark `StringType` to `VARCHAR(MAX)`, so 
`STRING COLLATE UTF8_LCASE` is read back as plain `STRING` and the function 
compares under UTF8_BINARY. For `CREATE FUNCTION f(p STRING COLLATE UTF8_LCASE) 
RETURNS BOOLEAN RETURN p = 'A'`, `SELECT f('a')` returns `false` from Paimon 
and `true` from the same function created as `TEMPORARY`. A collated `RETURNS` 
type is lost the same way, and so, on 4.1, is a return type inferred from the 
body.
   
   Paimon has nowhere to keep a function's string collation, so 
`SQLFunctionConverter.toPaimonFunction` now rejects a parameter or return type 
containing a string collated other than UTF8_BINARY, including one nested in 
`ARRAY`, `MAP` or `STRUCT`. The check runs before anything is persisted or 
dropped. Plain `STRING` and an explicit `COLLATE UTF8_BINARY` still work. 
`SparkTypeUtils` is unchanged because table schemas go through it too.
   
   The check lives in its own `SQLFunctionTypes` object so that 
version-specific forks of `SQLFunctionConverter` can call it. #9265 adds such 
forks for Spark 4.0 and 4.1.
   
   ### Tests
   
   New cases in `PaimonSQLFunctionTestBase`, which runs in `paimon-spark-4.0` 
and `paimon-spark-4.1`:
   
   - a collated parameter type, as `STRING`, inside `ARRAY`, as a `MAP` key or 
value, and inside `STRUCT`, is rejected and leaves no function behind;
   - a collated `RETURNS` type is rejected, and on 4.1 so is an inferred 
collated return type (4.0 requires `RETURNS`);
   - `COLLATE UTF8_BINARY` is still accepted and compares as before;
   - a rejected `CREATE OR REPLACE` keeps the existing function.
   
   Without the fix the rejection cases fail with `Expected exception 
java.lang.Exception to be thrown, but no exception was thrown`. Moving 
`dropV1Function` ahead of the conversion in the 4.0 command fails the `CREATE 
OR REPLACE` case with `FunctionNotExistException`.
   
   ### API and Format
   
   no
   
   ### Documentation
   
   no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to