LuciferYang opened a new issue, #10396: URL: https://github.com/apache/paimon/issues/10396
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found no similar issues. ### Paimon version 2.0.0. master is affected as well. ### Compute Engine Spark 4.0 (checked against 4.0.3) with `paimon-spark-4.0`. ### Minimal reproduce step With the Paimon SQL extensions enabled: ```sql CREATE TABLE t (id INT) USING paimon; INSERT INTO t VALUES 1; INSERT INTO t VALUES 2; SELECT * FROM paimon_incremental_query('t', 1, 2); ``` ### What doesn't meet your expectations? The query fails during analysis: ``` java.lang.NoSuchMethodError: 'scala.Option org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation$.create$default$5()' ``` Every Paimon table-valued function hits it: `paimon_incremental_query`, `paimon_incremental_between_timestamp`, `paimon_incremental_to_auto_tag`, `vector_search`, `hybrid_search` and `full_text_search`. ### Anything else? Since #7648 the spark4 profile compiles `paimon-spark-common` against Spark 4.1.2, and `paimon-spark-4.0` ships those classes to Spark 4.0. Spark 4.1 added a fifth `timeTravelSpec` parameter with a default to `DataSourceV2Relation.create`, so the four-argument calls in `PaimonTableValuedFunctions` compile to `create$default$5()` plus the five-argument overload. Spark 4.0 has neither. `javap` on the released `paimon-spark-4.0_2.13-2.0.0.jar` shows five such call sites in `PaimonTableValuedFunctions$`. The shared `TableValuedFunctionsTest` only runs in `paimon-spark-ut`, against the newest Spark in the profile, so CI never runs these functions on 4.0. 1.4.x compiled the shared module against Spark 4.0.2 and does not emit these calls. ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
