LuciferYang opened a new pull request, #10397: URL: https://github.com/apache/paimon/pull/10397
### Purpose Every Paimon table-valued function fails on Spark 4.0 with `NoSuchMethodError: DataSourceV2Relation$.create$default$5()`, in 2.0.0 and on master. The spark4 profile compiles `paimon-spark-common` against Spark 4.1.2, where `DataSourceV2Relation.create` has a fifth `timeTravelSpec` parameter with a default, so the four-argument calls in `PaimonTableValuedFunctions` compile to `create$default$5()` plus the five-argument overload. Spark 4.0 has neither. This adds `SparkShim.createDataSourceV2Relation`. `Spark3Shim`, the `paimon-spark4-common` `Spark4Shim` and the `paimon-spark-4.0` `Spark4Shim` each implement it with the same four-argument `create` call, compiled against their own Spark. The five call sites in `PaimonTableValuedFunctions` now go through it. On Spark 3.x and 4.1 the emitted call is the same as before. The three-argument `create` calls elsewhere exist on every supported version and are left alone. This closes #10396. ### Tests `TableValuedFunctionsCompatibilityTest` in `paimon-spark-4.0` runs each of the five call sites on a Spark 4.0 runtime: - `paimon_incremental_query` on its scan path, and on the Spark-side diff it takes when the tags' bucket counts differ (the case asserts the `Except`). - `vector_search`, `hybrid_search` and `full_text_search`. These are only analyzed and checked to resolve to their search table, since the relation is built during analysis. The shared `TableValuedFunctionsTest` does not run in `paimon-spark-4.0`, which is why this went unnoticed. Before the fix the new suite aborts with the `NoSuchMethodError`. With the four call sites other than the plain incremental one reverted to the four-argument `create`, each matching case fails on its own while the plain incremental case still passes. ### API and Format No. ### Documentation No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
