LsomeYeah commented on PR #9023: URL: https://github.com/apache/paimon/pull/9023#issuecomment-5177150293
Addressed the four review findings in `2337128407`: 1. Implemented `dataSourceInputPartitions` in the Spark 4.0 shim; local Spark 4.0/4.1 reactor packaging now succeeds. 2. Fixed the empty-side multi-source ambiguity. Spark replaces unchanged sources with `LocalRelation` before `foreachBatch`, so they cannot be recovered from RDD/InputPartition lineage. The helper now uses Spark's callback query id to inspect that query's complete `StreamExecution.sources`, requires exactly one distinct Paimon source, and fails closed if the internal lookup is unavailable. Added a reproducing integration test. 3. Added an invocation-local `schemaId -> TableSchema` cache, so different sparse projections under the same schema do not reread the schema file. Added a load-count regression test. 4. Made `PaimonMicroBatchMetadata` transient on the InputPartition, preserving driver-side lookup while avoiding task serialization. Added a serialization regression test. Local verification: - Core data-evolution tests: 17 passed. - `PaimonSourceTest`: 22 passed. - Spark 3.2/3.3/3.4/3.5 compile: passed. - Spark 4.0/4.1 package with JDK 21: passed. - Spotless/checkstyle validation and `git diff --check`: passed for changed Core/Spark 3 sources; Spark 4 source was formatted and compiled in the reactor package. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
