wwj6591812 opened a new pull request, #8914: URL: https://github.com/apache/paimon/pull/8914
### Purpose Large batch scans can enable `scan.dedicated-split-generation` to move split planning and emission from the JobManager source coordinator to a dedicated TaskManager operator. This avoids retaining and checkpointing a very large set of split assignments in the JobManager for tables with millions of files. However, the dedicated reader path did not preserve Blob conversion semantics. `ReadOperator` always wrapped rows with `FlinkRowData`, so an internal Blob value was read as `byte[]` and failed with `ClassCastException`. This prevented Blob tables from using dedicated split generation. ### Changes - Propagate the physical read type and `blob-as-descriptor` option from `FlinkSourceBuilder` through `MonitorSource` to `ReadOperator`. - Use `FlinkRowDataWithBlob` when the projected read type contains `BLOB`, `ARRAY<BLOB>`, or `MAP<*, BLOB>` fields. - Keep the existing constructors and `MonitorSource.buildSource` overload for compatibility. - Preserve the existing outer projection and field reordering behavior. ### Tests - Added `ReadOperatorBlobTest` for raw Blob bytes and serialized descriptors. - Added a Flink integration test covering `scan.dedicated-split-generation=true`, raw/descriptor modes, projection, and field reordering. - Flink 1 profile: 3 targeted tests passed, plus 7 related regression tests. - Flink 2 profile: the same 3 targeted tests passed. - Checkstyle, Spotless, and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
