voonhous commented on issue #19231: URL: https://github.com/apache/hudi/issues/19231#issuecomment-5164533965
A note on scope after digging into the migrated module: `HudiReadOptimizedDirectoryLister` and `MIN_BLOCK_SIZE` never made it into `hudi-trino`. They were dropped in the RFC-105 migration (#18837, c3c936790727) when the connector was rewritten around `FileSlice` with `HudiSnapshotDirectoryLister` and the filegroup reader (MOR support), so the fix site this issue describes does not exist in this repo. The class only survives as prose in rfc-105.md. The blockSize=0 root cause did get migrated though. `HudiTrinoStorage.convertToPathInfo` (and `getPathInfo`) still hardcode 0, and the sole consumer today is `HudiSplitFactory`, which sizes base file splits with `max(target_split_size, blockSize)`. With blockSize always 0 (direct listing) or 32MB (metadata table listing via `getDefaultBlockSize`), that floor is inert and slicing is governed by `hudi.target-split-size` (default 128MB). So the 4x inflation from trinodb/trino#29842 does not reproduce here: a ~120MB file already yields one split, and only files above ~141MB (128MB x 1.1 split slop) get sliced, which is essentially upstream's post-fix behavior (they cap at a configurable 120MB). Re-scoping the port accordingly: - `HudiTrinoStorage` reports the file length as the block size (upstream parity at the storage layer). - `HudiSplitFactory` slices solely by `target_split_size`. Porting blockSize=length while keeping the old `max()` would have silently disabled the knob and made every file a single split; dropping the `max()` keeps the config honest and preserves scan parallelism on large base files. Anyone wanting upstream's strict one-split-per-file behavior can raise `target_split_size`. - Regression tests: one split for a ~120MB file at the default target, target-boundary slicing above it, and blockSize == length assertions on the storage layer. PR coming up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
