voonhous opened a new issue, #19231:
URL: https://github.com/apache/hudi/issues/19231
Subtask of #18780 (RFC-105 hudi-trino migration).
### What
Port the fix for trinodb/trino#29842 into the Hudi-side connector module, so
the migrated `hudi-trino` artifact does not re-ship the bug.
`HudiTrinoStorage.convertToPathInfo` (Hudi side:
`hudi-trino/src/main/java/io/trino/plugin/hudi/storage/HudiTrinoStorage.java`,
same code as Trino-side `TrinoHudiStorage`) hardcodes `blockSize = 0` when
converting a `FileEntry` to `StoragePathInfo`:
```java
return new StoragePathInfo(
convertToPath(fileEntry.location()),
fileEntry.length(),
false,
(short) 0,
0, // <- block size hardcoded to 0
fileEntry.lastModified().toEpochMilli());
```
Downstream, `HudiReadOptimizedDirectoryLister.listStatus` computes
`max(blockSize, min(fileLength, MIN_BLOCK_SIZE))`, which collapses to the 32MB
`MIN_BLOCK_SIZE` fallback, and `HudiSplitFactory` then slices every large file
at 32MB boundaries. On S3 (no real block boundaries) a ~120MB parquet file
yields 4 splits instead of 1; trinodb/trino#29842 measured ~4x split inflation
(5,834 -> 23,411 splits on a ~6000-file table) with compounding physical-IO
overhead from extra GetObject calls and redundant footer reads.
### Fix (per the upstream issue)
- Pass `fileEntry.length()` as the block size in `convertToPathInfo` (1
split per file on object stores).
- The `max/min` guard and `MIN_BLOCK_SIZE` in the directory lister then
simplify to `fileEntry.getLength()` and the constant can be removed.
- Add/port a regression test asserting one split per large file.
### Why here
Under the RFC-105 ownership flip, connector internals (including this class)
live in `org.apache.hudi:hudi-trino`; fixes reported against the Trino-side
plugin must land in this repo to reach users of the shim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
