voonhous opened a new issue, #20632:
URL: https://github.com/apache/hudi/issues/20632

   Part of #20625. Line references are against master `db90a0da`.
   
   ### Problem
   `HudiSplit` carries each log file's size, but `HudiUtil.convertToFileSlice` 
(`HudiUtil.java:547`) rebuilds each log as `new HoodieLogFile(path)` with size 
-1, so every later size lookup goes to storage. For table version 10 native log 
files (`<file>.log.parquet`) this means, per log file: several `getPathInfo` 
sequences (`directoryExists` + `exists` + `length` + `lastModified`), a footer 
read for the block header, and a second footer read plus a length probe when 
the connector's page source opens the file. A base-only split pays one footer 
read and no probes.
   
   ### Direction
   Find the calls that repeat `length()`/`exists()`/listing on the same log 
file (the native log reader, `FSUtils.getFileSize`, 
`HudiTrinoStorage.getPathInfo`) and reuse the size and status already known 
from the file slice.
   
   ### Exit criteria
   - `HudiMorReadBenchmark` scan with `-Dbench.nativeLogs=true`: <= 1 HEAD and 
0 LIST per log file (total scan `_rt` HEAD <= number of base + log files; LIST 
limited to partition and timeline listings).
   - Scan `_rt` wall time with native logs within 1.3x of the legacy-log run at 
the same latency profile.
   - Correctness asserts pass with native logs; existing native-log tests green.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to