voonhous opened a new issue, #20636:
URL: https://github.com/apache/hudi/issues/20636

   Part of #20625. Line references are against master `db90a0da`.
   
   ### Problem
   `HudiPageSource.getMemoryUsage`, `getCompletedBytes` and `getReadTimeNanos` 
delegate only to the base file page source (`HudiPageSource.java:91-107`, 
`128-132`). The output page builder, the spillable merge map (up to about 80% 
of `hoodie.memory.merge.max.size`, 1 GiB by default, per split) and all log 
file reads are invisible to Trino's memory accounting and query stats. Trino 
cannot enforce memory limits on this path, and MOR physical input bytes and 
read time are understated.
   
   ### Direction
   Report the merge map's in-memory size and the page builder in 
`getMemoryUsage`, and count log file bytes and read time in `getCompletedBytes` 
/ `getReadTimeNanos`.
   
   ### Exit criteria
   - A test showing `getMemoryUsage` grows with buffered log records and 
`getCompletedBytes` includes log file bytes.
   - `HudiMorReadBenchmark` scan `_rt` physical input reflects base plus log 
bytes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to