voonhous opened a new issue, #20628:
URL: https://github.com/apache/hudi/issues/20628

   Part of #20625. Line references are against master `db90a0da`.
   
   ### Problem
   For every split with log files, `HudiPageSourceProvider.createPageSource`:
   
   - builds a new `HoodieTableMetaClient` 
(`HudiPageSourceProvider.java:232-234`, has a TODO), which reads 
`hoodie.properties`, plus the index definitions file when one is registered;
   - unless `hudi.table.resolve-column-name-casing.enabled=true` (default 
false), runs `TableSchemaResolver.getTableSchema()`: a recursive timeline 
listing, at least one instant file read with full commit metadata parsing, and 
an operation-field probe that can open a log file.
   
   Nothing is cached across splits. Measured locally with 
`HudiMorReadBenchmark` scan (48 splits): `_rt` does 50 `hoodie.properties` 
reads and 97 timeline reads, against 2 and 1 for `_ro`.
   
   ### Direction
   Resolve the schema, and what the readers need from the table config, once on 
the coordinator and carry it in `HudiTableHandle` or the split always, not only 
when casing resolution is on. Stop building a meta client per split; 
`TableState` is serializable and `HoodieFileGroupReader` accepts it in place of 
a meta client.
   
   ### Exit criteria
   - `HudiMorReadBenchmark` scan `_rt` with the default session: timeline reads 
<= 2 and `hoodie.properties` reads <= 2 per query, identical at 
`-Dbench.partitions=48` and `96`.
   - No "Fetched table schema" log line on workers during a MOR query.
   - `TestHudiNoCacheFileOperations` and the cache variants updated: the 
per-split `TABLE_PROPERTIES` reads disappear from the expected multisets.
   - Schema-evolution and column-casing tests green: a schema change between 
queries is picked up by the next query.
   - Scan `_rt` wall time at the default latency profile improves against 
master.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to