voonhous commented on issue #20628:
URL: https://github.com/apache/hudi/issues/20628#issuecomment-6822382673

   Findings so far (local; fix is on a local branch, PR to follow).
   
   **Per-split cost on master, beyond the issue description.**
   - The schema resolver's operation-field probe reads the schema of a file 
from the latest commit: for a parquet base file that is an exception and an 
INFO log line per split with no I/O; for a legacy Avro log file it opens the 
file and walks every block header.
   - With `hudi.table.resolve-column-name-casing.enabled=true` the per-split 
schema work goes away but the per-split meta client stays. On tables before 
version 8, log block commit validation then loads the commits timeline per 
split through the meta-client-backed `TableState`, so the flag does not remove 
the timeline listing there.
   - That flag also changes predicate key mapping in index pruning 
(`IndexSupportFactory`), so it is not a pure performance switch.
   
   **Fix measured.** The coordinator resolves the table schema once per query 
for MERGE_ON_READ handles and ships it in the handle along with the table 
config map (without the create schema) and, for tables before version 8, the 
committed instants. Workers build a `TableState` from it and no longer build a 
meta client. Index pruning still checks the casing flag, so its behavior is 
unchanged.
   
   `HudiMorReadBenchmark` scan `_rt`, default session, default latency profile 
(20/15/40 ms), 48 partitions:
   
   | | master | fix |
   |---|---|---|
   | `hoodie.properties` reads | 50 | 2 |
   | timeline reads | 97 | 2 |
   | log file GETs | 432 | 385 |
   | total storage requests | 723 | 533 |
   | wall | 4.22 s | 3.59 s |
   
   The request counts are identical at 96 partitions. "Fetched table schema" is 
logged once per query, on a coordinator thread.
   
   Handle size on a 40-column table: about 1.2 KB for the config map, plus the 
schema JSON (about 5.9 KB), which was already shipped when casing resolution 
was on.
   
   Behavior changes to review: schema resolution now runs once per MOR snapshot 
query on the coordinator even when no split has log files, and a resolution 
failure surfaces from handle serialization instead of from each split.
   
   The existing file-operation tests only query read-optimized tables, which 
never reach the merge path, so they had no per-split reads to remove; 
snapshot-read cases were added instead (version 8 table with 3 file groups: 
`TABLE_PROPERTIES` 5 to 2, log opens 9 to 7).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to