taklwu commented on PR #8851: URL: https://github.com/apache/hbase/pull/8851#issuecomment-6802801307
I used Cursor and added a benchmark test code in my private branch to see how much we can save with mini HDFS - It starts a mini HDFS cluster, creates a region with 10, 50 or 100 store files, and times only the region open. It does one warm-up open, then averages 5. - getFileInfo and getBlockLocations counts come from the NameNode's own metrics, so they include calls the HDFS client makes internally. Client-side getFileStatus calls are counted through a wrapper filesystem. - The wrapper can also add a delay to each getFileStatus, to model a slow NameNode or an object-store HEAD request. <img width="941" height="162" alt="Screenshot 2026-11-11 at 9 26 25 AM" src="https://github.com/user-attachments/assets/9972bb46-e1cb-42ae-a518-24ea7ff0b1f4"; /> getFileInfo in HDFS is cheap for the NameNode and only needs 1ms or less. But for Ozone (OFS) and for object stores, this is where it matters. Metadata calls there cost several milliseconds (Ozone Manager calls) to tens of milliseconds (S3 HEAD requests). If my understanding is correct, current cache does not help for this filesystem metadata -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
