[ 
https://issues.apache.org/jira/browse/HDDS-16783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-16783:
----------------------------------
    Labels: ozone-hbase-perf pull-request-available  (was: 
pull-request-available)

> Support FileSystem#setVerifyChecksum in ofs and o3fs
> ----------------------------------------------------
>
>                 Key: HDDS-16783
>                 URL: https://issues.apache.org/jira/browse/HDDS-16783
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Filesystem
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>              Labels: ozone-hbase-perf, pull-request-available
>             Fix For: 2.3.0
>
>
> h3. Problem
> HBase stores its own checksums in HFile blocks. When 
> {{hbase.regionserver.checksum.verify}} is true (the default), {{HFileSystem}} 
> opens a second FileSystem instance, [calls 
> {{setVerifyChecksum(false)}}|https://github.com/apache/hbase/blob/rel/2.6.7/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java#L110]
>  on it, and reads HFile blocks through that instance. On HDFS this skips the 
> file system checksum.
> {{BasicOzoneFileSystem}} and {{BasicRootedOzoneFileSystem}} do not override 
> {{{}setVerifyChecksum{}}}, so the call reaches the empty default in 
> {{FileSystem}} and the Ozone client still verifies every read. This costs 
> HBase twice:
>  * the data is checksummed by both Ozone and HBase
>  * with verification on, each read is widened to checksum boundaries 
> ({{{}ozone.client.bytes.per.checksum{}}}), so with a 1MB value a 64KB HFile 
> block read fetches and checksums 1MB
> {{ozone.client.verify.checksum=false}} is not a good substitute. It applies 
> to the whole process, including the FileSystem instance HBase falls back to 
> when an HFile checksum fails, so that retry would no longer be verified by 
> anything.
> h3. Proposal
> Override {{setVerifyChecksum}} in both file systems and pass the flag through 
> {{OzoneClientAdapter}} and {{ClientProtocol}} to the client's own 
> {{{}OzoneClientConfig{}}}, so that input streams opened afterwards by that 
> FileSystem instance skip checksum verification. The setting is per instance, 
> the same as {{{}DistributedFileSystem{}}}.
> No protocol change is needed. The existing read paths already read this flag 
> when a stream is opened: ReadChunk, short-circuit read, and streaming 
> ReadBlock, which passes it to the datanode as {{includeChecksums}} 
> (HDDS-16258).
> h3. Expected gain
> 64KB random positional reads ({{{}readFully(pos, buf, 0, len){}}} on one 
> shared input stream per file), 16 threads, operations per second, median of 
> three rounds. "Verify off" was measured with 
> {{{}ozone.client.verify.checksum=false{}}}, which is what the API call would 
> toggle per instance.
> ||Read path||Verify on||Verify off||
> |gRPC, 1MB bytes per checksum|6,100|22,600|
> |gRPC, 16KB bytes per checksum|22,100|22,600|
> |Short-circuit read, 1MB bytes per checksum|38,500|937,800|
> |Short-circuit read, 16KB bytes per checksum|534,900|911,400|
> Environment:
>  * Ozone 2.3.0-SNAPSHOT, master at 24cf9aaa28e, Hadoop 3.4.3, Ratis 3.3.1
>  * MiniOzoneCluster with 3 datanodes in one JVM ({{{}-Xmx8192m{}}}, G1), 16 
> files of 64MB, RATIS/THREE
>  * {{apache/ozone-runner:20260626-1-jdk25}} container (Rocky Linux 9.8, 
> OpenJDK 25.0.2), with native libhadoop for Hadoop 3.4.3 (linux aarch64) for 
> short-circuit read
>  * Docker Desktop Linux VM on arm64 with 14 CPUs and 23.4 GB, on an Apple M4 
> Pro host (14 cores, 48 GB)
>  * 3 s warm up and 8 s per measurement, host at least 89% idle at the start 
> of each run
>  * Single machine, loopback network, all data in the page cache, security 
> off. Treat the figures as an upper bound.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to