ChenSammi opened a new pull request, #7236: URL: https://github.com/apache/ozone/pull/7236
## What changes were proposed in this pull request? support short-circuit read for datanode. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-10685 ## How was this patch tested? new unit tests - TestDomainSocketFactory - TestXceiverServerDomainSocket - TestShortCircuitChunkInputStream manual test with freon ``` sammi@SAMMICHEN-MB0 ozone-1.5.0-SNAPSHOT % bin/ozone --jvmargs -Djava.library.path=/Users/sammi/workspace/hadoop-ozone/hadoop-hdds/client/src/test/resources freon rk --num-of-volumes=1 --num-of-buckets=10 --num-of-keys=1000 --replication=ONE --replication-type=RATIS --validate-writes --validate-channel short-circuit --num-of-validate-threads=10 --key-size 1MB SLF4J(W): Class path contains multiple SLF4J providers. SLF4J(W): Found provider [org.slf4j.reload4j.Reload4jServiceProvider@18ef96] SLF4J(W): Found provider [org.slf4j.simple.SimpleServiceProvider@6956de9] SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J(I): Actual provider is of type [org.slf4j.reload4j.Reload4jServiceProvider@18ef96] 2024-09-24 15:37:01,844 [main] INFO impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties 2024-09-24 15:37:01,877 [main] INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2024-09-24 15:37:01,877 [main] INFO impl.MetricsSystemImpl: ozone-freon metrics system started 2024-09-24 15:37:02,469 [main] INFO storage.DomainSocketFactory: Trying to load the custom-built native-hadoop library... 2024-09-24 15:37:02,469 [main] INFO storage.DomainSocketFactory: Loaded the native-hadoop library 2024-09-24 15:37:02,469 [main] INFO storage.DomainSocketFactory: short-circuit local reads is enabled within 294209 ns. 2024-09-24 15:37:02,516 [main] INFO freon.RandomKeyGenerator: Number of Threads: 10 2024-09-24 15:37:02,524 [main] INFO freon.RandomKeyGenerator: Number of Volumes: 1. 2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Number of Buckets per Volume: 10. 2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Number of Keys per Bucket: 1000. 2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Key size: 1048576 bytes 2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Buffer size: 4096 bytes 2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: validateWrites : true 2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: Number of Validate Threads: 10 2024-09-24 15:37:02,525 [main] INFO freon.RandomKeyGenerator: cleanObjects : false 2024-09-24 15:37:02,527 [main] INFO freon.RandomKeyGenerator: Data validation is enabled. 2024-09-24 15:37:02,527 [main] INFO freon.RandomKeyGenerator: Starting progress bar Thread. 0.00% |█ | 0/10000 Time: 0:00:00| 2024-09-24 15:37:02,550 [pool-3-thread-1] INFO rpc.RpcClient: Creating Volume: vol-0-30931, with sammi as owner and space quota set to -1 bytes, counts quota set to -1 2024-09-24 15:37:02,616 [pool-3-thread-8] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-6-04589, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,616 [pool-3-thread-2] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-0-10826, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,616 [pool-3-thread-5] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-3-67064, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,616 [pool-3-thread-7] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-5-65340, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,616 [pool-3-thread-1] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-9-04405, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,616 [pool-3-thread-4] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-2-22496, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,619 [pool-3-thread-3] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-1-81033, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,619 [pool-3-thread-6] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-4-66996, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,621 [pool-3-thread-10] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-8-83075, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,626 [pool-3-thread-9] INFO rpc.RpcClient: Creating Bucket: vol-0-30931/bucket-7-63535, with server-side default bucket layout, sammi as owner, Versioning false, Storage Type set to DISK and Encryption set to false, Replication Type set to server-side default replication type, Namespace Quota set to -1, Space Quota set to -1 2024-09-24 15:37:02,817 [pool-3-thread-6] WARN impl.MetricsSystemImpl: ozone-freon metrics system already initialized! 2024-09-24 15:37:03,022 [pool-3-thread-4] INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.dropwizard3.Dm3MetricRegistriesImpl 0.10% |█ | 10/10000 Time: 0:00:01| 2024-09-24 15:37:03,773 [pool-5-thread-7] INFO scm.XceiverClientShortCircuit: Pipeline-df315eb9-30ee-4765-9e68-7fc16335ce48-XceiverClientShortCircuit is created 2024-09-24 15:37:03,775 [pool-5-thread-7] INFO storage.DomainSocketFactory: DomainSocket(fd=311,path=/Users/sammi/ozone_dn_socket) is created within 1572500 ns 100.00% |█████████████████████████████████████████████████████████████████████████████████████████████████████| 10000/10000 Time: 0:00:24| 2024-09-24 15:37:27,539 [main] INFO freon.RandomKeyGenerator: Data generation is completed 2024-09-24 15:37:27,539 [main] INFO freon.RandomKeyGenerator: Data validation is completed 2024-09-24 15:37:31,231 [Pipeline-df315eb9-30ee-4765-9e68-7fc16335ce48-XceiverClientShortCircuit-ReceiveResponse] INFO scm.XceiverClientShortCircuit: receiveResponseTask is closed with java.nio.channels.ClosedChannelException 2024-09-24 15:37:31,243 [main] INFO scm.XceiverClientShortCircuit: DomainSocket(fd=311,path=/Users/sammi/ozone_dn_socket) is closed for 249539d9-f678-4ecf-aa6d-fcbcea75f627(localhost/127.0.0.1) 2024-09-24 15:37:31,243 [Pipeline-df315eb9-30ee-4765-9e68-7fc16335ce48-XceiverClientShortCircuit-SendRequest] INFO scm.XceiverClientShortCircuit: sendRequestTask is interrupted *************************************************** Status: Success Git Base Revision: bab77db646e17201dd953cc9f0a805502d24ea45 Number of Volumes created: 1 Number of Buckets created: 10 Number of Keys added: 10000 Replication: RATIS/ONE Average Time spent in volume creation: 00:00:00,008 Average Time spent in bucket creation: 00:00:00,014 Average Time spent in key creation: 00:00:03,689 Average Time spent in key write: 00:00:02,927 Total bytes written: 10485760000 Total number of writes validated: 10000 Writes validated: 100.0 % Successful validation: 10000 Unsuccessful validation: 0 Total Execution time: 00:00:29,326 *************************************************** ``` Still working on whether it's possible to test it with robot test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
