[
https://issues.apache.org/jira/browse/HDDS-9446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17777039#comment-17777039
]
Duong commented on HDDS-9446:
-----------------------------
[~weichiu] I didn't dig deep emough into grpc threads settings/status.
I ran the same test, with the same workload and the same number of client-side
threads, OM with GRPC cannot get 20 requests concurrently while OM with
HadoopRPC always processed 100 requests concurrently (with 100 hRPC rpc handler
threads). Test OM server has cores.
The GRPC problem can be anywhere, client or server side. I feel that it could
be an easy config tunning for someone familiar with GRPC.
> S3 default GRPC transport doesn't utilize enough parallelism on OM server-side
> ------------------------------------------------------------------------------
>
> Key: HDDS-9446
> URL: https://issues.apache.org/jira/browse/HDDS-9446
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM, S3
> Reporter: Duong
> Priority: Blocker
> Labels: performance, pull-request-available
>
> GRPC is enabled by default as the transport between S3->OM, as per HDDS-7309.
> We did a S3 GET stress test using warp in a cluster and realized that with
> GRPC transport, S3 can't invoke enough parallelism in OM. We tested with more
> than 300 warp client threads spread on 15 S3 instances, but OM doesn't use up
> to more than 15 threads concurrently (we use the metric added by HDDS-9424 to
> access).
>
> {code:java}
> warp get --warp-client=1.1.1.{34...49} --host=localhost:9879 --tls
> --insecure --access-key=hdfs/[email protected]
> --secret-key=ada0758fe2d12a55dbea8ff6b7df52c377bf46a8e2886d6d17fae5b77a7c4380
> --obj.size=1kb --concurrent=20 --bucket bucket-obs {code}
> And this has caused a degradation (30%) in S3->OM performance.
> It's also verified that switching back to HadoopRPC3 did utilize 100 threads
> in OM RPC server and brought back the performance before.
> I think this is purely a problem with GRPC default configs.
> We need to either tune GRPC default configs, or make HadoopRPC the default
> transport.
> cc. [~xBis] [~ritesh]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
