applike-ss opened a new issue, #17809:
URL: https://github.com/apache/druid/issues/17809
My performance is comparibly low when using AWS S3 as my deep storage and
then using a fresh historical that doesn't hold any data on its volumes yet.
I do get around 50MB/s, while when using the aws-cli on the same pod for
downloading the complete deep storage folder it is more like 400-500MB/s.
### Affected Version
v32.0.0
### Description
Please include as much detailed information about the problem as possible.
- Cluster size:
any
- Configurations in use
Tried many different ones with these settings changes
- druid.segmentCache.numLoadingThreads (1-100)
- xms/xmx/XX:MaxDirectMemorySize java settings
- druid.segmentCache.numBootstrapThreads (1-100)
- aws.maxConnections=500 (likely non-existent option)
- druid.s3.maxConnections=500 (likely non-existent option)
- druid.server.http.numThreads (60-200)
- Steps to reproduce the problem
add s3 as deep storage, empty your historical in use, restart it, see slow
download speeds
With these settings I do get very high download speeds with aws-cli, even
though it is just python code:
```
$ aws configure set default.s3.multipart_threshold 64MB
$ aws configure set default.s3.multipart_chunksize 16MB
$ aws configure set default.s3.max_concurrent_requests 100
$ aws configure set default.s3.max_queue_size 10000
$ aws configure set default.s3.max_bandwidth 1024MB/s
$ aws configure set default.s3.preferred_transfer_client crt #<- most
effective change
```
Neither CPU nor Memory or Network seem too much utilized during druids
segment download, so i don't know where to dig in deeper. Any help is
appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
