jasperjiaguo commented on issue #8910: URL: https://github.com/apache/pinot/issues/8910#issuecomment-1158173016
Hello @ankitsultana imiting concurrency and throttling are both implemented. You can take a look at this feature https://github.com/apache/pinot/pull/8753 and also this one https://github.com/apache/pinot/pull/8694. Basically what we do here is: https://github.com/apache/pinot/pull/8753: 1. When your segment is **unencrypted**, stream download and untar, so that server wouldn't write the tar ball to disk, as will save around 1/3 of bytes written to disk. config `pinot.server.instance.segment.stream.download.untar` 2. When the above feature is on, control download rate per segment , in bytes/sec config `pinot.server.instance.segment.stream.download.untar.rate.limit.bytes.per.sec` https://github.com/apache/pinot/pull/8694: At the table level control the number of segments downloading in parallel. `pinot.server.instance.table.level.max.parallel.segment.downloads` These two features work **independently**. But when they are configured together, you essentially have the ability to control the per table level download rate, and avoid saturating the disk. Please let me know if you have other questions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
