Hey Samarth,
I wrote a comment in the issue - thanks for looking at this, it is a
valuable issue IMO.
On Mon, Feb 18, 2019 at 4:26 PM Samarth Jain wrote:
> I have created an issue
> https://github.com/apache/incubator-druid/issues/7068 that outlines the
> limitations in current approach which
I have created an issue
https://github.com/apache/incubator-druid/issues/7068 that outlines the
limitations in current approach which prevents us from parallelizing the
segment load/drop workload. Also have raised a PR
https://github.com/apache/incubator-druid/pull/7088 to help address it.
On
I believe today, if you use the (experimental) HTTP-based load queues, they
will parallelize segment downloads. Adding similar functionality for the
ZK-based load queues would definitely be useful though, since at this time
nobody seems to be actively driving a migration to HTTP-based load queues
I *think* the HTTP coordination already enables this
On Wed, Jan 30, 2019 at 4:20 PM Samarth Jain wrote:
> We noticed that it takes a long time for the historicals to download
> segments from deep storage (in our case S3). Looking closer at the code in
> ZKCoordinator, I noticed that the
We noticed that it takes a long time for the historicals to download
segments from deep storage (in our case S3). Looking closer at the code in
ZKCoordinator, I noticed that the segment download is happening in a single
threaded fashion. This download happens in the SingleThreadedExecutor
service
