himanshug commented on issue #7192: Segments are not cleaned up properly after unannouncing them in HTTP-based segment loading if numLoadingThreads > 1 URL: https://github.com/apache/incubator-druid/issues/7192#issuecomment-529016474 It is indeed possible for `HttpLoadQueuePeon` to retry if request to load segment by historical failed for whatever reason (but historical got the request and started loading the segment). but it maybe possible that there is some bug as well. That said, Retrying needs to be part of protocol and expectation is that historical would be able to take care of duplication which I think is handled by #7203 > Multiple threads can load the same segment and announce it in different ZooKeeper paths that looks like a bug in `BatchDataSegmentAnnouncer` which does appear to have necessary synchronization and deduplication code. that said, above problem does not exist at all if HTTP based segment discovery is used . historicals could be told to not announce any segments in zookeeper then ( `druid.announcer.skipSegmentAnnouncementOnZk=false` ) I think we should just switch to HTTP for both segment load/drop and discovery when we switch. sidenote: I have been using HTTP based segment management, discovery and HTTP based task management (with patch https://github.com/apache/incubator-druid/pull/8427 ) on my cluster since long time.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
