gianm commented on a change in pull request #7203: Fix race in historical when
loading segments in parallel
URL: https://github.com/apache/incubator-druid/pull/7203#discussion_r263498799
##########
File path: server/src/main/java/org/apache/druid/server/SegmentManager.java
##########
@@ -171,9 +167,16 @@ public boolean loadSegment(final DataSegment segment)
throws SegmentLoadingExcep
);
if ((entry != null) &&
(entry.getChunk(segment.getShardSpec().getPartitionNum()) != null)) {
- log.warn("Told to load a adapter for a segment[%s] that already
exists", segment.getId());
+ log.warn("Told to load an adapter for segment[%s] that already
exists", segment.getId());
resultSupplier.set(false);
} else {
+ final Segment adapter;
+ try {
+ adapter = getAdapter(segment);
Review comment:
I think this means we will only be able to load 1 segment at a time for a
datasource (ConcurrentHashMap.compute synchronizes on the hashtable Node when
applying the `remappingFunction`). If so, it's still not parallel enough.
Locking at the level of an individual segment would be best; maybe something
that involves tracking a Map of currently-loading segments to Objects we can
use for synchronization?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
