QiuMM edited a comment on issue #6336: update insert pending segments logic to synchronous URL: https://github.com/apache/incubator-druid/pull/6336#issuecomment-450610474 > How about improving the lock granularity for taskLockbox? There's no need to use a single lock, but it can use more granular lock per dataSource per interval. @jihoonson I have tried like below: ```java /** * Perform the given action with a guarantee that we synchronize actions on the same (Datasource, Interval) pair. * This method just checks that the pair of (Datasource, Interval) is valid, does not care about locks for the given * task and interval, so you should care about the locks by yourself. * * @param task task performing a {@link SyncDataSourceAndIntervalAction} * @param interval interval * @param action action to be performed */ public <T> T doInSyncDataSourceAndInterval( Task task, Interval interval, SyncDataSourceAndIntervalAction<T> action ) throws Exception { Pair<String, Interval> pair = dataSourceIntervals.getOrDefault(task.getDataSource(), new HashMap<>()).get(interval); if (pair != null) { synchronized (pair) { return action.perform(); } } else { log.warn("Invalid dataSource and interval pair: (%s, %s), can not sync on it!", task.getDataSource(), interval); return null; } } ``` However, the overlord console still loading very slowly after I restart the overlord since all newly created tasks post a segment allocate action. But about one hours later, I didn't see any delay when open the console except the moment that all tasks start to allocate new segments. I think it's still terrible because every time I restart the overlord I would like to open the console to see the status of the overlord, It's hard to bear if it loading slow. Maybe we should not restart all tasks at the same moment, how about add some delay? Besides, after I use such granular lock, some errors occured: ``` 2018-12-29 18:19:10,252 ERROR [qtp685428529-136] com.sun.jersey.spi.container.ContainerResponse - The RuntimeException could not be mapped to a response, re-throwing to the HTTP container io.druid.java.util.common.ISE: Segments not covered by locks for task: index_kafka_huoshan_all_map_3c8768ae0f0f563_ihpmdnek at io.druid.indexing.common.actions.TaskActionPreconditions.checkLockCoversSegments(TaskActionPreconditions.java:45) ~[druid-indexing-service-0.12.2.jar:0.12.2] at io.druid.indexing.common.actions.SegmentTransactionalInsertAction.perform(SegmentTransactionalInsertAction.java:107) ~[druid-indexing-service-0.12.2.jar:0.12.2] at io.druid.indexing.common.actions.SegmentTransactionalInsertAction.perform(SegmentTransactionalInsertAction.java:47) ~[druid-indexing-service-0.12.2.jar:0.12.2] at io.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:64) ~[druid-indexing-service-0.12.2.jar:0.12.2] at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:368) ~[druid-indexing-service-0.12.2.jar:0.12.2] at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:357) ~[druid-indexing-service-0.12.2.jar:0.12.2] at io.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:683) ~[druid-indexing-service-0.12.2.jar:0.12.2] at io.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:354) ~[druid-indexing-service-0.12.2.jar:0.12.2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_131] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131] ``` Such error occurs after I restart the overlord and all newly created tasks would pending for some minutes which is weird and make me confused (I have never modified `SegmentTransactionalInsertAction`). But after all tasks are running normally, I never see any errors again. > BTW, I guess your pendingSegments table might be very large if allocatePendingSegment takes too long. Does cleaning up old entries in pendingSegments table help? I have already set `druid.coordinator.kill.pendingSegments.on` to true, so old entries would be cleaned up.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
