kfaraz commented on issue #19117: URL: https://github.com/apache/druid/issues/19117#issuecomment-4028858772
@gianm , the discussion here https://github.com/apache/druid/pull/18950#discussion_r2757715069 has some details. > ah i was thinking the case when the umbrella interval is smaller than the original interval (which is the case in CompactionTaskRunBase, the segment is hourly and there's only segments in hour 0 - hour 3, so compaction task only locks for 3 hours, but the interval can be all day (see NativeCompactionTaskRunTest). MSQ runner failed when getting a task lock with the following change: Essentially, if a `compact` task is trying to compact a DAY interval but that interval currently contains only a couple of HOUR segments, we would end up locking an interval smaller than the whole DAY unless we consider the `segmentProvider.interval` as well. This would mean that other tasks could potentially write concurrently to the remainer of the DAY interval, even though we had intended to lock the whole thing up. A solution to the large number of intervals (since we have encountered this issue earlier as well) could be to impose a limit on `Granularity.getIterable()` or at least have a safe variant of that method which imposes a limit on the number of intervals created. If that number is exceeded, we could throw an exception, and advise the user to either choose a smaller interval or a bigger granularity. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
