dulu98Kurz commented on PR #15116: URL: https://github.com/apache/druid/pull/15116#issuecomment-1784670248
Hi @cryptoe @kfaraz @AmatyaAvadhanula thanks a lot for the input, seems there are more than one concerns on refactoring to `int` I have reverted related change. However I believe I found a way to make everyone happy without refactoring to `int`, remember the origination of issue https://github.com/apache/druid/issues/15091 is: `2023-09-29T02:45:16,308 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task. java.lang.IllegalArgumentException: fromKey > toKey at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1368) ~[?:1.8.0_302] at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1855) ~[?:1.8.0_302] at java.util.TreeMap.subMap(TreeMap.java:913) ~[?:1.8.0_302] at org.apache.druid.timeline.partition.OvershadowableManager.entryIteratorGreaterThan(OvershadowableManager.java:423) ~[druid-processing-2023.03.1-iap.jar:2023.03.1-iap] ` Now since we heavily relies on Short.toUnsignedInt when comparing and determining overlapping, we can make sure we never encounter `java.lang.IllegalArgumentException: fromKey > toKey` by setting the toKey to 0xFFFF instead of Short.MAX_VALUE, and 0xFFFF is -1 in signed short, so instead of doing: `final RootPartitionRange highFence = new RootPartitionRange(Short.MAX_VALUE, Short.MAX_VALUE);` We do : `final RootPartitionRange highFence = new RootPartitionRange((short) (-1), (short) (-1));` To show a few examples that illustrate why -1 works: <img width="576" alt="image" src="https://github.com/apache/druid/assets/14854898/f72ac0dc-685b-4c73-a27e-ba9694310d66";> This change should address the logic mistakes in the code and keep ingestion task alive, without increasing memory footprint, please advise! Thanks again for the time on this! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
