Hi Gian and Maytas, I'm writing this email to you to bring an old ingestion issue for your attention to discuss.
It's about lag while tasks are rolling, here's the link: https://github.com/apache/druid/issues/11414 >From the issue, the root cause is that tasks take several seconds to start up during which messages can't be consumed from Kafka. Giann linked some PRs in that issue which improved performance of notice processing, but this didn't solve the problem completely, and the last reply in this thread suggested that on Druid 27, this problem still exists. I also noticed that Maytas said in the HADOOP INGESTION SUPPORT thread that he is going to use K8S-based ingestion to replace Middle Managers which makes sense to me because it improves the resource utilization. But the above lag issue might be magnified because K8S scheduling introduces some extra delay, for example resource allocation at K8S side, pulling image from repository, it can be seconds to do so, which means K8S-based ingestion tasks generally have slower start up. So I include Maytas in the hope that this problem has already been noticed or even solved by his team. If you have any suggestions/ideas, please reply to the original issue, so that all information is in place. Thanks and regards.
