sakshichitnis27 opened a new pull request, #19858:
URL: https://github.com/apache/druid/pull/19858
Fixes #19575
### Description
`KafkaLookupExtractorFactory` previously reported a lookup as started
immediately after the consumer's first `poll()`, even when the consumer had not
finished reading the existing topic data. This allowed newly started brokers to
serve queries using a partially populated lookup.
This change:
- Captures the end offsets after the consumer receives its initial partition
assignment.
- Processes polled records before checking catch-up progress.
- Releases the startup latch only after every assigned partition reaches its
captured end offset.
- Refreshes the captured offsets if the partition assignment changes during
startup.
- Stops checking offsets after startup completes.
The existing `connectTimeout` continues to bound how long `start()` waits. A
value of `0` retains the existing do-not-wait behavior.
A regression test verifies that `start()` remains blocked while the consumer
is behind and succeeds after all initial records have been applied to the
lookup.
### Testing
```bash
mvn test -pl extensions-core/kafka-extraction-namespace -am \
-Dtest=org.apache.druid.query.lookup.KafkaLookupExtractorFactoryTest \
-Dsurefire.failIfNoSpecifiedTests=false \
-Dweb.console.skip=true \
-T1C
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
