phet commented on code in PR #3805:
URL: https://github.com/apache/gobblin/pull/3805#discussion_r1369648784
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -237,13 +257,16 @@ protected void startUp() {
private void consume() {
try {
Iterator<KafkaConsumerRecord> itr = gobblinKafkaConsumerClient.consume();
+ // TODO: we may be committing too early and only want to commit after
process messages
if(!enableAutoCommit) {
commitOffsets();
}
while (itr.hasNext()) {
KafkaConsumerRecord record = itr.next();
int idx = record.getPartition() % numThreads;
queues[idx].put(record);
+ // Increment queue size metric
+ updateQueueSizes(idx);
Review Comment:
this is a pretty fundamental change to put into the fast path! we gotta be
certain that no critical use is adversely affected by perf regression.
I see `ConcurrentLinkedQueue::size` javadoc saying `O(n)`, but I can't find
a statement on `LinkedBlockingQueue::size`...
(it's possible it IS `O(1)`, but an approximation or else exact, but at the
expense of coordination overhead.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]