phet commented on code in PR #3805:
URL: https://github.com/apache/gobblin/pull/3805#discussion_r1369656381


##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/metrics/RuntimeMetrics.java:
##########
@@ -28,6 +28,7 @@ public class RuntimeMetrics {
   // Metric names
   public static final String GOBBLIN_KAFKA_HIGH_LEVEL_CONSUMER_MESSAGES_READ =
       "gobblin.kafka.highLevelConsumer.messagesRead";
+  public static final String GOBBLIN_KAFKA_HIGH_LEVEL_CONSUMER_QUEUE_SIZE = 
"gobblin.kafka.highLevelConsumer.queueSize";

Review Comment:
   nit: seems to be used as a prefix



##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -197,6 +201,22 @@ protected void shutdownMetrics() throws IOException {
    */
   protected void createMetrics() {
     this.messagesRead = 
this.metricContext.counter(RuntimeMetrics.GOBBLIN_KAFKA_HIGH_LEVEL_CONSUMER_MESSAGES_READ);
+    this.queueSizeGauges = new ContextAwareGauge[numThreads];
+    this.queueSizeGaugeValues = new AtomicIntegerArray(numThreads);
+    for (int i=0; i < numThreads; i++) {
+      int finalI = i;
+      this.queueSizeGauges[i] = this.metricContext.newContextAwareGauge(
+          RuntimeMetrics.GOBBLIN_KAFKA_HIGH_LEVEL_CONSUMER_QUEUE_SIZE + "-" + 
i,
+          () -> queueSizeGaugeValues.get(finalI));

Review Comment:
   w/o a comment explaining the reason, I'm left to wonder: why couldn't this 
be written:
   ```
   () -> queues[finalI].size()
   ```
   ?  that would dispense w/ the need for `queueSizeGaugeValues` and with it 
the potential perf hit of updating that array upon every enqueing.
   
   and on that topic... why is `updateQueueSizes` only recalculated at enqueue 
time, rather than at dequeue time as well?



##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -197,6 +201,22 @@ protected void shutdownMetrics() throws IOException {
    */
   protected void createMetrics() {
     this.messagesRead = 
this.metricContext.counter(RuntimeMetrics.GOBBLIN_KAFKA_HIGH_LEVEL_CONSUMER_MESSAGES_READ);
+    this.queueSizeGauges = new ContextAwareGauge[numThreads];
+    this.queueSizeGaugeValues = new AtomicIntegerArray(numThreads);
+    for (int i=0; i < numThreads; i++) {
+      int finalI = i;

Review Comment:
   I see what you're doing... but it may not immediately register to 
everyone... especially since `finalI` is not `final int`.  please add a comment 
calling attention to variable capture by the closure



##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -197,6 +201,22 @@ protected void shutdownMetrics() throws IOException {
    */
   protected void createMetrics() {
     this.messagesRead = 
this.metricContext.counter(RuntimeMetrics.GOBBLIN_KAFKA_HIGH_LEVEL_CONSUMER_MESSAGES_READ);
+    this.queueSizeGauges = new ContextAwareGauge[numThreads];
+    this.queueSizeGaugeValues = new AtomicIntegerArray(numThreads);
+    for (int i=0; i < numThreads; i++) {
+      int finalI = i;
+      this.queueSizeGauges[i] = this.metricContext.newContextAwareGauge(
+          RuntimeMetrics.GOBBLIN_KAFKA_HIGH_LEVEL_CONSUMER_QUEUE_SIZE + "-" + 
i,
+          () -> queueSizeGaugeValues.get(finalI));
+    }
+  }
+
+  /**
+   * Update atomic integer corresponding to queue size
+   * @param queueIndex used to index the queue list
+   */
+  private void updateQueueSizes(int queueIndex) {
+    this.queueSizeGaugeValues.set(queueIndex, queues[queueIndex].size());

Review Comment:
   since `AtomicIntegerArray::set` is not actually an atomic operation (of the 
kind `getAndIncrement` or even `getAndSet`), do we truly need such an instance?



##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -197,6 +201,22 @@ protected void shutdownMetrics() throws IOException {
    */
   protected void createMetrics() {
     this.messagesRead = 
this.metricContext.counter(RuntimeMetrics.GOBBLIN_KAFKA_HIGH_LEVEL_CONSUMER_MESSAGES_READ);
+    this.queueSizeGauges = new ContextAwareGauge[numThreads];
+    this.queueSizeGaugeValues = new AtomicIntegerArray(numThreads);
+    for (int i=0; i < numThreads; i++) {
+      int finalI = i;
+      this.queueSizeGauges[i] = this.metricContext.newContextAwareGauge(
+          RuntimeMetrics.GOBBLIN_KAFKA_HIGH_LEVEL_CONSUMER_QUEUE_SIZE + "-" + 
i,
+          () -> queueSizeGaugeValues.get(finalI));
+    }
+  }
+
+  /**
+   * Update atomic integer corresponding to queue size
+   * @param queueIndex used to index the queue list
+   */
+  private void updateQueueSizes(int queueIndex) {

Review Comment:
   nit: there's only one index updated, so `Sizes` should be singular



##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/kafka/HighLevelConsumer.java:
##########
@@ -237,13 +257,16 @@ protected void startUp() {
   private void consume() {
     try {
       Iterator<KafkaConsumerRecord> itr = gobblinKafkaConsumerClient.consume();
+      // TODO: we may be committing too early and only want to commit after 
process messages
       if(!enableAutoCommit) {
         commitOffsets();
       }
       while (itr.hasNext()) {
         KafkaConsumerRecord record = itr.next();
         int idx = record.getPartition() % numThreads;
         queues[idx].put(record);
+        // Increment queue size metric
+        updateQueueSizes(idx);

Review Comment:
   this is a pretty fundamental change to put into the fast path!  we gotta be 
certain that no critical use is adversely affected by perf regression.
   
   I see `ConcurrentLinkedQueue::size` may not be `O(1)`, but I can't find a 
statement on `LinkedBlockingQueue::size`...
   
   (it's possible it IS `O(1)`, but an approximation or else exact, but at the 
expense of coordination overhead.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to