raghavyadav01 opened a new pull request, #20214:
URL: https://github.com/apache/pinot/pull/20214

   ## Summary
   
   When a REALTIME table is deleted, `RealtimeTableDataManager.doShutdown()` 
first calls `IngestionDelayTracker.shutdown()`, then 
`releaseAndRemoveAllSegments()`, which interrupts the consumer threads.
   
   `IngestionDelayTracker.shutdown()` stopped its polling executor with 
`shutdown()` and did not wait for it. It then closed the stream metadata 
providers. If the polling task was mid-fetch of the upstream offset, closing 
the same Kafka consumer threw:
   
   ```
   java.util.ConcurrentModificationException: KafkaConsumer is not safe for 
multi-threaded access
     at KafkaConsumer.close(KafkaConsumer.java:1757)
     at KafkaStreamMetadataProvider.close(KafkaStreamMetadataProvider.java:346)
     at IngestionDelayTracker.shutdown(IngestionDelayTracker.java:691)
     at RealtimeTableDataManager.doShutdown(RealtimeTableDataManager.java:294)
   ```
   
   The surrounding `catch (IOException)` did not cover it, so the exception 
escaped `doShutdown()` before the segments were released. The table's consuming 
segment managers kept running after the table was gone. In one occurrence, six 
of them kept polling the stream for days and failed authentication on every 
attempt, and only a server restart cleaned them up. It is a race: one of three 
replicas hit it.
   
   ## Changes
   
   - `IngestionDelayTracker.shutdown()` calls `shutdownNow()` on the polling 
executor and waits (bounded at 10s) for the in-flight task to stop before 
closing providers. It catches any exception from `close()`, not just 
`IOException`.
   - `RealtimeTableDataManager.doShutdown()` guards the tracker shutdown, so a 
failure there can no longer skip releasing the segments.
   
   ## Testing
   
   - `testShutdownClosesProviderOnlyAfterInFlightFetchStops`: a provider whose 
fetch blocks like a consumer waiting on the broker; asserts that `close()` 
never runs while the fetch is in flight. Fails without this change.
   - `testShutdownSurvivesProviderCloseFailure`: a provider whose `close()` 
throws a runtime exception; shutdown must not throw. Fails without this change.
   - `IngestionDelayTrackerTest`: 11/11 pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to