Copilot commented on code in PR #11190:
URL: https://github.com/apache/ozone/pull/11190#discussion_r3927301971
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java:
##########
@@ -485,12 +488,27 @@ public CompletableFuture<Message>
applyTransaction(TransactionContext trx) {
// lastAppliedIndex in OzoneManager StateMachine, even if other
// executor has completed the transactions with id more.
- //if there are too many pending requests, wait for doubleBuffer flushing
- ozoneManagerDoubleBuffer.acquireUnFlushedTransactions(1);
+ enterApplyTransaction();
+ try {
+ //if there are too many pending requests, wait for doubleBuffer
flushing
+ ozoneManagerDoubleBuffer.acquireUnFlushedTransactions(1);
+ } catch (Exception ex) {
+ exitApplyTransaction();
+ throw ex;
+ }
- return CompletableFuture.supplyAsync(() -> runCommand(request,
termIndex), executorService)
+ return CompletableFuture.supplyAsync(() -> {
+ try {
+ return runCommand(request, termIndex);
+ } finally {
+ exitApplyTransaction();
Review Comment:
applyTransaction() increments inFlightApplyTransactions before scheduling
runCommand(), but if CompletableFuture.supplyAsync(..., executorService) throws
synchronously (eg RejectedExecutionException), exitApplyTransaction() is never
called and the unflushed-transactions permit is never released. This can cause
pause() to block forever waiting for in-flight apply work to drain and can also
deadlock backpressure via leaked permits.
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java:
##########
@@ -324,6 +324,58 @@ public void testApplyTransactionBackpressureInterrupt()
throws Exception {
assertInstanceOf(InterruptedException.class, ex.getCause());
}
+ @Test
+ public void testPauseBlocksApplyUntilUnpause() throws Exception {
+ OzoneManagerStateMachine testSm = new OzoneManagerStateMachine(
+ om, doubleBuffer, handler, executor, null) {
+ @Override
+ public OzoneManagerDoubleBuffer buildDoubleBufferForRatis() {
+ return doubleBuffer;
+ }
+ };
+ try {
+ OMRequest request = sampleWriteRequest();
+ TransactionContext trx = mockTrx(request, 1, 5);
+
+ OMResponse expectedResponse = OMResponse.newBuilder()
+ .setCmdType(Type.CreateKey)
+ .setStatus(Status.OK)
+ .setSuccess(true)
+ .build();
+ OMClientResponse clientResponse = mock(OMClientResponse.class);
+ when(clientResponse.getOMResponse()).thenReturn(expectedResponse);
+ when(clientResponse.getOmLockDetails()).thenReturn(null);
+
+ CountDownLatch handlerInvoked = new CountDownLatch(1);
+ doAnswer(invocation -> {
+ handlerInvoked.countDown();
+ return clientResponse;
+ }).when(handler).handleWriteRequest(eq(request), any(),
eq(doubleBuffer));
+
+ testSm.pause();
+ CompletableFuture<CompletableFuture<Message>> applyFuture =
CompletableFuture.supplyAsync(() -> {
+ try {
+ return testSm.applyTransaction(trx);
+ } catch (Exception ex) {
+ throw new RuntimeException(ex);
+ }
+ });
+
+ assertFalse(handlerInvoked.await(200, TimeUnit.MILLISECONDS),
+ "apply should stay blocked while state machine is paused");
+ assertFalse(applyFuture.isDone(), "applyTransaction should block while
paused");
+
+ testSm.unpause(5, 1);
+
+ CompletableFuture<Message> future = applyFuture.get(2, TimeUnit.SECONDS);
+ assertTrue(handlerInvoked.await(2, TimeUnit.SECONDS),
+ "apply should continue after state machine unpause");
+ assertNotNull(future.get(2, TimeUnit.SECONDS));
+ } finally {
+ testSm.stop();
+ }
+ }
+
Review Comment:
The PR description mentions coverage for executor scheduling failure (eg
RejectedExecutionException) so pause() does not hang, but this test file only
adds pause/unpause coverage. Adding a regression test for executor rejection
would prevent reintroducing apply-drain leaks during checkpoint pause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
