lhotari commented on PR #26106: URL: https://github.com/apache/pulsar/pull/26106#issuecomment-4848365164
Recent CI failure where this production issue showed up: https://github.com/apache/pulsar/actions/runs/28466180778/attempts/1?pr=26122 That run is on an unrelated PR (#26122, a different flaky-test fix) whose branch is based on `master` **without** this fix. The `CI - Unit - Brokers - Broker Group 5` job failed in `ReplicatorTest.testReplicationWithSchema`: ``` java.lang.AssertionError: expected [true] but found [false] at org.apache.pulsar.broker.service.ReplicatorTest.testReplicationWithSchema(ReplicatorTest.java:414) ``` Line 414 is `assertTrue(msg1 != null && msg2 != null && msg3 != null)` — a remote-cluster consumer timed out (`receive(10, SECONDS)` returned `null`) on a message after the schema boundary. The test sends 500 messages with one Avro schema, then 500 with a second schema starting at id 500, and expects all three clusters to receive every message. ### Why this is the same bug this PR fixes It is the same `readEntriesComplete()` skip-branch stall, reached through the **schema-fetch** rewind rather than the failed-publish rewind described in the PR motivation: 1. At the schema boundary, `GeoPersistentReplicator.replicateEntries()` rewinds the cursor to re-read with the new schema: `beforeTerminateOrCursorRewinding(ReasonOfWaitForCursorRewinding.Fetching_Schema)` (`GeoPersistentReplicator.java:270`), and after the schema is fetched, `doRewindCursor(true)` (`GeoPersistentReplicator.java:286`). 2. If a cursor read was already dispatched at that moment (made naturally reachable by the read pipelining in #26005), `cursor.cancelPendingReadRequest()` returns `false`, so `cancelPendingReadTasks(false)` only flags the in-flight task (`skipReadResultDueToCursorRewind = true`) and leaves it with `entries == null` — it does not complete it. 3. When that dispatched read later completes, the `readEntriesComplete()` skip branch discards the result without calling `setEntries(...)`, so the task stays `entries == null` forever → `hasPendingRead()` stays true and `getPermitsIfNoPendingRead()` returns 0. The `readMoreEntries()` issued by `doRewindCursor(true)` then early-returns, so replication never resumes past the schema boundary. The remote consumer times out and the assertion fails. So `Fetching_Schema` is a second trigger of the same root cause as the `Failed_Publishing` path in the PR description; because the fix is in the shared `readEntriesComplete()` skip branch, it covers both rewind reasons. Consistent with that, `Broker Group 5` (which runs `testReplicationWithSchema`) is green on this PR's latest CI run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
