david-streamlio opened a new issue, #70:
URL: https://github.com/apache/pulsar-connectors/issues/70
## Description
`DebeziumMysqlSourceTest` and `DebeziumPostgresSourceTest` (both on
`master`) can hang indefinitely rather than fail if their connector delivers
fewer CDC records than expected. When that happens the GitHub Actions job runs
until its `timeout-minutes: 45` and reports a **cancellation**, not a test
failure — no useful diagnostics, 45 minutes of runner time consumed.
This is not hypothetical. It was observed in CI on #58: the `Tests -
Connectors` job ran for exactly 45m15s and was cancelled. The mechanism was
reproduced locally.
## Root cause
`AbstractKafkaConnectSource.read()` is a `while (true)` loop whose only
`return` yields a record; an empty poll hits `continue`. **It never returns
`null`.**
The tests do:
```java
Awaitility.await()
.atMost(60, TimeUnit.SECONDS)
.untilAsserted(() -> {
int recordCount = 0;
Record<KeyValue<byte[], byte[]>> record;
while ((record = source.read()) != null) { // never null
...
if (recordCount >= 2) break; // only escape
}
assertTrue(recordCount >= 2, ...);
});
```
If fewer than 2 records ever arrive, `read()` spins forever. Awaitility's
`atMost(60s)` cannot save it: `untilAsserted` does not interrupt a blocked
assertion, so the deadline never fires.
Neither test declares a `timeOut`, so TestNG does not bound them either.
They pass today only because MySQL and Postgres reliably emit their snapshot
records — the moment either under-delivers for any environmental reason, CI
hangs.
## Reproduction
Raising the expected record count above the number of seeded rows produces
the hang on the unmodified harness. With the fix applied (see below), the same
condition fails in ~37 seconds with a `TimeoutException` naming the connector.
## Fix
Read each record on a bounded worker thread (per-record deadline,
`cancel(true)` on timeout, `shutdownNow()` in cleanup) and add a test-level
`timeOut` as a backstop, so under-delivery fails promptly with a diagnosable
message. This pattern is already applied in #58.
## Scope
- `debezium/mysql` — `DebeziumMysqlSourceTest` (no `timeOut`)
- `debezium/postgres` — `DebeziumPostgresSourceTest` (no `timeOut`)
Also affected, tracked in their own PRs:
- `debezium/mongodb` — fixed in #58
- `debezium/mssql` — #60 has the same pattern and no `timeOut`
- `debezium/oracle` — #62 has a `timeOut` backstop but still reads on the
test thread
- `debezium/mariadb` — #69 has a `timeOut` backstop but still reads on the
test thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
