david-streamlio opened a new issue, #89: URL: https://github.com/apache/pulsar-connectors/issues/89
## Description The `Tests - Connectors` job on #86 ran for **45m16s** and was **cancelled** by the runner's `timeout-minutes: 45`, not failed. No diagnostics, 45 minutes of runner time consumed, and a red PR that had nothing wrong with it. Run: https://github.com/apache/pulsar-connectors/actions/runs/29100595779 ## Where it hung `:mqtt:integrationTest` started at `14:50:30` and **never emitted another line**. Everything else in the job had already completed — #86's own four modules (mongo, hdfs3, influxdb, hbase) ran and passed by 14:47, and `:jdbc:pulsar-io-jdbc-sqlite:test` printed its last result at 14:50:36. Then 35 minutes of silence, ending with the runner killing three orphaned `java` processes. On healthy runs the same task prints its test result within milliseconds of starting (e.g. [run 29102710509](https://github.com/apache/pulsar-connectors/actions/runs/29102710509)). ## Why it can hang `MqttSinkIntegrationTest` is careful *inside* the test — every wait is bounded (`ackLatch.await(10, SECONDS)`, `receivedPayloads.poll(10, SECONDS)`, `connect().get(10, SECONDS)`). The unbounded step is container startup, in `@BeforeClass`: ```java private final GenericContainer<?> mqttContainer = new GenericContainer<>(MOSQUITTO_IMAGE) .withExposedPorts(MQTT_PORT); // no waitingFor(), no withStartupTimeout() @BeforeClass(alwaysRun = true) public void beforeClass() { mqttContainer.start(); // image pull is not bounded } ``` A hang in `@BeforeClass` produces exactly the observed signature: the Gradle task starts, no test ever reports, nothing times out. The test method also carries no `@Test(timeOut = ...)`, so TestNG cannot bound it either, and Gradle has no per-test timeout. **The 45-minute job limit is the only backstop.** Image pull is the most likely culprit — Testcontainers does not bound it, and this job pulls a lot of images now (Debezium MongoDB, Kafka, Solr, DynamoDB/LocalStack, InfluxDB v1+v2, MariaDB, ClickHouse, Postgres, Mosquitto). Docker Hub throttling would stall the pull silently. Note this test only began running in CI at all when #74 added `integrationTest` to the matrix, so the hazard is newly live, not longstanding. ## Fix 1. Bound the container: `.waitingFor(...)` with an explicit `withStartupTimeout(Duration.ofMinutes(2))`. 2. Add `@Test(timeOut = ...)` as a backstop, consistent with the Debezium tests hardened in #71. 3. Surface Testcontainers logs in CI — no Testcontainers output appears anywhere in the job log, which is why the cause had to be inferred from timestamps rather than read. Same principle as #70: a test that cannot fail fast will consume the whole job budget and report a cancellation, which reads like an infrastructure problem rather than a test problem. ## Related The CI test job has roughly doubled in duration today (≈8–10 min → ≈20 min) as container-backed tests landed. At 45 minutes the ceiling is not close, but a single hang exhausts it. Worth considering splitting `integrationTest` into its own parallel job. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
