xiangfu0 opened a new pull request, #18985: URL: https://github.com/apache/pinot/pull/18985
## Problem The compatibility regression test jobs fail during cluster setup because the Kafka binary download never succeeds (example: [failing run](https://github.com/apache/pinot/actions/runs/29234038987/job/86764483195?pr=18870)): ``` Setting up Kafka from https://downloads.apache.org/kafka/3.9.2/kafka_2.13-3.9.2.tgz Downloading Kafka from https://downloads.apache.org/kafka/3.9.2/kafka_2.13-3.9.2.tgz Primary download failed, trying archive mirror: https://archive.apache.org/dist/kafka/3.9.2/kafka_2.13-3.9.2.tgz ##[error]Process completed with exit code 1. ``` Two compounding issues: 1. **Kafka 3.9.2 was removed from `downloads.apache.org`** — the live mirror now only hosts 4.1.2 / 4.2.1 / 4.3.1, so the primary URL 404s on every run (same failure mode #18082 fixed when 3.7.x disappeared). 2. **The `archive.apache.org` fallback can't finish in time** — the archive is bandwidth-throttled, so each `--max-time 300` attempt times out on the ~130MB tarball. With `--retry 3` the job burns ~20 minutes (4 × 300s) and then exits 1. ## Fix - Bump the default Kafka to **4.2.1**, which is on the live (fast, CDN-backed) mirror. Download order is unchanged: the `downloads.apache.org` URL is tried first; on failure (404 once the version rotates off) it falls back to `archive.apache.org`, now logging the HTTP status of the failed primary attempt. - Kafka 4.x is KRaft-only, so convert the single-node setup from ZooKeeper mode to a combined broker+controller KRaft node: KRaft server properties, a controller listener on port 19093 (checked for availability like the other ports), and a `kafka-storage.sh format` step before start (the data dir is wiped on every setup, so it re-formats each time). - Make the archive-mirror fallback stall-tolerant instead of hard-capped at 300s: abort only if throughput stays below 100KB/s for 60s (too slow to ever finish the ~130MB tarball) or after 30 minutes, with a single retry so the worst case stays well within the job timeout. Partial downloads are removed on failure so re-runs in the same working dir don't extract a truncated tarball. When 4.2.1 eventually rotates off the live mirror, the fallback degrades to slow instead of broken. Nothing else needed changing: topic creation in `StreamOp` uses `AdminClient` over bootstrap servers, the kafka30 consumer plugin (used by both old and new Pinot builds in the suite) never reads the ZK broker URL, and 3.x Kafka clients are wire-compatible with 4.x brokers (Kafka 4.x requires clients ≥ 2.1 per KIP-896 — release-1.5.0 and master both pin kafka-clients 3.9.x, so both matrix legs are safe; only a manually-triggered run against a very old `oldCommit` with a pre-2.1 client would be affected). The CI runners use JDK 21, satisfying Kafka 4.x's JDK 17+ broker requirement. ## Testing Exercised the modified `setupKafkaBinary`/`startKafkaService`/`stopKafkaService` functions locally (sourcing them from the script as-is): download from the live mirror, KRaft format + start (broker up in ~4s), topic create / produce / consume round-trip, a stop/restart cycle (re-format after data-dir wipe), and clean shutdown via `kafka-server-stop.sh` all pass. Also exercised the fallback branch with a 404ing primary URL: the failure is detected immediately (no retry burn on 404, `HTTP 404` logged) and the archive download proceeds at the throttled rate without tripping the stall detector. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
