xiangfu0 opened a new pull request, #17790: URL: https://github.com/apache/pinot/pull/17790
## Summary - Replace `KafkaServerStartable` (Docker CLI-based) with `EmbeddedKafkaCluster` using Kafka's `KafkaClusterTestKit` in KRaft mode for integration tests - Eliminates Docker dependency for tests, improving startup speed (~5s vs ~30-60s) and removing the primary source of CI flakiness on GitHub Actions (image pulls, network creation races, port mapping issues, container timeouts) - Simplify `BaseClusterIntegrationTest` Kafka lifecycle — no retry loop, no Docker container management, no port scanning ## Changes - **New**: `EmbeddedKafkaCluster` — in-process KRaft cluster wrapper implementing `StreamDataServerStartable`, supporting multi-broker clusters and Kafka transactions - **Modified**: `BaseClusterIntegrationTest` — rewrote Kafka start/stop to use embedded cluster; fixed replication factor calculation for multi-broker clusters; removed Docker-specific dead code (~170 lines) - **Modified**: `KafkaPartitionLevelConsumerTest` — use `EmbeddedKafkaCluster` instead of Docker - **Modified**: `pom.xml` (root, pinot-kafka-3.0, pinot-integration-test-base, pinot-integration-tests) — added kafka test-jar dependencies for embedded broker support - **Unchanged**: `KafkaServerStartable` (production Docker-based class for QuickStart) — not modified ## Test plan - [x] `KafkaPartitionLevelConsumerTest` — 9 tests pass (single broker) - [x] `LLCRealtimeClusterIntegrationTest` — passes (2 brokers, non-transactional) - [x] `ExactlyOnceKafkaRealtimeClusterIntegrationTest` — passes (3 brokers, Kafka transactions with `read_committed`) - [ ] Full integration test suite via CI 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
