GitHub user Yuukadesu edited a discussion: Implementing Integration E2E Tests for Java Client
# Proposal: Implementing Integration E2E Tests for Java Client ## Motivation To ensure the stability and reliability of the StreamPipes Java client library, this implementation adopts an end-to-end testing strategy aligned with the Go Client E2E. First, a test environment closely resembling production is set up (a load-balanced topology based on Docker Compose), including the Backend, multiple Extension service instances, Kafka, CouchDB, and other required dependencies, so that test results are authentic and reproducible. Second, test cases are designed to cover two main objectives: **data path correctness** and **load balancing effectiveness**. They verify not only the end-to-end data flow (Kafka topic-in → adapter/source → processor → sink → Kafka topic-out), but also assert under the load of 70 pipelines and 70 adapters that **adapters** and **pipeline elements** are each distributed across multiple extension instances, with each instance’s share roughly equal (within a configurable deviation, default 20% from the average). To improve test effici ency and consistency, automated tests are written with JUnit 5, and connection parameters are injected via custom startup scripts and system properties. Finally, E2E tests are integrated into GitHub Actions CI: on every PR, images are built, the load-balanced environment is started, and Java Client E2E is executed, enabling continuous monitoring and timely feedback on the Java client and load-balancing behavior, and delivering a reliable and efficient Java client library for StreamPipes users. ## Goals - **Verify data path**: Ensure that adapters and pipelines created via the Java Client correctly complete the end-to-end flow (Kafka input → adapter → Boolean filter processor → Kafka output), and that filter semantics are validated (e.g. only messages with `sensor_fault_flags == false` are passed through). - **Verify load balancing**: The real balancer equalizes **instance load** (CPU, memory, etc.), not adapter/pipeline counts. In the test, under **uniform workload** (same adapter and pipeline type for all 70), equal **count** per instance approximates equal **load** per instance. Verify separately for adapters and pipeline elements that: distinct instances used ≥ configured minimum (default 3); and each instance’s count is roughly equal (within default 20% of the average). - **Early defect detection**: Through automated, repeatable E2E flows and CI integration, find and fix defects related to the Java Client API, Compact Adapter/Pipeline APIs, and load-balancing strategy as early as possible, reducing the cost of production fixes. - **Alignment with Go E2E style**: Execution flow is aligned with Go Client E2E (cleanup → create → start → verify → cleanup), making it easier to maintain and compare multi-language E2E tests later. ## Implementation The implementation ensures that the Java E2E test flow is consistent with the Go E2E test flow. A Docker-based environment is used with custom startup scripts that accept host, port, and other configuration parameters. E2E tests are triggered automatically on each PR via a GitHub Actions workflow and a script chain: the workflow builds images, starts the load-balanced stack with `docker-compose-loadbalancer.yml`, then runs `start-streampipes-client-e2e.sh -t java-client-e2e.sh`, which performs login, API key retrieval, and `mvn test` with the appropriate system properties; any failure fails the workflow so that code quality and load-balancing behavior are continuously monitored. GitHub link: https://github.com/apache/streampipes/discussions/4216 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
