kfaraz opened a new pull request, #18147:
URL: https://github.com/apache/druid/pull/18147
### Description
This patch adds a unit test framework that can be used to run Druid cluster
simulations.
#### Salient features:
- Run IT-equivalent tests but much faster
- Define cluster setup in the test class itself
- Complete control over all aspects of each service
- Verify interplay of services for a new or updated feature
- Scale and stress testing of services
- Easily debug failing tests
- Discover race conditions due to the fast nature of the test
### Design
- Run each Druid service in an embedded mode, that internally just uses the
corresponding `Cli*` runnable
- Run embedded Zookeeper, metadata store (Derby), Kafka server
- Write out task logs and segments to temp directory
### Changes
- Wire up `JettyServerInitializer` properly so that multiple Jetty servers
may be bound correctly within the same JVM
- Add `OverlordClient.postSupervisor`
- Add `EmbeddedZookeeper` which uses the existing `CuratorTestBase`
- Add `EmbeddedKafkaServer` which mostly copies code from `TestBroker` in
kafka extension
- Add `EmbeddedDruidServer` and various implementations
### Usage
1. Define a cluster
2. Create a `@Rule` or `@ClassRule`
3. Write a `@Test` method
4. Interact with the cluster resources directly
<details>
<summary><b>Code snippet</b></summary>
```java
private static final EmbeddedDruidCluster CLUSTER
= EmbeddedDruidCluster.builder()
.with(new EmbeddedCoordinator())
.with(EmbeddedIndexer.withProps(Map.of("druid.worker.capacity", "25")))
.with(new EmbeddedOverlord())
.with(new EmbeddedHistorical())
.with(new EmbeddedBroker())
.withDb()
.build();
@ClassRule
public static final RuleChain CLUSTER_RULE_CHAIN = CLUSTER.ruleChain();
@Test
public void test_runIndexTask_forInlineDatasource()
{
final String inlineData =
"time,item,value\n2025-06-01,shirt,105\n2025-06-02,jeans,210";
final Task task = createIndexTaskForInlineData(TestDataSource.WIKI,
txnData10Days);
final String taskId = task.getId();
// Submit task and verify that it has succeeded
getResult(OVERLORD.client().runTask(taskId, task));
OVERLORD.waitUntilTaskFinishes(taskId);
final TaskStatusResponse currentStatus = getResult(
OVERLORD.client().taskStatus(taskId)
);
Assert.assertEquals(TaskState.SUCCESS,
currentStatus.getStatus().getStatusCode());
// Get the segment metadata published by the task
final List<DataSegment> segments = new ArrayList<>(
OVERLORD.segmentsMetadataStorage().retrieveAllUsedSegments(TestDataSource.WIKI,
null)
);
// Query the data
final Object result = getResult(
CLUSTER.anyBroker().submitSqlQuery(
new ClientSqlQuery("SELECT * FROM wiki", null, true, true, true,
Map.of(), List.of())
)
);
}
```
</details>
### Pending work
- Iron out some flakes
- Clean up some of the files
<hr>
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
(Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
