clintropolis opened a new pull request #10463:
URL: https://github.com/apache/druid/pull/10463
### Description
This PR adjust the Kafka integration tests to slightly adjust the order in
which the `druid.test.config.properties.kafka.test.property.*` overrides are
applied, allowing `bootstrap.servers` to be overridden in the docker tests to
point at Azure event hubs. Additionally, an option to disable transaction tests
(Azure Event Hubs does not support Kafka transactions) in
`ITKafkaIndexingServiceDataFormatTest`. Note that this PR does not actually add
any new integration tests, it just allows more configurability making this
stuff possible.
With these adjustments I was able to run
`ITKafkaIndexingServiceDataFormatTest` and
`ITKafkaIndexingServiceNonTransactionalSerializedTest` with all tests passing
by overriding the kafka configs with something like this:
```
-Ddruid.test.config.properties.kafka.test.property.security.protocol=SASL_SSL
-Ddruid.test.config.properties.kafka.test.property.sasl.mechanism=PLAIN
-Ddruid.test.config.properties.kafka.test.property.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule
required username=\"$ConnectionString\"
password=\"Endpoint=sb://{my-hub-name}.servicebus.windows.net/;SharedAccessKeyName={key-name};SharedAccessKey={shared-key}";"
-Ddruid.test.config.properties.kafka.test.property.bootstrap.servers={my-hub-name}.servicebus.windows.net:9093
```
For `ITKafkaIndexingServiceNonTransactionalParallelizedTest`, I encountered
an exception in the test itself with the admin client API trying to change the
partition count for `testKafkaIndexDataWithKafkaReshardSplit`:
```
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidRequestException: This most likely occurs
because of a request being malformed by the client library or the message was
sent to an incompatible broker. See the broker logs for more details.
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at
org.apache.druid.testing.utils.KafkaAdminClient.updatePartitionCount(KafkaAdminClient.java:82)
at
org.apache.druid.tests.indexer.AbstractStreamIndexingTest.testIndexWithStreamReshardHelper(AbstractStreamIndexingTest.java:427)
```
so it cannot be run successfully at this time (the other test,
`testKafkaIndexDataWithStartStopSupervisor` does succeed), and none of the
transactional tests can run since it isn't supported.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
