greyp9 commented on PR #9687: URL: https://github.com/apache/nifi/pull/9687#issuecomment-2634932870
@dan-s1 Firstly, thanks very much for your code cleanup efforts in general, and here in particular. Nice to know someone is providing momentum for this aspect of the project. I pulled your PR commits and ran the Kafka ITs, and I see the same errors as the CI. I agree that the `kafka-cp` version bump is a good idea. It might be nice at some point to hoist that out of the code, but no need here. There is a particular test `KafkaConnectivityIT` that I used to verify the testcontainers harness (avoids NiFi), so I started there. Since that worked, I moved on. I was able to narrow things down by running `Kafka3ConnectionServiceSSLIT` by itself, and noted this error in the Kafka container startup log: ``` 2025-02-04 13:54:34 ===> User 2025-02-04 13:54:34 uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) 2025-02-04 13:54:34 ===> Configuring ... 2025-02-04 13:54:34 Running in KRaft mode... 2025-02-04 13:54:37 ===> Running preflight checks ... 2025-02-04 13:54:37 ===> Check if /var/lib/kafka/data is writable ... 2025-02-04 13:54:37 ===> Running in KRaft mode, skipping Zookeeper health check... 2025-02-04 13:54:37 ===> Using provided cluster id 4L6g3nShT-eMCtK--X86sw ... 2025-02-04 13:54:38 Exception in thread "main" java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9094,BROKER://0.0.0.0:9093': No security protocol defined for listener CONTROLLER at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:216) at kafka.server.KafkaConfig.listeners(KafkaConfig.scala:1104) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:754) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:545) at kafka.tools.StorageTool$. ``` This suggested that the use of `ConfluentKafkaContainer` somehow added another listener, which needed to be accounted for in the startup configuration for the non-plaintext Kafka instances. Here is a patch you can apply to your PR (3 lines). The first is the image version update. The other two configure the Kafka container to start with the additional needed listener configuration. With these changes, I ran: `greyp9/nifi/nifi-extension-bundles/nifi-kafka-bundle % mvn verify -P integration-tests` successfully. As an aside, my use of `org.testcontainers.containers.KafkaContainer` with `confluentinc/cp-kafka` was wrong. I gather from reading up, and the test output, that `org.testcontainers.kafka.ConfluentKafkaContainer` should have been used from the start. So thanks for fixing this defect! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
