methodmissing commented on pull request #12794: URL: https://github.com/apache/beam/pull/12794#issuecomment-692706541
Hi Luke, Apologies for the lag on this. TLDR I have the integration test against `KafkaIOTest` going but seem to not be able to force the `kafka-clients` module override. Executed with `./gradlew clean sdks:java:io:kafka:buildDependent` Main test task `:sdks:java:io:kafka:test` ``` <===========--> 86% EXECUTING [2m 7s] > IDLE > :sdks:java:io:expansion-service:shadowDistZip > :sdks:java:extensions:sql:expansion-service:shadowJar > :runners:spark:spotbugsMain > :sdks:java:io:kafka:test > Executing test org.apache.beam.sdk.io.kafka.KafkaIOTest <<<<<<<<<< > :sdks:java:testing:nexmark:spotbugsMain > :runners:spark:job-server:shadowJar > :sdks:java:io:kafka:test > 8 tests completed ``` Kafka client API 2.1.0 specific task `:sdks:java:io:kafka:kafkaVersion210Test` ``` <===========--> 88% EXECUTING [4m 2s] > :runners:spark:test > Executing test org.apache.beam...metrics.sink.SparkMetricsSinkTest > :sdks:java:extensions:sql:expansion-service:shadowJar > :sdks:java:testing:nexmark:test > 9 tests completed, 1 skipped > :sdks:java:testing:nexmark:test > Executing test org.apache.beam.sdk.nexmark.queries.QueryTest > :sdks:java:io:kafka:kafkaVersion210Test > Executing test org.apache.beam.sdk.io.kafka.KafkaIOTest <<<<<<<<<< > :sdks:java:io:kafka:kafkaVersion210Test > 1 test completed > :runners:spark:job-server:shadowJar > :runners:spark:test > 10 tests completed > :runners:spark:test > Executing test org.apache.beam.runners.spark.metrics.SparkMetricsPusherTest > :runners:spark:test > Executing test org.apache...streaming.TrackStreamingSourcesTest ``` ``` methodmissing:beam lourens$ ls -la /Users/lourens/src/github.com/Shopify/cdc/beam/sdks/java/io/kafka/build/reports/tests total 0 drwxr-xr-x 4 lourens staff 128 Sep 15 11:00 . drwxr-xr-x 5 lourens staff 160 Sep 15 11:06 .. drwxr-xr-x 7 lourens staff 224 Sep 15 11:06 kafkaVersion210Test <<<<<<<<<< drwxr-xr-x 7 lourens staff 224 Sep 15 11:04 test ``` Any failure in task `:sdks:java:io:kafka:test` means the 2.1.0 specific test doesn't run though. Given this is an integration test, units that fail should not run the dependent integration test either, that is default Gradle behavior and should be fine for this case as well. I opted to run the whole `KafkaIOTest` test with the new task as it exercises a larger net of the deserialization flows and doesn't add significantly to runtime. **!!! FAIL:** However I'm failing at being able to inject the 2.1.0 kafka-client module dependency for the integration test and I'm wondering if you have any obvious thoughts on what I've done wrong in specifying the test module dependency? ----------------------- I have some ideas (based on the python specific tasks and multi-version support) on how to support test tasks for all Kafka client API versions, but beyond the scope of this pull request. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
