[
https://issues.apache.org/jira/browse/BEAM-7003?focusedWorklogId=512332&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-512332
]
ASF GitHub Bot logged work on BEAM-7003:
----------------------------------------
Author: ASF GitHub Bot
Created on: 16/Nov/20 12:57
Start Date: 16/Nov/20 12:57
Worklog Time Spent: 10m
Work Description: piotr-szuberski commented on a change in pull request
#13026:
URL: https://github.com/apache/beam/pull/13026#discussion_r524248460
##########
File path: sdks/java/io/kafka/build.gradle
##########
@@ -65,26 +76,68 @@ dependencies {
testCompile library.java.junit
testCompile library.java.powermock
testCompile library.java.powermock_mockito
+ testCompile library.java.testcontainers_kafka
testRuntimeOnly library.java.slf4j_jdk14
testRuntimeOnly project(path: ":runners:direct-java", configuration:
"shadow")
- kafkaVersion210 "org.apache.kafka:kafka-clients:2.1.0"
+ kafkaVersions.each {"kafkaVersion$it.key"
"org.apache.kafka:kafka-clients:$it.value"}
Review comment:
That's super useful! Thank you!
I can't see any Kafka clients in the created jar. Here is the output of
:sdks:java:io:kafka:dependencies
```
compileClasspath - Compile classpath for source set 'main'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT
FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 ->
2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
compileOnly - Compile only dependencies for source set 'main'. (n)
No dependencies
default - Configuration for default artifacts. (n)
No dependencies
implementation - Implementation only dependencies for source set 'main'. (n)
No dependencies
runtimeClasspath - Runtime classpath of source set 'main'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT
FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 ->
2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
runtimeElements - Elements of runtime for main. (n)
No dependencies
runtimeOnly - Runtime only dependencies for source set 'main'. (n)
No dependencies
testAnnotationProcessor - Annotation processors and their dependencies for
source set 'test'.
No dependencies
testCompileClasspath - Compile classpath for source set 'test'.
\--- org.apache.beam:beam-sdks-java-io-kafka:2.27.0-SNAPSHOT
+--- org.checkerframework:checker-qual:3.7.0 FAILED
+--- org.apache.beam:beam-vendor-guava-26_0-jre:0.1
+--- org.apache.beam:beam-sdks-java-core:2.27.0-SNAPSHOT FAILED
+--- org.apache.beam:beam-sdks-java-expansion-service:2.27.0-SNAPSHOT
FAILED
+--- org.slf4j:slf4j-api:1.7.30
+--- joda-time:joda-time:2.10.5
+--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
+--- com.fasterxml.jackson.core:jackson-databind:2.10.2
| +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2
| \--- com.fasterxml.jackson.core:jackson-core:2.10.2
+--- org.springframework:spring-expression:4.3.18.RELEASE FAILED
\--- io.confluent:kafka-avro-serializer:5.3.2
+--- org.apache.avro:avro:1.8.1
| +--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- org.codehaus.jackson:jackson-mapper-asl:1.9.13
| | \--- org.codehaus.jackson:jackson-core-asl:1.9.13
| +--- com.thoughtworks.paranamer:paranamer:2.7
| +--- org.xerial.snappy:snappy-java:1.1.1.3
| +--- org.apache.commons:commons-compress:1.8.1
| +--- org.tukaani:xz:1.5
| \--- org.slf4j:slf4j-api:1.7.7 -> 1.7.30
+--- io.confluent:kafka-schema-registry-client:5.3.2
| +--- io.confluent:common-config:5.3.2
| | +--- io.confluent:common-utils:5.3.2
| | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| | | \--- com.101tec:zkclient:0.10
| | | \--- org.slf4j:slf4j-api:1.6.1 -> 1.7.30
| | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30
| +--- org.apache.avro:avro:1.8.1 (*)
| +--- com.fasterxml.jackson.core:jackson-databind:2.9.10.1 ->
2.10.2 (*)
| \--- io.confluent:common-utils:5.3.2 (*)
+--- io.confluent:common-config:5.3.2 (*)
\--- io.confluent:common-utils:5.3.2 (*)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 512332)
Time Spent: 4.5h (was: 4h 20m)
> Test KafkaIO against different versions of Kafka
> ------------------------------------------------
>
> Key: BEAM-7003
> URL: https://issues.apache.org/jira/browse/BEAM-7003
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
> Reporter: Ismaël Mejía
> Assignee: Piotr Szuberski
> Priority: P3
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> KafkaIO is supposed to support multiple versions of Kafka (> 0.9) but our
> current test infrastructure only validates this against the version 1.0.0. We
> should find a mecanism to parametrize KafkaIO tests with different versions
> of Kafka to ensure that multi-version support works as expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
