[
https://issues.apache.org/jira/browse/BEAM-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815436#comment-16815436
]
Michael Luckey commented on BEAM-7003:
--------------------------------------
Hi [~iemejia],
after creating a POC implementation [1], I wanted to discuss this first.
Implementation of BEAM-4087 enables adhoc replacement of the used Kafka client
version. With this we change version for the complete build. So this could be
considered a test, whether we would be able to compile with different version.
This implementation replaces the dependency during runtime. So this seems to
reflect an user exchanging her Kafka client jar for some other version. But as
Kafka client promises compatibility anyway - apart from versions lower some
0.10.2?), what do we intend to test here? Or, asked diffently, why would an
user exchange the Kafka client jar anyway?
One obvious problem seems to be, that KafkaIOTest relies on mocks provided by
the client jar. So I wonder whether we will mostly fail on some incompatible
mock change, which is probably not, what is intended. Especially executing
these tests below fail on the test code referencing non existing api in
versions < 0.11. Which not necessarily has to imply KafkaIO would not work with
lower versions (also KafkaIO is referencing non existing classes, so will also
not work with those versions).
On the other hand, there might be nevertheless a use case in providing such
functionality. E.g. I could imagine it being valuable to check HDFS filesystem
with different Hadoop versions.
Coming back to KafkaIO, I tend to believe, that the thing we require is some
test against a 'real' cluster. I.e. testing KafkaIO against some docker
containers running those versions we want to support?
Thoughts? You might just paste below snippet to build gradle and try yourself
with e.g.
{noformat}
./gradlew -p sdks/java/io/kafka/ clean check --continue --no-build-cache
{noformat}
[1]
{code}
// to be appended to sdks/java/io/kafka/build.gradle
ext.testVersions = [
"0.9.0.1",
"0.10.0.1",
"0.11.0.2",
"1.1.0",
"2.0.0",
"2.1.0",
"2.2.0",
]
configurations {
variant1_0_0
}
dependencies {
variant1_0_0 library.java.kafka_clients
}
def previous = ''
testVersions.each {
configurations.create("variant${it.replaceAll('\\.', '_')}")
dependencies.add "variant${it.replaceAll('\\.', '_')}",
"org.apache.kafka:kafka-clients:$it"
task "test$it"(type: Test) { task ->
description = "Runs variant tests for $it"
group = 'verification'
classpath = classpath - configurations.variant1_0_0 +
configurations."variant${it.replaceAll('\\.', '_')}"
// do we care about order here? they arent executed in parallel. Even if,
does it matter here?
shouldRunAfter "test$previous"
}
previous = it
check.dependsOn("test$it")
}
{code}
> Test KafkaIO against different versions of Kafka
> ------------------------------------------------
>
> Key: BEAM-7003
> URL: https://issues.apache.org/jira/browse/BEAM-7003
> Project: Beam
> Issue Type: Improvement
> Components: io-java-kafka
> Reporter: Ismaël Mejía
> Priority: Minor
>
> KafkaIO is supposed to support multiple versions of Kafka (> 0.9) but our
> current test infrastructure only validates this against the version 1.0.0. We
> should find a mecanism to parametrize KafkaIO tests with different versions
> of Kafka to ensure that multi-version support works as expected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
