[
https://issues.apache.org/jira/browse/KAFKA-12767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342014#comment-17342014
]
John Roesler commented on KAFKA-12767:
--------------------------------------
Marked "affects version" to 3.0.0, since this problem was first introduced in
trunk during the 3.0 development cycle. All it means is that we don't want or
need to try cherry-picking the fixes to older branches.
> Properly set Streams system test runtime classpath
> --------------------------------------------------
>
> Key: KAFKA-12767
> URL: https://issues.apache.org/jira/browse/KAFKA-12767
> Project: Kafka
> Issue Type: Task
> Components: streams, system tests
> Affects Versions: 3.0.0
> Reporter: John Roesler
> Priority: Minor
> Labels: newbie++
>
> Some of the streams system tests started to fail recently when we stopped
> exporting our transitive dependencies in the test jar.
> [~lct45] was kind enough to submit
> [https://github.com/apache/kafka/pull/10631] to get the system tests running
> again, but that PR is only a stop-gap.
> The real solution is to properly package the transitive dependencies and make
> them available to the system test runtime.
> Here is the reason: PR#10631 gets past the issue by removing runtime usages
> on Hamcrest, but Hamcrest is still present in the compiletime classpath.
> Tomorrow, a new PR could add a reference to Hamcrest, and all the unit and
> integration tests would pass, but we would again see the mysterious system
> test failures. Only after another round of costly investigation would we
> realize the root cause, and we might again decide to just patch the test to
> remove the reference.
> It would be far better in the long run to fix the underlying condition: the
> difference between the compiletime and runtime classpaths for the system
> tests.
>
> To help get you started, I'll share some of the groundwork for this task,
> which I put together while trying to understand the nature of the problem.
> The first step is to actually copy the transitive dependencies. We had to
> stop placing these dependencies in the `dependant-libs` build directory
> because that results in us actually shipping those dependencies with our
> releases. Copying a similar mechanism from the `:core` project, we can add a
> new build directory (arbitrarily: `dependant-testlibs`), and again copy the
> runtime dependencies there. Here is a commit in my fork that does just that:
> [https://github.com/vvcephei/kafka/commit/8d4552dee05f2a963b8072b86aae756415ea2482]
> The next step is to place those jars on the classpath of the system test
> code. The mechanism for that is `kafka-run-class.sh`:
> [https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh]
> A specific example of this is the special logic for upgrade tests:
> # If we are running upgrade tests, then we set the artifact directories to
> the relevant version. Otherwise, we use the current build artifacts.
> [https://github.com/apache/kafka/blob/fc405d792de12a50956195827eaf57bbf64444c9/bin/kafka-run-class.sh#L77-L85]
> # The, here's where we specifically pull in Hamcrest from those build
> artifact directories:
> [https://github.com/apache/kafka/blob/fc405d792de12a50956195827eaf57bbf64444c9/bin/kafka-run-class.sh#L128-L136]
> It seems to me that since Hamcrest is actually a more general dependency of
> the tests, we might as well pack it up in `dependant-testlibs` and then pull
> it into the classpath from there any time we're running tests. It looks like
> we ought to set `streams_dependant_clients_lib_dir` to `dependant-testlibs`
> any time `INCLUDE_TEST_JARS` is `true`. But if we do have
> `UPGRADE_KAFKA_STREAMS_TEST_VERSION` set, then it should override the lib
> dir, since those artifacts to copy over the transitive dependencies for those
> older versions already.
>
> Although the proposed fix itself is pretty small, I think the testing will
> take a few days. You might want to just put some `echo` statements in
> kafka-run-class.sh to see what jars are included on the classpath while you
> run different tests, both locally using Docker, and remotely using Jenkins.
> I marked this ticket as `newbie++` because you don't need deep knowledge of
> the codebase to tackle this ticket, just a high pain tolerance for digging
> though gradle/docker/bash script debugging to make sure the right bits make
> it into the system tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
