This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3dfedf69cc39 [SPARK-49842][BUILD] Add `byte-buddy` dependency for
modules that depend on `mockito-core` to ensure `sbt test` uses the correct
`byte-buddy` with Java 21
3dfedf69cc39 is described below
commit 3dfedf69cc3966083ac4fce9245dedab53c560a1
Author: yangjie01 <[email protected]>
AuthorDate: Fri Oct 4 11:21:19 2024 +0800
[SPARK-49842][BUILD] Add `byte-buddy` dependency for modules that depend on
`mockito-core` to ensure `sbt test` uses the correct `byte-buddy` with Java 21
### What changes were proposed in this pull request?
This pr add `byte-buddy` dependency for modules that depend on
`mockito-core` to ensure `sbt test` uses the correct `byte-buddy`
### Why are the changes needed?
To ensure that `sbt test` can use the correct version of `byte-buddy`, I
have only observed this issue when using `sbt test` with Java 21. This issue
has not been observed when using `sbt test` with Java 17(Perhaps it just didn't
print the Warning Message) or `maven test` with Java 17/21(Maven can confirm
that there is no such issue).
Java 21 sbt daily test:
- https://github.com/apache/spark/actions/runs/11099131939/job/30881044324
```
WARNING: A Java agent has been loaded dynamically
(/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/bytebuddy/byte-buddy-agent/1.14.15/byte-buddy-agent-1.14.15.jar)
```
- https://github.com/apache/spark/actions/runs/11099131939/job/30881045891
```
WARNING: A Java agent has been loaded dynamically
(/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/bytebuddy/byte-buddy-agent/1.14.15/byte-buddy-agent-1.14.15.jar)
```
- https://github.com/apache/spark/actions/runs/11099131939/job/30881047740
```
WARNING: A Java agent has been loaded dynamically
(/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/bytebuddy/byte-buddy-agent/1.14.15/byte-buddy-agent-1.14.15.jar)
```
We can see that `byte-buddy-agent-1.14.15.jar` is being used in the sbt
tests with Java 21, but the version defined in the `dependencyManagement` of
the parent `pom.xml` is `1.14.17`:
https://github.com/apache/spark/blob/3093ad68d2a3c6bab9c1605381d27e700766be22/pom.xml#L1221-L1231
Inconsistent dependency versions pose potential risks, such as the
possibility that maven tests may pass while sbt tests may fail. Therefore, we
should correct it.Meanwhile, for consistency, I have added a test dependency on
`byte-buddy` for all modules that depend on `mockito-core`.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass GitHub Actions
- Manual inspection shows that `sbt test` with Java 21 is using version
1.14.17 of `byte-buddy-agent`
-
https://github.com/LuciferYang/spark/actions/runs/11072600528/job/30922654197
-
https://github.com/LuciferYang/spark/actions/runs/11072600528/job/30922654969
-
https://github.com/LuciferYang/spark/actions/runs/11072600528/job/30922655852
-
https://github.com/LuciferYang/spark/actions/runs/11072600528/job/30922656840
```
WARNING: A Java agent has been loaded dynamically
(/home/runner/.cache/coursier/v1/https/maven-central.storage-download.googleapis.com/maven2/net/bytebuddy/byte-buddy-agent/1.14.17/byte-buddy-agent-1.14.17.jar)
```
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #48281 from LuciferYang/byte-buddy.
Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
common/network-common/pom.xml | 10 ++++++++++
common/network-shuffle/pom.xml | 10 ++++++++++
common/unsafe/pom.xml | 10 ++++++++++
connector/kafka-0-10-sql/pom.xml | 10 ++++++++++
connector/kafka-0-10-token-provider/pom.xml | 10 ++++++++++
connector/kafka-0-10/pom.xml | 10 ++++++++++
connector/kinesis-asl/pom.xml | 10 ++++++++++
core/pom.xml | 10 ++++++++++
launcher/pom.xml | 10 ++++++++++
mllib-local/pom.xml | 10 ++++++++++
mllib/pom.xml | 10 ++++++++++
repl/pom.xml | 10 ++++++++++
resource-managers/yarn/pom.xml | 10 ++++++++++
sql/catalyst/pom.xml | 10 ++++++++++
sql/connect/server/pom.xml | 10 ++++++++++
sql/core/pom.xml | 10 ++++++++++
sql/hive-thriftserver/pom.xml | 10 ++++++++++
streaming/pom.xml | 10 ++++++++++
18 files changed, 180 insertions(+)
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index cdb5bd72158a..cbe4836b58da 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -194,6 +194,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
</dependencies>
<build>
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index 0f7036ef746c..49e6e0847615 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -113,6 +113,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
diff --git a/common/unsafe/pom.xml b/common/unsafe/pom.xml
index a5ef9847859a..cf1530127330 100644
--- a/common/unsafe/pom.xml
+++ b/common/unsafe/pom.xml
@@ -104,6 +104,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.scalacheck</groupId>
<artifactId>scalacheck_${scala.binary.version}</artifactId>
diff --git a/connector/kafka-0-10-sql/pom.xml b/connector/kafka-0-10-sql/pom.xml
index 35f58134f1a8..66e1c24e821c 100644
--- a/connector/kafka-0-10-sql/pom.xml
+++ b/connector/kafka-0-10-sql/pom.xml
@@ -148,6 +148,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.scalacheck</groupId>
<artifactId>scalacheck_${scala.binary.version}</artifactId>
diff --git a/connector/kafka-0-10-token-provider/pom.xml
b/connector/kafka-0-10-token-provider/pom.xml
index 2b2707b9da32..3cbfc34e7d80 100644
--- a/connector/kafka-0-10-token-provider/pom.xml
+++ b/connector/kafka-0-10-token-provider/pom.xml
@@ -64,6 +64,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-runtime</artifactId>
diff --git a/connector/kafka-0-10/pom.xml b/connector/kafka-0-10/pom.xml
index 1b26839a371c..a42410e6ce88 100644
--- a/connector/kafka-0-10/pom.xml
+++ b/connector/kafka-0-10/pom.xml
@@ -119,6 +119,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-tags_${scala.binary.version}</artifactId>
diff --git a/connector/kinesis-asl/pom.xml b/connector/kinesis-asl/pom.xml
index 9a7f40443bbc..7eba26ffdff7 100644
--- a/connector/kinesis-asl/pom.xml
+++ b/connector/kinesis-asl/pom.xml
@@ -81,6 +81,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.scalacheck</groupId>
<artifactId>scalacheck_${scala.binary.version}</artifactId>
diff --git a/core/pom.xml b/core/pom.xml
index 19f58940ed94..7805a3f37ae5 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -393,6 +393,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.scalacheck</groupId>
<artifactId>scalacheck_${scala.binary.version}</artifactId>
diff --git a/launcher/pom.xml b/launcher/pom.xml
index c47244ff887a..e8feb7b68455 100644
--- a/launcher/pom.xml
+++ b/launcher/pom.xml
@@ -57,6 +57,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
diff --git a/mllib-local/pom.xml b/mllib-local/pom.xml
index ecfe45f046f2..3b35a481adb1 100644
--- a/mllib-local/pom.xml
+++ b/mllib-local/pom.xml
@@ -52,6 +52,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-tags_${scala.binary.version}</artifactId>
diff --git a/mllib/pom.xml b/mllib/pom.xml
index 4f983a325a0c..c342519ca428 100644
--- a/mllib/pom.xml
+++ b/mllib/pom.xml
@@ -117,6 +117,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.binary.version}</artifactId>
diff --git a/repl/pom.xml b/repl/pom.xml
index 831379467a29..1a1c6b92c922 100644
--- a/repl/pom.xml
+++ b/repl/pom.xml
@@ -82,6 +82,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-tags_${scala.binary.version}</artifactId>
diff --git a/resource-managers/yarn/pom.xml b/resource-managers/yarn/pom.xml
index 694d81b3c25e..770a550030f5 100644
--- a/resource-managers/yarn/pom.xml
+++ b/resource-managers/yarn/pom.xml
@@ -156,6 +156,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<!--
Jersey 1 dependencies only required for YARN integration testing.
Creating a YARN cluster
diff --git a/sql/catalyst/pom.xml b/sql/catalyst/pom.xml
index ac944eb9d807..7ce4609de51f 100644
--- a/sql/catalyst/pom.xml
+++ b/sql/catalyst/pom.xml
@@ -72,6 +72,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
diff --git a/sql/connect/server/pom.xml b/sql/connect/server/pom.xml
index 12e3ed903043..d0d982934d2c 100644
--- a/sql/connect/server/pom.xml
+++ b/sql/connect/server/pom.xml
@@ -240,6 +240,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
diff --git a/sql/core/pom.xml b/sql/core/pom.xml
index 4352c44a4fed..972cf76d2753 100644
--- a/sql/core/pom.xml
+++ b/sql/core/pom.xml
@@ -226,6 +226,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
diff --git a/sql/hive-thriftserver/pom.xml b/sql/hive-thriftserver/pom.xml
index 6a352f8a530d..d066e235ebea 100644
--- a/sql/hive-thriftserver/pom.xml
+++ b/sql/hive-thriftserver/pom.xml
@@ -138,6 +138,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>net.sf.jpam</groupId>
<artifactId>jpam</artifactId>
diff --git a/streaming/pom.xml b/streaming/pom.xml
index 85a4d268d2a2..704886bfdd1f 100644
--- a/streaming/pom.xml
+++ b/streaming/pom.xml
@@ -121,6 +121,16 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>net.bytebuddy</groupId>
+ <artifactId>byte-buddy-agent</artifactId>
+ <scope>test</scope>
+ </dependency>
</dependencies>
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
