This is an automated email from the ASF dual-hosted git repository.
agrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new a0734dcc Make it possible to run 'make benchmark-%' using jvm 17+
(#823)
a0734dcc is described below
commit a0734dcc67857497fafbe7224f861e3a20d3075b
Author: Emil Ejbyfeldt <[email protected]>
AuthorDate: Tue Aug 13 18:37:22 2024 +0200
Make it possible to run 'make benchmark-%' using jvm 17+ (#823)
When using jvm 17+ spark needs extra jvm args to avoid getting errors
like
```
[ERROR] Failed to execute goal
org.codehaus.mojo:exec-maven-plugin:3.2.0:java (default-cli) on project
comet-spark-spark3.4_2.12: An exception occurred while executing the Java
class. class org.apache.spark.storage.StorageUtils$ (in unnamed module
@0x35b75242) cannot access class sun.nio.ch.DirectBuffer (in module java.base)
because module java.base does not export sun.nio.ch to unnamed module
@0x35b75242 -> [Help 1]
```
These args are already present inside the main pom.xml. To avoid
duplicating the args we using maven to extract them. In order avoid
slowing down the Makefile the args are defined as a function and
therefore only evaluated when they are needed.
---
Makefile | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index 3715aac3..d4e2887f 100644
--- a/Makefile
+++ b/Makefile
@@ -17,6 +17,10 @@
.PHONY: all core jvm test clean release-linux release bench
+define spark_jvm_17_extra_args
+$(shell ./mvnw help:evaluate -Dexpression=extraJavaTestArgs | grep -v '\[')
+endef
+
all: core jvm
core:
@@ -80,7 +84,7 @@ release:
release-nogit:
cd native && RUSTFLAGS="-Ctarget-cpu=native" cargo build --release
./mvnw install -Prelease -DskipTests $(PROFILES)
-Dmaven.gitcommitid.skip=true
-benchmark-%: clean release
- cd spark && COMET_CONF_DIR=$(shell pwd)/conf MAVEN_OPTS='-Xmx20g'
../mvnw exec:java -Dexec.mainClass="$*" -Dexec.classpathScope="test"
-Dexec.cleanupDaemonThreads="false" -Dexec.args="$(filter-out
$@,$(MAKECMDGOALS))" $(PROFILES)
+benchmark-%: release
+ cd spark && COMET_CONF_DIR=$(shell pwd)/conf MAVEN_OPTS='-Xmx20g ${call
spark_jvm_17_extra_args}' ../mvnw exec:java -Dexec.mainClass="$*"
-Dexec.classpathScope="test" -Dexec.cleanupDaemonThreads="false"
-Dexec.args="$(filter-out $@,$(MAKECMDGOALS))" $(PROFILES)
.DEFAULT:
@: # ignore arguments provided to benchmarks e.g. "make benchmark-foo
-- --bar", we do not want to treat "--bar" as target
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
