This is an automated email from the ASF dual-hosted git repository. alexey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/kudu.git
commit a1cc227df5d5dc7586d3971135643320682b2709 Author: Alexey Serbin <[email protected]> AuthorDate: Wed Jul 31 11:58:55 2024 -0700 [cmake] work around of Gradle-related issues Once Gradle started being invoked to build kudu-subprocess and kudu-hms JARs as dependency for the C++ server-side of the project, it's become an issue to get 100% success rate when building it from source. With the recent Gradle upgrade the build success rate has reduced further, it seems. I decided to address that by adding a few extra flags to the Gradle's invocations under cmake when building corresponding targets. Probably, there is a better way of addressing that (say, finding the root cause of the issues, reporting, and fixing them), but this simple solution works well for me. Also, this solution doesn't affect the build times of the project (even when using ccache) since the targets built with Gradle constitute a small part of the rest of the targets that are built concurrently in a very robust manner. Prior to this patch, Gradle would error out on follow-up invocations after the build process has been interrupted, not being able to access its build cache with an error like below: -------------------------------------------------- FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':kudu-hive:compileJava'. > Could not create service of type DefaultGeneralCompileCaches using GradleScopeCompileServices.createGeneralCompileCaches(). > Cannot lock Java compile cache (...) as it has already been locked by this process. -------------------------------------------------- Prior to this patch, building the project would fail sometimes when invoking 'cmake -jN' (where N > 1) with errors like below while Gradle was generating kudu-subprocess.jar and hms-plugin.jar concurrently: -------------------------------------------------- ... [ 4%] Generating ../../../bin/kudu-subprocess.jar ... [ 6%] Generating ../../../bin/hms-plugin.jar ... java.lang.ArrayIndexOutOfBoundsException: 6 at shadow.org.objectweb.asm.ClassReader.readShort(ClassReader.java:3573) ... at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56) at java.lang.Thread.run(Thread.java:750) FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':kudu-client:shadowJar'. > 6 ... BUILD FAILED in 1m 0s make[2]: *** [bin/hms-plugin.jar] Error 1 make[1]: *** [src/kudu/hms/CMakeFiles/hms_plugin_jar.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs.... java.lang.ArrayIndexOutOfBoundsException: 6 at shadow.org.objectweb.asm.ClassReader.readShort(ClassReader.java:3573) ... at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:624) at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnabl e.run(ThreadFactoryImpl.java:56) at java.lang.Thread.run(Thread.java:750) FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':kudu-subprocess:shadowJar'. > 6 ... BUILD FAILED in 1m 3s make[2]: *** [bin/kudu-subprocess.jar] Error 1 make[1]: *** [src/kudu/subprocess/CMakeFiles/subprocess_jar.dir/all] Error 2 make: *** [all] Error 2 -------------------------------------------------- Change-Id: Ib7d1637854233592ede5388bfc2287df96ec9865 Reviewed-on: http://gerrit.cloudera.org:8080/21626 Reviewed-by: Abhishek Chennaka <[email protected]> Tested-by: Alexey Serbin <[email protected]> --- CMakeLists.txt | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 0f6201447..3ebdd1b1a 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -109,8 +109,24 @@ set(JAVA_DIR ${CMAKE_CURRENT_SOURCE_DIR}/java) # This solves the problem of escaped spaces in the EXTRA_GRADLE_FLAGS environment variable. list(APPEND GRADLE_FLAGS $ENV{EXTRA_GRADLE_FLAGS}) separate_arguments(GRADLE_FLAGS) -# We always want Gradle to use the plain console and quiet flag when called from cmake. -list(APPEND GRADLE_FLAGS --quiet --console=plain) +# When invoking Gradle: +# * Log errors only, reducing the verbosity of the Gradle's output +# * Make the output better fit for plain consoles, avoiding garbled logs +# * Avoid building projects in parallel: when the kudu-subprocess.jar and +# the hms-plugin.jar targets are built in parallel, it often ends up with +# java.lang.ArrayIndexOutOfBoundsException while building shadowJar +# for corresponding Java dependencies, and while it's not yet clear whether +# the issue is in shadowJar implementation or elsewhere, it's quite simple +# to address the issue by not building multiple Gradle targets concurrently +# * Disable the build cache: sometimes, once a build process is interrupted, +# follow-up Gradle invocations end up with an error like below, and while +# it's not yet clear whether the issue is in not properly invoking Gradle +# or it's a bug in signal handling elsewhere, it's quite simple to address +# the issue by not using the Gradle's build cache +# +# Cannot lock Java compile cache (...) as it has already been locked by this process. +# +list(APPEND GRADLE_FLAGS --quiet --console=plain --no-parallel --no-build-cache) list(REMOVE_DUPLICATES GRADLE_FLAGS) # Initialize Gradle to ensure the wrapper is downloaded.
