+1 (non-binding) as reducing test execution time will improve developer productivity!
On Mon, Nov 17, 2025 at 5:12 AM Maximilian Michels <[email protected]> wrote: > Hi, > > What do you think about the following change? > https://github.com/apache/iceberg/pull/13675 > > Background: > > We build modules in parallel, but test execution is serial in Iceberg. > I found that increasing the test parallelism for a module greatly > reduces the test execution time. For example, on my machine (MacBook > Pro M1 Max) it takes ~32 minutes to execute the Flink tests via: > > ./gradlew :iceberg-flink:iceberg-flink-2.1:check > > But if I increase `maxParallelForks` in the test section of the Flink > build file to 8, it will only take 9 minutes, which is a speedup of > 3.5x. > > Instead of doing this manually, we can do it via the property added in > above PR: > > ./gradlew :iceberg-flink:iceberg-flink-2.1:check -DtestParallelism=8 > > I also added an 'auto' config where we take half of the available CPUs, > e.g.: > > ./gradlew :iceberg-flink:iceberg-flink-2.1:check -DtestParallelism=auto > > It is important to note that any modules which already set > `maxParallelForks` will not be affected by this change. So we can > still enforce non-parallel test execution. Also, the change is opt-in > and doesn't change the default build configuration. > > Any objections against merging this functionality? > > Cheers, > Max >
