This is an automated email from the ASF dual-hosted git repository.
zeroshade pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-go.git
The following commit(s) were added to refs/heads/main by this push:
new 164c0c3b ci: run parquet/file alone on macOS instead of serializing
all packages (#1301)
164c0c3b is described below
commit 164c0c3be6cbf33f6e7dcf55f1c6f2f412a8920f
Author: Matt Topol <[email protected]>
AuthorDate: Tue Sep 8 14:58:41 2026 -0400
ci: run parquet/file alone on macOS instead of serializing all packages
(#1301)
### Rationale for this change
#1275 added `-p=1` for Darwin to keep the 7 GB macOS ARM64 runners from
OOM-killing `parquet/file`. That works, but it serializes *every*
parquet
package, across both the `assert` and `assert,noasm` passes, under
`-race`.
The cost is about 4.5 minutes per macOS job, which ate the entire
headroom
against the `timeout-minutes: 20` cap on the `macos` job.
Job durations on `main`, 631 samples over Aug 6 – Sep 8, split at the
#1275
merge on Sep 3:
| macOS ARM64 job | before avg | after avg | after max | cap |
| --- | --- | --- | --- | --- |
| Go 1.25 | 11.6m | 16.4m | 20.1m | 20 |
| Go 1.26 | 11.5m | 16.0m | 20.4m | 20 |
| Go 1.25 - CGO | 17.9m | 19.7m | 24.1m | 25 |
| Go 1.26 - CGO | 14.5m | 18.5m | 24.8m | 25 |
Linux is flat across the same window (AMD64 Debian −0.1/+0.2, ARM64
Debian
−0.1/+0.4), which isolates the change to the Darwin-only branch rather
than
new tests or runner drift.
Three macOS jobs on `main` have since been killed at the 20-minute mark
(20.1m, 20.4m, 20.4m), and the CGO jobs at 24.8m are one bad run from
the
same fate.
### What changes are included in this PR?
Only `parquet/file` holds the large-value regression tests that exhaust
the
runner, so give that one package its own `go test` invocation on Darwin
and
let the other 17 parquet packages keep their default parallelism. The
memory
fix from #1275 is preserved — `parquet/file` still never shares a runner
with
another parquet package — without paying for it across the whole tree.
Non-Darwin behaviour is byte-identical to before: two invocations over
`./...`, no `-p` flag.
### Are these changes tested?
- `bash -n ci/scripts/test.sh`
- `shellcheck ci/scripts/test.sh` — clean
- `shfmt -d ci/scripts/test.sh` — clean
- Dry-ran the package partition against this tree: on Darwin it yields 1
serial package (`.../parquet/file`) and 17 parallel, with no leakage of
`file` into the parallel set; on non-Darwin it yields exactly `./...`.
- `go test -tags assert -run XXX ./...` and `go test -tags assert,noasm
-run
XXX ./...` in `parquet/` both compile clean, confirming the invocation
form.
The real check is the macOS timings on this PR's own CI run.
### Are there any user-facing changes?
No.
---
ci/scripts/test.sh | 28 +++++++++++++++++++++-------
1 file changed, 21 insertions(+), 7 deletions(-)
diff --git a/ci/scripts/test.sh b/ci/scripts/test.sh
index 21677047..411fd35a 100755
--- a/ci/scripts/test.sh
+++ b/ci/scripts/test.sh
@@ -75,15 +75,29 @@ popd
pushd "${source_dir}/parquet"
-parquet_test_args=("${test_args[@]}")
+# parquet/file holds the large-value regression tests, which are what exhaust
+# the 7 GB macOS ARM64 runners when they run alongside other packages. Give
+# that one package its own invocation there and let the rest of parquet keep
+# its default parallelism: serializing every package instead (-p=1) cost the
+# macOS jobs about 4.5 minutes each and pushed them into their CI timeout.
+parquet_pkgs=("./...")
+serial_pkgs=()
if [[ "$(go env GOOS)" = "darwin" ]]; then
- # Keep package-level memory bounded on the 7 GB macOS ARM64 runners.
- parquet_test_args+=("-p=1")
+ parquet_pkgs=()
+ while IFS= read -r pkg; do
+ if [[ "${pkg}" = */parquet/file ]]; then
+ serial_pkgs+=("${pkg}")
+ else
+ parquet_pkgs+=("${pkg}")
+ fi
+ done < <(go list ./...)
fi
-go test "${parquet_test_args[@]}" -tags assert ./...
-
-# run the tests again but with the noasm tag
-go test "${parquet_test_args[@]}" -tags assert,noasm ./...
+for parquet_tags in assert assert,noasm; do
+ go test "${test_args[@]}" -tags "${parquet_tags}" "${parquet_pkgs[@]}"
+ if [[ ${#serial_pkgs[@]} -gt 0 ]]; then
+ go test "${test_args[@]}" -tags "${parquet_tags}" "${serial_pkgs[@]}"
+ fi
+done
popd
