kevinjqliu opened a new pull request, #1445: URL: https://github.com/apache/datafusion-python/pull/1445
## Which issue does this PR close? Fixes OOM (`Killed process ... (rustc) total-vm:25086084kB, anon-rss:15361808kB`) during manylinux x86_64 release builds, where `rustc` consumed ~15 GB and exhausted the runner's memory. ## Rationale As the dependency tree has grown (DataFusion + Substrait + Arrow + object_store with aws/gcp/azure/http features), the release build's peak memory during LTO linking has exceeded what the GitHub runner can provide. ## What changes are included in this PR? **Cargo profile (`Cargo.toml`):** - Switch from fat LTO (`lto = true`) to thin LTO (`lto = "thin"`) -- this is the biggest win, reducing peak memory by ~50-70% since LLVM no longer needs to merge all bitcode into a single module - Increase `codegen-units` from 1 to 2 -- splits LLVM's workload, further reducing peak RSS **CI workflow (`.github/workflows/build.yml`):** - Add 8 GB swap to the `build-manylinux-x86_64` job as a safety net (matching the existing pattern in the aarch64 job) - Reduce `build-manylinux-aarch64` swap from 16 GB to 8 GB for consistency ## Tradeoffs Thin LTO + codegen-units=2 may produce binaries that are ~1-4% slower in micro-benchmarks vs fat LTO + codegen-units=1. In practice, this is unlikely to be measurable for a Python extension where the Python-Rust FFI boundary and PyArrow serialization dominate execution time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
