[
https://issues.apache.org/jira/browse/THRIFT-6324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jens Geyer resolved THRIFT-6324.
--------------------------------
Fix Version/s: 0.26.0
Assignee: Jens Geyer
Resolution: Fixed
> Cut the AppVeyor build matrix and build in parallel
> ---------------------------------------------------
>
> Key: THRIFT-6324
> URL: https://issues.apache.org/jira/browse/THRIFT-6324
> Project: Thrift
> Issue Type: Improvement
> Components: Build Process
> Reporter: Jens Geyer
> Assignee: Jens Geyer
> Priority: Major
> Fix For: 0.26.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> h3. Problem
> Six jobs, roughly 3 h 07 min of job time squeezed into 1 h 45 min of wall
> clock. Measured on build 0.26.0.10733 and cross-checked against 0.26.0.10724,
> a different pull request with near-identical timings.
> *The Python matrix is five nearly identical builds.* Five of the six jobs are
> MSVC2022 builds that differ only in {{PYTHON_VERSION}} (3.10 through 3.14).
> Each one builds the whole C++ library, the compiler and every test executable
> -- 21 min 27 s -- in order to run 12 Python tests that account for 112 s of
> the 248 s {{ctest}} run. The other 54 tests are C++ and identical across all
> five jobs.
> GitHub Actions already runs the full 3.10 to 3.14 matrix twice, in
> {{lib-python}} on ubuntu-24.04 and in {{lib-python-macos}}, both off the same
> YAML anchor in {{.github/workflows/build.yml}}. What AppVeyor adds on top is
> the MSVC build of the {{fastbinary}} C extension ({{lib/py/setup.py}}) and
> the Windows behaviour of the pure-Python tests. Neither of those changes
> between 3.11, 3.12 and 3.13 -- breakage shows up at the edges of the range,
> not in the middle.
> *Nothing is built in parallel.* {{cmake --build}} is invoked without
> {{--parallel}} in both scripts:
> * MSVC: MSBuild builds one project at a time. The log shows some 40 test
> executables completing strictly in sequence, 3 to 15 s apart.
> * MinGW: {{mingw32-make}} runs single-threaded for 34 min 13 s of that job's
> 46 min.
> h3. Change
> * {{appveyor.yml}}: keep Python 3.11 and 3.14, drop 3.10, 3.12 and 3.13. 3.11
> rather than 3.10 as the lower edge because 3.10 reaches end of life on
> 2026-10-31, so the matrix holds for a year; 3.14 becomes 3.15 once the
> AppVeyor image carries it.
> * {{build/appveyor/MSVC-appveyor-full.bat}} and
> {{build/appveyor/MINGW-appveyor-full.bat}}: pass {{--parallel
> %NUMBER_OF_PROCESSORS%}} to {{cmake --build}}. An explicit number rather than
> a bare {{--parallel}}, which for the Makefile generator means an unbounded
> {{make -j}}.
> * {{build/appveyor/win_showenv.bat}}: report {{NUMBER_OF_PROCESSORS}}, so the
> logs say how much parallelism a worker actually offers.
> Expected effect: job time down from about 3 h 07 min to about 1 h 42 min from
> the matrix change alone, plus whatever the parallel build adds on top of that.
> h3. Not in scope
> Running time is the smaller half of the problem. Over the last 23 finished
> builds the median time spent waiting in the queue was 6 h 26 min, against 1 h
> 43 min of actual building. That is a project setting on the AppVeyor side and
> is being raised separately with ASF Infra.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
