Krisztian Szucs created ARROW-16589:
---------------------------------------
Summary: [CI][Dev] Make tasks.yml easier to maintain
Key: ARROW-16589
URL: https://issues.apache.org/jira/browse/ARROW-16589
Project: Apache Arrow
Issue Type: New Feature
Components: Continuous Integration, Developer Tools
Reporter: Krisztian Szucs
I think {{dev/tasks/tasks.yml}} has reached its limits by using jinja2
templated yml.
We should think about a better way to define crossbow jobs while:
- keeping it readable
- in a dialect which is natively supported by editors
- while supporting tasks parametrization
Just one idea is to use python files containing python objects, e.g.:
{code}
Task(
name="wheel-macos-big-sur-cp38-arm64",
ci="github",
template="python-wheels/github.osx.arm64.yml",
params=dict(
arch="arm64",
arrow_simd_level="DEFAULT",
python_version="3.8",
macos_deployment_target="11.0"
),
artifacts=[
"pyarrow-{no_rc_version}-cp38-cp38-macosx_11_0_arm64.whl"
]
)
{code}
where {{Task}} would be the crossbow task class (which could be refactored to
use pydantic or another alternative for less boilerplate). Of course porting to
the tasks definitions to plain python could make the situation even worse by
accessing too many scripting utilities. We could try a dynamic config language
which sits between yaml and python like HCL.
[~kou] what syntax would you be comfortable to work with? Do you have any
alternatives we could use?
cc [~amol-] [~raulcd] [~assignUser]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
