[
https://issues.apache.org/jira/browse/ARROW-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17537982#comment-17537982
]
Antoine Pitrou commented on ARROW-16589:
----------------------------------------
My experience writing configuration using Python (Buildbot configuration files)
is that it encourages contributors to write more and more dynamic functionality
that becomes hard to maintain.
> [CI][Dev] Make tasks.yml easier to maintain
> -------------------------------------------
>
> Key: ARROW-16589
> URL: https://issues.apache.org/jira/browse/ARROW-16589
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Continuous Integration, Developer Tools
> Reporter: Krisztian Szucs
> Priority: Major
>
> I think {{dev/tasks/tasks.yml}} has reached its limits by using jinja2
> templated yml.
> We should think about a better way to define crossbow jobs while:
> - keeping it readable
> - in a dialect which is natively supported by editors
> - while supporting tasks parametrization
> Just one idea is to use python files containing python objects, e.g.:
> {code}
> Task(
> name="wheel-macos-big-sur-cp38-arm64",
> ci="github",
> template="python-wheels/github.osx.arm64.yml",
> params=dict(
> arch="arm64",
> arrow_simd_level="DEFAULT",
> python_version="3.8",
> macos_deployment_target="11.0"
> ),
> artifacts=[
> "pyarrow-{no_rc_version}-cp38-cp38-macosx_11_0_arm64.whl"
> ]
> )
> {code}
> where {{Task}} would be the crossbow task class (which could be refactored to
> use pydantic or another alternative for less boilerplate). Of course porting
> to the tasks definitions to plain python could make the situation even worse
> by accessing too many scripting utilities. We could try a dynamic config
> language which sits between yaml and python like HCL.
> [~kou] what syntax would you be comfortable to work with? Do you have any
> alternatives we could use?
> cc [~amol-] [~raulcd] [~assignUser]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
