[
https://issues.apache.org/jira/browse/ARROW-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538084#comment-17538084
]
Jacob Wujciak-Jens commented on ARROW-16589:
--------------------------------------------
One aspect I would like to add as a requirement for any alternative is the
ability to provide documentation, both in-line and outside of the code e.g.
generated doc files or CLI help pages describing e.g. the purpose/use case of
each task with examples, possible pre-requisites, caveats etc.
Because at the moment we only have docs describing the overall usage of
archery/crossbow but it is not very apparent to contributors which builds they
should run locally/trigger on PRs and there is no place this information can be
found (docker-compse/tasks.yml only provide clues for the most part).
> [CI][Dev] Make tasks.yml easier to maintain
> -------------------------------------------
>
> Key: ARROW-16589
> URL: https://issues.apache.org/jira/browse/ARROW-16589
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Continuous Integration, Developer Tools
> Reporter: Krisztian Szucs
> Priority: Major
>
> I think {{dev/tasks/tasks.yml}} has reached its limits by using jinja2
> templated yml.
> We should think about a better way to define crossbow jobs while:
> - keeping it readable
> - in a dialect which is natively supported by editors
> - while supporting tasks parametrization
> Just one idea is to use python files containing python objects, e.g.:
> {code}
> Task(
> name="wheel-macos-big-sur-cp38-arm64",
> ci="github",
> template="python-wheels/github.osx.arm64.yml",
> params=dict(
> arch="arm64",
> arrow_simd_level="DEFAULT",
> python_version="3.8",
> macos_deployment_target="11.0"
> ),
> artifacts=[
> "pyarrow-{no_rc_version}-cp38-cp38-macosx_11_0_arm64.whl"
> ]
> )
> {code}
> where {{Task}} would be the crossbow task class (which could be refactored to
> use pydantic or another alternative for less boilerplate). Of course porting
> to the tasks definitions to plain python could make the situation even worse
> by accessing too many scripting utilities. We could try a dynamic config
> language which sits between yaml and python like HCL.
> [~kou] what syntax would you be comfortable to work with? Do you have any
> alternatives we could use?
> cc [~amol-] [~raulcd] [~assignUser]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
