Source: https://bitbucket.org/fedoraqa/taskotron-trigger/branch/pony
Diff: https://phab.qadevel.cloud.fedoraproject.org/D872
This started as simple bike-shedding to make more sense in naming (so
everything is not named "Trigger"), but it went further :D
The main change here is, what I call "configurable trigger" - at the moment,
every time we want to add support for even the most basic new task (like the
package-specific task for docker), changes are needed in the trigger's source
code.
These changes add a concept of a "rules engine", that decides what tasks to
schedule based on data extracted from the received FedMessage, and a
set of rules.
The rules-engine is YAML, in a format like this::
```
- do:
- {tasks: [depcheck, upgradepath]}
when: {message_type: KojiTagChanged}
- do:
- {tasks: [dockerautotest]}
when: {message_type: KojiBuildCompleted, name: docker}
- do:
- {tasks: [abicheck]}
when:
message_type: KojiBuildCompleted
name:
$in: ${critpath_pkgs}
$nin: ['docker'] # critpath excludes
```
The rules are split in two parts `when` and `do`, the `when` clause is
a mongo query that will get evaluated against the dataset provided by
the FedMsg consumer. For example, the KojiBuildCompletedJobTrigger now
publishes this (values are fake, to make it more descriptive::
message_data = {
"_msg": {...snipped...},
"message_type": "KojiBuildCompleted",
"item": "docker-1.9.1-6.git6ec29ef.fc23",
"item_type": "koji_build",
"name": "docker",
"version": "1.9.1-6.git6ec29ef",
"release": "fc23",
"critpath_pkgs": [..., "docker", ...]
"distgit_branch": "f23",
}
So taking the rules, and the data, going from the top:
# First rule's `when` is `False` as `message_type` is not `KojiTagChanged`
# Second rule is `True` because both the `message_type` and name in the
`when` clause match the data
# Third rule does _not_ schedule anything, because even though `docker` is
in `critpath_pkgs`, it also is part of the critpath excludes list, and
so the rule is ignored
The `when` clauses are in fact mongo queries
<https://docs.mongodb.com/manual/tutorial/query-documents/>,
evaluated using a Python library that implements it for querying Python objects.
The rules engine then takes the `do` clauses of the 'passed' rules, and
produces arguments for the `trigger_tasks()` calls. By default, `item`, and
`item_type` are taken from the `message_data`, `arches` is set to
`config.valid_arches`, and then all the key/values from the `do`'s body are
added on top. This means, that we can have a task, that for example forces
an architecture different than default::
```
- do:
- {tasks: [awesome_arm_check], arches: [armhfp]}
when: {message_type: KojiBuildCompleted}
```
The `do` clause can have multiple items in it, so something like this is
possible::
```
- do:
- {tasks: [rpmlint]}
- {tasks: [awesome_arm_check], arches: [armhfp]}
when: {message_type: KojiBuildCompleted}
```
Triggering `rpmlint` on the default architectures, and `awesome_arm_check`
on `armhfp` for each package built in Koji.
This means, that when we want to trigger new (somewhat specific) tasks,
no changes are needed in the trigger's code, but just in the configuration,
to alter the rules. If we come to the point where more functionality is
needed, than it obviously calls for changes in the underlying code, in order
to add more key/values to the data provided by the Fedmsg consumer, or
adding more general functionality overall.
A good example of this is the dist-git style tasks problem. To solve it
I have added a new command (`$discover`)to the `do` section, that crawls the
provided git repo/branch, and schedules jobs for all `runtask.yml`'s found::
```
- do:
- {$discover: {repo:
'http://pkgs.fedoraproject.org/git/rpms-checks/${name}.git', branch:
'${distgit_branch}'}}
when: {message_type: KojiBuildCompleted}
```
In the bigger picture, this 'rules engine' functionality can be used to
make (for example) a web interface, that allows creating/altering the rules,
instead of changing the config file (the rules can as easily be taken from
a database, as from the config file), or even to provide a per-user triggering
capability - we could add a piece of code, that checks (selected) users'
Fedorapeople profile for a file, that contains rules in this format, and
then could simply run the engine on those rules+data from Fedmsg to decide
whether the user-defined tasks should be run.
It also somewhat reduces the tight bond between the trigger and FedMessage,
as the rules engine does not really care where did the data (used to evaluate
the rules) come from.
This is by no means final, but it IMO shows quite an interesting PoW/idea, that
was not that complicated to implement, and made the trigger lot better at what
it can do.
Please, share thoughts/comments/rants!
_______________________________________________
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org
