justinmclean opened a new issue, #378: URL: https://github.com/apache/airflow-steward/issues/378
## Summary Add a `--list-tags` flag to `skill_evals.runner` that walks the case directories under the supplied `path`, collects every distinct tag from `case-meta.json` files, prints them with the case count per tag, and exits zero without running prompts. ## Background The runner already supports `--tag <name>` to filter cases that opted in via `case-meta.json` (see [`tools/skill-evals/README.md` § Case tags](https://github.com/apache/airflow-steward/blob/main/tools/skill-evals/README.md#case-tags)). Today the only way to learn which tags exist for a skill (and therefore what `--tag` values are valid) is to grep every `case-meta.json` under that skill. A discovery flag closes that gap and makes `--tag` self-documenting. ## Where to look - `tools/skill-evals/src/skill_evals/runner.py` — `load_case_tags` at line 205 already returns the tag set for one case. Reuse it. - Same file, the `argparse` block in `main()` around line 419. Add `--list-tags` next to `--tag`. - Same file, `find_cases` (the function that produces the case list the existing `--tag` filter iterates over). `--list-tags` should walk the same iterable. - `tools/skill-evals/tests/test_runner.py` — where the regression test goes. ## Acceptance criteria - [ ] `python3 -m skill_evals.runner --list-tags <path>` prints every distinct tag found under `<path>`, sorted alphabetically, with the count of cases that carry each tag (for example `llama 12`). - [ ] `--list-tags` exits 0 without running any prompts and without requiring `--cli`. - [ ] If `<path>` contains no tagged cases, the runner exits 0 with a single informational line (for example "no tags found"). - [ ] Existing `--tag` filtering behaviour is unchanged. - [ ] A test in `tests/test_runner.py` creates a temp tree with two cases carrying distinct tags, invokes the CLI with `--list-tags`, and asserts the counts. - [ ] `uv run --directory tools/skill-evals --group dev pytest` passes. ## Estimated effort ~1-2 hours for someone new to the codebase. ## Getting started - [How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution) --- _This issue was drafted with the help of an AI-assisted tool and reviewed by an apache/airflow-steward maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
