vikramkoka opened a new pull request, #68037:
URL: https://github.com/apache/airflow/pull/68037
A key addition here is an AI validation step.
The example DAG was producing hallucinated output including fabricated
completion percentages, invented blockers, and missed shipped work. Many
reasons including the fact that the evidence pipeline was too thin and the
prompts too permissive.
Key changes:
- Add AIP registry with Confluence page IDs, GitHub search aliases, and
codebase directory paths for multi-strategy evidence gathering
- Fetch GitHub file tree (Git Trees API) for codebase-level evidence
- Replace flat 3000-char spec truncation with section-aware parsing
- Replace completion_pct/blockers Pydantic model with per-deliverable
DeliverableStatus (name, status, evidence, confidence)
- Add grounding rules to analysis/synthesis/validation system prompts
- Add three-layer quality pipeline: AI validation (LLMOperator) identifies
ungrounded claims, deterministic apply_validation task does mechanical
find-and-replace, human reviews the corrected report
- Add arithmetic validation that cross-checks X/Y fractions against
structured analysis data (catches validator-introduced errors)
- Set temperature=0 on all LLM calls for run-to-run consistency
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
---
##### Was generative AI tooling used to co-author this PR?
- [ x] Yes (please specify the tool below)
<!--
Generated-by: [Claude] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
