imbajin opened a new pull request, #405: URL: https://github.com/apache/hugegraph-ai/pull/405
Mirrors [hugegraph/hugegraph-ai#87](https://github.com/hugegraph/hugegraph-ai/pull/87) from the existing `hugegraph:feat/eval-sdk` branch. The verification results below are reported by the source PR; ASF CI has not yet run for this pull request. ## Summary This PR adds `hugegraph-llm-eval`, an independently installable evaluation SDK for GraphRAG pipelines and external Python projects. It is the measurement layer of the repository: retrieval, answer and property-graph scoring rules live here, while `hugegraph-llm` keeps the pipeline, persistence, reports and CLI. Callers provide inputs, inject a judge model only for metrics that need one, and receive structured records they can persist elsewhere or feed back into a repair loop. Targets Python 3.12–3.14 and the dependency ranges introduced by #389. > [!NOTE] > The distribution is not published to an index by this change. `graphrag/eval/README.md` documents editable and wheel installs. ## Repository layout ```mermaid graph TD root["hugegraph-ai"] root --> app["hugegraph-llm<br/>pipeline · persistence · reports · CLI"] root --> sdk["hugegraph-llm-sdk/<br/>schema design · graph extraction"] root --> graphrag["graphrag/"] graphrag --> evaluation["eval/ — hugegraph-llm-eval<br/>retrieval · answers · property graphs"] graphrag -.-> ontology["ontology/ (future)"] ``` > [!NOTE] > `graphrag/` groups independently installable SDK packages. The extraction SDK stays at `hugegraph-llm-sdk/` in this PR; moving it under `graphrag/` is a separate layout decision that can follow once agreed. ## What it measures Three dimensions, each scored against a reference or against the source text. **Graph extraction** — candidate graph versus reference graph, plus candidate graph versus source text. | Metric | What it scores | Judge model | | --- | --- | :-: | | Entity extraction precision / recall / F1 | candidate vertices against reference vertices, matched semantically; label, name and property differences are reported as issues on an accepted match | ✓ (`graph` extra) | | Relation extraction precision / recall / F1 | candidate edges against reference edges; relation-type and endpoint-label differences are soft issues, endpoint or direction conflicts prevent a match | ✓ (`graph` extra) | | Schema validity | label conformance, type constraints, required-property fill and illegal-edge rate against the declared schema | — | | Graph faithfulness | every vertex and edge against the source text, with eight controlled error categories and separate support and error reasons | ✓ | **Retrieval** — candidate ranking and retrieved contexts versus reference documents and evidence. | Metric | What it scores | Judge model | | --- | --- | :-: | | `recall@k`, `hit@k` (any/all), `MRR` | candidate document ranking against reference document ids | — | | Context precision, context relevancy (0/1/2) | whether each retrieved context is relevant, and how completely it answers the question | ✓ | | Context entity recall, evidence recall | whether reference entities and evidence are covered by the retrieved contexts | ✓ | **Answers** — candidate answer versus reference answer and retrieved contexts. | Metric | What it scores | Judge model | | --- | --- | :-: | | ROUGE-L (English and Chinese) | lexical overlap with the reference answer | — | | Answer correctness | claim-level true/false positives and false negatives against the reference answer | ✓ | | Factual correctness | claim-level support against the question and reference | ✓ | | Faithfulness | whether each answer claim is supported by the retrieved contexts | ✓ | | Noise sensitivity | whether the answer stays correct when unsupporting contexts are present | ✓ | Three properties behind the scores: - **Claim-level scoring.** Correctness and faithfulness decompose both sides into claims, so a score explains which claims matched and which did not instead of returning one opaque number. - **Global one-to-one matching.** Each reference item matches at most one candidate and each candidate matches at most one reference item, so matching is a stable assignment rather than a greedy first hit. - **Evidence sensitivity.** Noise sensitivity re-checks the answer with and without supporting contexts, so an answer that happens to be right on the wrong evidence is distinguishable from a grounded one. ## Execution model ```mermaid flowchart LR builders["Input builders<br/>retrieval · answer · graph"] --> input["EvaluationInput<br/>content-addressed artifacts"] input --> evaluators["Evaluators"] evaluators --> judge{"Calls an LLM?"} judge -- yes --> session["JudgeSession<br/>limiter · retries · deadline · cancellation"] judge -- no --> metric["Deterministic metric"] session --> record["EvaluationRecord<br/>measurements · findings · judge calls · usage"] metric --> record record --> caller["Caller: persist, report or repair loop"] ``` | Option | Bounds | Default | Typical batch | | --- | --- | :-: | :-: | | `max_samples` | inputs running at once in `evaluate_many` | 1 | 4 | | `max_concurrency` | evaluators running inside one input | 1 | 4 | | `max_attempts` | provider attempts per judge call | 3 (maximum 20) | 3 | | `budget_usd` | one cost budget per call, shared by all samples | — | set per run | | `deadline`, `cancellation` | cooperative, checked before every judge call | — | set per run | > Defaults favour predictability over throughput: a library should not multiply provider traffic unless the caller asks for it. Batch runs normally set both concurrency values explicitly, and the run-scoped limiter keeps total provider concurrency at their product. > [!IMPORTANT] > One run-scoped limiter caps in-flight judge calls at `max_samples × max_concurrency`. Nested work — such as the candidate batches of a semantic matcher — waits on that same limiter instead of opening a second pool, so the cap holds however deep the evaluator batches go. - Results keep evaluator order; completion order never reorders records. - Record-level failures are data, not exceptions: an unmaterializable input, a provider failure or an invalid evaluator return all become failed records, and the rest of the batch is kept. - Cancellation and deadline mark the remaining records `cancelled` or `failed` and preserve completed records. - Evaluator and judge implementations run on worker threads and must not rely on shared mutable state. ## Quick start ```bash uv add --editable /path/to/hugegraph-ai/graphrag/eval --extra graph --extra rouge python examples/all_metrics.py ``` | Example | Scope | | --- | --- | | `examples/all_metrics.py` | every metric plus `evaluate` and `evaluate_many` in one process | | `examples/_stand_in_judge.py` | deterministic judge behind the example, so it runs without credentials | | `examples/basic.py` | ranking metrics only, base install, no judge | | `examples/real_model.py` | graph faithfulness against a real OpenAI-compatible endpoint | `examples/all_metrics.py` builds retrieval, answer and graph inputs with the input builders, prints measurements, findings, record-level failures and run status, and falls back to the stand-in judge unless a model is configured. Set `EVAL_JUDGE_MODEL` and `EVAL_JUDGE_API_KEY` (optionally `EVAL_JUDGE_BASE_URL`, `EVAL_JUDGE_PROVIDER`) to score with a real model. CI runs `basic.py` in the base wheel and `all_metrics.py` in the extras wheel. ```text retrieval: completed recall_at_k recall@1=0.0, recall@2=1.0 context_precision context_precision=1.0 answers: completed rouge_l rouge_l_precision=0.5556, rouge_l_recall=0.8333, rouge_l_f1=0.6667 graph review: completed schema_validity schema_validity=1.0, type_constraint_pass=1.0, ... graph extraction: completed semantic_triple semantic_triple_precision=1.0, semantic_triple_recall=1.0, semantic_triple_f1=1.0 batch: 2 samples, statuses=['completed', 'completed'] ``` ## Extension points ```mermaid flowchart LR caller["Your code"] --> builders["samples.*<br/>input builders"] builders --> input["EvaluationInput"] input --> engine["EvaluationEngine"] engine --> evaluator["Evaluator<br/>built-in or yours"] evaluator --> judge["Judge<br/>openai_judge or yours"] evaluator --> prompts["JudgeContent<br/>built-in or yours"] evaluator --> store["ArtifactStore<br/>in-memory or yours"] engine --> result["EvaluationResult<br/>records · findings"] ``` | Need | Interface | | --- | --- | | Different model or provider | implement `Judge.judge`, or call `openai_judge(...)`; vendor imports stay lazy | | New metric | implement `Evaluator`, or extend `JudgedEvaluatorLifecycle`, with a descriptor and input schema | | Different prompt or rubric | pass `JudgeContent`; built-in Chinese and English prompts ship in `prompt_templates/` | | Different input shape | add a helper in `samples.py`; evaluators keep reading the artifact contract | | Different storage | implement `ArtifactStore`; `InMemoryArtifactStore` is the default | Evaluator identity is a digest of the implementation, prompt, rubric and judge descriptor, so editing a scoring rule or a prompt changes the recorded identity instead of silently reusing an old score. ## Verification | Check | Result | | --- | --- | | `ruff check .` and `ruff format --check .` | pass | | `ty check graphrag/eval/src/hugegraph_llm_eval` | pass | | `pytest graphrag/eval/tests` | 98 passed | | `pytest scripts/tests` and `python scripts/version.py check` | 44 passed (30 subtests), metadata in sync | | wheel with `[graph,rouge,openai]` in a clean venv | 98 passed | | sdist with base dependencies in a clean venv | 58 passed, 8 skipped | | `examples/all_metrics.py` from the source tree and from the wheel | every metric completed | > [!NOTE] > The workflow builds the wheel and sdist outside the workspace and runs the suite against the installed distribution, not against the source tree. `tests/distribution_smoke.py` fails if the checkout shadows the installed package, and it asserts the base install imports no application or vendor module. ## Boundaries | In this package | Stays in the application | | --- | --- | | Domain contracts and judge ports | Persistence and artifact storage | | Evaluators, prompt registry and rubrics | Report rendering and CLI | | Concurrency, retry, deadline and cancellation semantics | Experiment orchestration and UI | | Structured findings usable as repair input | Product-specific pipeline adapters | - The base install depends on Pydantic only; `graph`, `rouge` and `openai` are extras with lazy vendor imports. - `import hugegraph_llm_eval` loads no application or vendor module; an isolation test asserts it. - Delegating `hugegraph-llm`'s evaluation code to this SDK is out of scope here: `upstream/main` contains no evaluation package to delegate from. ## Suggested review order | Step | Path | What to look at | | :-: | --- | --- | | 1 | `graphrag/eval/README.md` | public surface, install and usage | | 2 | `contracts/` (12 files) | input, record and judge contracts | | 3 | `evaluation/` (11 files) | engine, limiter, retries, deadline and cancellation | | 4 | `integrations/` (34 files) | three metric families and the OpenAI adapter | | 5 | `tests/` (17 files) | behaviour, failure paths and concurrency contracts | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
