adityamparikh opened a new pull request, #265: URL: https://github.com/apache/solr-mcp/pull/265
Part of #256. Stacked on #264: the first commit here is #264, and only the top commit (`test(eval): check the search arguments for a workshops prompt`) is new. Adds the first scenario: did the model build the search we expect? ## What it does `src/test/resources/eval/scenarios.json` has one case: | id | prompt | tool | args | |---|---|---|---| | `workshops-by-type` | List the workshops in the devnexus collection. | `search` | `collection=devnexus`, `filterQueries` contains `session_type:Workshop` | The search description says "Put filters in fq (filterQueries) instead of q", so a workshop filter in `q` fails. `ScenarioScorer` runs the three code checks from the spec: - `tool-selected`: the expected tool was called. - `no-forbidden-tool`: none of the scenario's confusable neighbours was called. - `args-match`: the listed arguments have the expected values, read from the **last** call to the expected tool. In the runs, the model sometimes ran a facet search first (`facetFields=[session_type]`, `rows=0`) to find the values, then the real one. Scoring the first call would fail that discovery step. A scalar matches by its text. A list matches when it contains each expected entry, ignoring case and double quotes, because `session_type:"workshop"` and `session_type:Workshop` find the same documents on the schemaless text field. `ToolSelectionEval` runs each scenario `-Peval.runs` times (default 5), one test per run, so the test report gives the pass rate. ## Result on `gpt-5-mini`, N=5 3 of 5 passed. The two failures: - `query=session_type:workshop`, `filterQueries=[]`: a real miss of the fq guidance. - `filterQueries=["session_type:(workshop Workshop)"]`: the right filter, written in a form text matching can't recognise. The spec gives equivalence like this to the Laya `args-express-intent` judgment, which comes later. ## Verification - `./gradlew build`: 479 tests, 0 skipped, 0 failed. That includes `ScenarioScorerTest` (9) and `ScenariosFileTest` (2). - `./gradlew eval`: `EvalHarnessEval` 3/3, `ToolSelectionEval` 3/5 as above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
