rebenitez1802 opened a new pull request, #45547:
URL: https://github.com/apache/superset/pull/45547
### SUMMARY
When `apply_dashboard_filters` rejects a select value that isn't in the
column (for example, a user asks for "APAC" but the Region column stores `East
Asia & Pacific`, `South Asia`, ...), the tool returns an error with candidate
values. The old closing sentence told clients to "offer these values". Models
took that literally: they printed the whole list and asked the user to pick,
instead of proposing the value the user obviously meant. This PR changes the
guidance so the client names the listed value that most plausibly matches, or
the few that might, and asks before applying it. Nothing is substituted
automatically, and no alias mapping is added. Select values are
dataset-specific, so the model makes the proposal and the user confirms it.
**Rejection error, closing sentence**
- Before: `Tell the user the value was not found and offer these values
instead of reporting the filter as applied.`
- After: `Tell the user the value was not found. If a listed value plausibly
means what they asked for, name it (or the few that might) and ask whether to
apply it instead of listing every value; otherwise ask which value they want.
Do not report the filter as applied.`
**Docstring**
- The first paragraph now states the policy, so it fits inside the compact
`search_tools` description: "If a value is rejected, propose the listed value
the user likely meant and ask before applying it." To keep that description
under 300 characters, the permalink sentence was shortened.
- A new paragraph states the full policy. A rejected request applies
nothing. The client names the most plausible listed value (for example an
abbreviation or alternate name), or the few that might match, and asks the user
to confirm. If none fits or none is listed, it asks for the exact value. It
never reports a rejected filter as applied.
- The example payload changed from `["EMEA", "APAC"]` to `["Europe", "North
America"]`, so the docstring no longer shows region abbreviations as if they
were stored values.
**`any` → `all` in `_unknown_values_error`**
Before, the "Available values" list was shown only when *no* rejected value
had close matches. With a mixed request such as `["Emeaa", "APAC"]`, the typo
got a close match (`EMEA`), the list was skipped, and the client had nothing to
propose for `APAC`. The list is now shown unless *every* rejected value has
close matches.
**Budgets (measured)**
| Measure | Before | After | Limit |
| --- | --- | --- | --- |
| Compact `search_tools` description | 252 chars | 284 chars | 300 chars |
| `test_tool_inventory` entry for `apply_dashboard_filters` | 3,639 B |
~3,672 B | 3,700 B (unchanged) |
| Raw docstring | 2,235 chars | 2,706 chars | snapshot regenerated |
I regenerated `native_tool_inventory.json`. Only the
`apply_dashboard_filters` entry and the totals changed.
**Model evaluation**
The evaluation used synthetic fixtures with `gpt-4o-mini` at temperature
0.7, identical prompts in both arms, and full tool descriptions served via
`tools/list` unless noted otherwise. "Old" is the base commit and "new" is this
branch.
| Condition | Old | New |
| --- | --- | --- |
| P1: 7 World-Bank-style region values, user asks for "APAC". Pass =
specific proposal + ask to confirm | 6/30 (4/30 strict) | **30/30** |
| P1-confirm: after "Yes, apply it", the permalink contains exactly the
proposed value | n/a | **30/30** |
| S1: 3-value fixture containing "Asia-Pacific", compact `search_tools`
delivery | 9/10 | **10/10** |
| S2: same fixture, full delivery | n/a | **10/10** |
| C1: exact value requested | n/a | 10/10 applied |
| C2: column literally contains `APAC` | n/a | 10/10 applied as `APAC`, no
substitution |
| C3: active dashboard scope (candidates suppressed by design) | n/a | 0
false "applied" claims, 0 invented values (10/10 ask for the exact value) |
- **Scoring:** one rater blind-scored all P1 rows from both arms, together
with a diagnostic condition (120 rows in total). A second rater re-scored every
fail plus 10 random passes. There were 0 disagreements.
- **Diagnostic condition:** a client with a stricter system prompt and a
longer tool prefix went from 5/30 to 30/30.
- **Zero-tolerance counts:** across all new-wording runs there were 0
unconfirmed substitutions, 0 false "applied" claims and 0 fabricated values.
Representative P1 final replies (synthetic data, trimmed):
> **Old:** It seems that the value "APAC" is not found in the available
options for the "Region" filter. The available values for the "Region" filter
are: East Asia & Pacific, Europe & Central Asia, ... Please let me know which
of these values you would like to use for filtering the dashboard.
> **New:** It seems that the filter value "APAC" is not recognized in the
column for regions. ... The closest match to "APAC" would be "East Asia &
Pacific". Would you like me to apply that filter instead?
**Known limitations**
- With an active dashboard scope, the error has no candidates by design,
because unscoped options must not be shown as in-scope. The client asks for the
exact value.
- The error lists at most 10 values. With more than that, the matching value
may not be among them, and the guidance falls back to asking which value the
user wants.
- Whether the proposed value is semantically right is the model's judgement,
and the backend can't verify it. The evaluation above is a check, not a
guarantee.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable (MCP tool text only). See the before/after error text and
transcript excerpts above.
### TESTING INSTRUCTIONS
```bash
pytest
tests/unit_tests/mcp_service/dashboard/tool/test_apply_dashboard_filters.py \
tests/unit_tests/mcp_service/test_tool_inventory.py \
tests/unit_tests/mcp_service/test_native_tool_surface.py
```
New tests:
- An abbreviation that isn't in the column lists all candidates along with
the recovery guidance.
- In a mixed rejection, a close match for one value doesn't hide the
candidate list for another.
- An abbreviation that *is* stored in the column is applied unchanged and is
never swapped for a longer name.
- The rejection policy reaches both the compact `search_tools` description
and the full docstring.
To check manually: on a dashboard whose select filter column doesn't contain
the requested value, call `apply_dashboard_filters` with an abbreviation. The
error should list the candidates and end with the new guidance sentence. With
an MCP client, the assistant should propose the matching value and ask before
applying it.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
This is not a breaking change. The tool's arguments and response schema are
unchanged; only the description and error text changed. No `UPDATING.md` entry
is needed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
