aminghadersohi opened a new pull request, #44284:
URL: https://github.com/apache/superset/pull/44284
## TL;DR
Preserve literal percent signs in the `where_in` Jinja filter. On Athena,
filtering for `50%` currently sends `50%%` and misses matching rows. This fixes
the Superset literal-rendering boundary without changing driver behavior or
rewriting raw SQL.
### SUMMARY
SQLAlchemy's `format`/`pyformat` compilers double percent signs for DBAPI
interpolation. `where_in` returns that compiled fragment as literal SQL, but
SQL Lab calls `cursor.execute(query)` without parameters. PyAthena consequently
preserves the doubled signs. The same behavior reproduces with PyAthena
**2.25.0 and 3.35.4**, using SQLAlchemy **2.0.52**.
Undo one compiler-escaping layer only within the macro's compiled literals,
conditional on `identifier_preparer._double_percents`. This matches the
existing `Database.compile_sqla_query` convention. It does not change dialect
state, inject empty parameter dictionaries, change dependencies, or unescape an
entire rendered query. Quote/backslash handling is retained.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
No UI layout change. SQL/result comparison:
| Input | Before | After |
| --- | --- | --- |
| `{{ ['50%'] \| where_in }}` | `('50%%')` | `('50%')` |
| `{{ ['%%'] \| where_in }}` | `('%%%%')` | `('%%')` |
| Athena equality/IN against a `50%` row | 0 matches | 1 match |
### TESTING INSTRUCTIONS
1. Run `pytest tests/unit_tests/jinja_context_test.py` — **202 passed**
locally, with PyAthena installed (no skips).
2. On an Athena connection with template processing enabled, execute:
```sql
WITH fixture(value) AS (VALUES ('50%'), ('not the value'))
SELECT value FROM fixture
WHERE value IN {{ ['50%'] | where_in }}
```
Expect exactly one row containing `50%`.
3. Repeat with `%%`, `%(value)s`, and a combined
percent/underscore/apostrophe/backslash value; compare with a raw literal and a
bound-parameter control.
Regression coverage includes MySQL format/pyformat, PostgreSQL
pyformat/named, SQLite qmark, MSSQL named, unchanged dialect compilation,
numeric/NULL/empty values, and chart recompilation through
`Database.compile_sqla_query` with an unrelated `%%` literal. A PyAthena
formatter test exercises parameterless and bound paths; it is optional when
PyAthena is not installed. Generic dialect regressions do not depend on
PyAthena.
Removing the fix made **33 new regression cases fail** before adding the
chart-compilation cases. Live AWS verification ran **12 queries per PyAthena
version** (four baseline/fixed payload pairs and four bound controls): baseline
doubled percents and matched zero rows; fixed preserved values and matched
exactly one. The live harness used exact-source AST extraction of the macro and
engine-spec execute method, real Jinja, and the real DBAPI; it was not a
browser/API acceptance test. All 48 result objects were deleted; no persistent
tables were created.
Changed-file pre-commit validation includes mypy, ruff and pylint.
### ADDITIONAL INFORMATION
- [ ] Has associated issue:
- [x] Required feature flags: `ENABLE_TEMPLATE_PROCESSING` (existing)
- [ ] Changes UI
- [ ] Includes DB Migration
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
Risk is limited to `where_in` output for percent-escaping dialects; rollback
is a revert. SQLAlchemy's separate positional-placeholder rewriting behavior is
not changed by this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
