bikash-barnwal commented on issue #42840:
URL: https://github.com/apache/superset/issues/42840#issuecomment-5300749628
This reproduces offline, without an Athena instance, and the data is lost
inside `pyathena` before Superset sees it — so the `validation:required` label
can be satisfied from a plain Python shell.
Superset's Athena spec supplies no type converter, so pyathena falls back to
`DefaultTypeConverter` (`pyathena/result_set.py:594`), which maps
`map`/`array`/`row` to its native-format parsers
(`pyathena/converter.py:390-412`). Those parsers bail out to `None` on any
bracket or parenthesis in the rendered text:
```python
# pyathena/converter.py:181
if any(char in inner for char in "()[]"):
# Contains complex structures (arrays, structs), skip parsing
return None
```
`_to_array` has the same bail at `converter.py:128-130`. Run against
pyathena 3.35.4 — the version Superset's own pin resolves to
(`pyproject.toml:134`, `pyathena[pandas]>=3.35.2,<4`):
```
'{key={"key2":[string]}}' -> None
'{a=[1, 2]}' -> None
'{a=(1,2)}' -> None
'{note=call me (asap)}' -> None <- plain string, just has parentheses
'{key=hello world}' -> {'key': 'hello world'} (control, parses fine)
```
Note the fourth case: the guard is meant to skip genuinely nested
structures, but it fires on ordinary prose that happens to contain a
parenthesis. `cursor.fetchall()` hands Superset a literal `None`; there is no
value left to serialize.
**Superset's own handling is correct.** The suggestion earlier in the thread
that `stringify_values()` fails on bracket-containing values does not hold —
#38172 and #41099 are already in `master` (`superset/result_set.py:62-97`).
Feeding it the values in question:
```
{'key': {'key2': ['string']}} -> '{"key": {"key2": ["string"]}}'
{'note': 'call me (asap)'} -> '{"note": "call me (asap)"}'
[[1, 2], [3]] -> '[[1, 2], [3]]'
```
All correct.
**Workaround for the reporter:** `CAST(col AS JSON)` routes through
`_to_json` (plain `json.loads`) and bypasses the native-format parser entirely.
I did not send a Superset-side patch. One is possible — injecting a custom
converter through `AthenaEngineSpec.adjust_engine_params` that falls back to
the raw varchar instead of `None` — but it would mean importing pyathena into
the engine spec to subclass its converter, papering over a third-party bug that
affects every pyathena consumer, and @rusackas said above he is taking this
upstream. Worth raising Superset's floor pin once the pyathena fix ships.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
