rusackas opened a new pull request, #38172:
URL: https://github.com/apache/superset/pull/38172
### SUMMARY
This PR fixes issue #25125 where JSON/JSONB data from databases (like
PostgreSQL) was being converted to strings instead of being preserved as Python
objects. This broke features like Handlebars templates that need to access JSON
data as objects (e.g., `{{this.json.key}}`).
**Root Cause:**
When processing query results, PyArrow detects JSON/JSONB columns as "nested
types" and the code was stringifying them for compatibility. However, this
meant the JSON objects became strings by the time they reached the frontend.
**Solution:**
The fix works by:
1. Tracking columns with nested/JSON data before stringification (for
PyArrow compatibility)
2. Restoring the original Python objects (dicts/lists) when converting to
pandas DataFrame
This approach:
- Maintains backward compatibility with the PyArrow table structure
- Preserves JSON objects as Python dicts/lists in the final DataFrame
- Works with heterogeneous JSON structures (different keys per row)
- Handles null values correctly
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable - this is a backend data serialization fix.
**Before:**
```json
{"source_id": "1", "json": "{'key': 'value'}"}
```
**After:**
```json
{"source_id": "1", "json": {"key": "value"}}
```
### TESTING INSTRUCTIONS
1. Create a PostgreSQL table with a JSONB column:
```sql
CREATE TABLE test_json (
id SERIAL PRIMARY KEY,
data JSONB
);
INSERT INTO test_json (data) VALUES
('{"key": "value1", "nested": {"a": 1}}'),
('{"key": "value2", "items": [1, 2, 3]}');
```
2. Create a Handlebars chart using this dataset
3. In the Handlebars template, try accessing nested JSON properties:
```handlebars
{{#each data}}
Key: {{this.data.key}}, Nested A: {{this.data.nested.a}}
{{/each}}
```
4. Verify that the JSON properties are accessible as objects, not strings
**Unit Tests:**
Run the new tests:
```bash
pytest
tests/unit_tests/result_set_test.py::test_json_data_type_preserved_as_objects -v
pytest
tests/unit_tests/result_set_test.py::test_json_data_with_homogeneous_structure
-v
pytest tests/unit_tests/result_set_test.py::test_array_data_type_preserved -v
```
### ADDITIONAL INFORMATION
- [x] Has associated issue: Fixes #25125
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
