aminghadersohi opened a new pull request, #40004:
URL: https://github.com/apache/superset/pull/40004
### SUMMARY
When a Jinja conditional metric expression (e.g. `{% if filter_values('x')
%}SUM(col){% else %}null{% endif %}`) evaluates to `null` because no dashboard
filter is selected, the SQL query returns an all-NULL column for that metric.
pandas `pivot_table` with `dropna=True` (the default, controlled by
`drop_missing_columns: !show_empty_columns` in `pivotOperator.ts`) then
**silently drops** that column from the pivot result.
Downstream post-processing steps — `rename` and `rolling` — use
`validate_column_args` to assert that all referenced columns exist before
executing. Because the metric column was dropped, they raise:
```
InvalidPostProcessingError("Referenced columns not available in DataFrame.")
```
This surfaces as an error on mixed-timeseries charts whenever `rename`
(triggered by groupby + `truncate_metric`) or `rolling` is configured.
**Root cause chain:**
1. Jinja template with `filter_values()` evaluates to `null` when no filter
selected
2. SQL returns all-NULL column for the metric
3. `pivot_table(dropna=True)` drops the all-NaN column
4. `rename` / `rolling` post-processing fails because column is missing
**Fix:** Introduce `_restore_dropped_metric_columns()` in `pivot.py` which
re-adds any expected metric columns that `pivot_table` dropped due to all-NaN
values. This preserves the expected DataFrame schema for downstream
post-processing, while still dropping sparse category combinations (the
intended behavior of `drop_missing_columns`).
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before: Mixed-timeseries chart with Jinja SQL metric and no active dashboard
filter shows error "Referenced columns not available in DataFrame."
After: Chart renders correctly (empty series since metric evaluates to NULL,
but no error).
### TESTING INSTRUCTIONS
1. Create a mixed-timeseries chart with a SQL-type custom metric using a
Jinja conditional:
```sql
{% if filter_values('my_column') %}
SUM(CASE WHEN my_column IN {{ filter_values('my_column') | where_in }}
THEN value ELSE 0 END)
{% else %}
null
{% endif %}
```
2. View the chart without any dashboard filter selected
3. **Before fix:** chart shows "Referenced columns not available in
DataFrame" error
4. **After fix:** chart renders without error (no data series shown for the
NULL metric)
### ADDITIONAL INFORMATION
- [x] Has associated issue: SC-100398
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
Regression tests added in
`tests/unit_tests/pandas_postprocessing/test_pivot.py`:
- `test_pivot_preserves_all_nan_metric_flat`: flat pivot (no groupby) —
metric column restored as all-NaN
- `test_pivot_preserves_all_nan_metric_with_columns`: MultiIndex pivot (with
groupby) — metric level preserved at level 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
