SEPURI-SAI-KRISHNA opened a new issue, #44827:
URL: https://github.com/apache/superset/issues/44827
### Bug description
`cum` fills gaps with `0` before cumulating, for all four operators:
```python
# superset/utils/pandas_postprocessing/cum.py
columns = columns or {}
df_cum = df.loc[:, columns.keys()]
df_cum = df_cum.fillna(0) # applied to sum, prod, min and max
operation = "cum" + operator
```
`0` is the identity for addition only. For `prod` a single gap zeroes the
entire rest of the series, and for `min`/`max` it makes `0` the running extreme
even though `0` need not appear anywhere in the data. There is no error — the
numbers are simply wrong.
Measured on current master:
| operator | input | current output | expected |
|---|---|---|---|
| `sum` | `[1, None, 2, 3]` | `[1, 1, 3, 6]` | `[1, 1, 3, 6]` ✅ |
| `prod` | `[2, None, 3, 4]` | `[2, 0, 0, 0]` | `[2, 2, 6, 24]` |
| `min` | `[5, None, 3, 7]` | `[5, 0, 0, 0]` | `[5, 5, 3, 3]` |
| `max` | `[-5, None, -3, -9]` | `[-5, 0, 0, 0]` | `[-5, -5, -3, -3]` |
Reproduce:
```python
import pandas as pd
from superset.utils.pandas_postprocessing import cum
cum(df=pd.DataFrame({"y": [2.0, None, 3.0, 4.0]}), operator="prod",
columns={"y": "y"})["y"].tolist()
# [2.0, 0.0, 0.0, 0.0]
```
**How it is reached.** The Explore "Rolling window" control only offers
`cumsum`, so this is not hit through that control. `cum` is in
`pandas_postprocessing.OPERATIONS`, so any chart-data request may pass
`{"operation": "cum", "options": {"operator": "prod", ...}}`; the operator is
validated against `ALLOWLIST_CUMULATIVE_FUNCTIONS` (`cummax`, `cummin`,
`cumprod`, `cumsum`) and nothing else. Clients using the API, the embedded SDK,
or a viz plugin that emits `cum` therefore get silently corrupted series
whenever the data has gaps.
**Where it came from.** #26429 (merged 2024-01-09, closing #21093) added
`fillna(0)` to close gaps in **cumsum** charts, where `0` is the right
identity. It was applied to all four operators rather than only to `sum`. The
line has not changed since.
**Why it was not caught.**
`tests/unit_tests/pandas_postprocessing/test_cum.py` exercises `prod` and `min`
only against `timeseries_df`, which has no gaps (`y = [1, 2, 3, 4]`). The one
gap-bearing test, `test_cum_with_gap`, uses `sum`, where the behaviour is
correct.
**Suggested fix.** Cumulate first, then carry the last cumulative value
across the gap:
```python
df_cum = _append_columns(df, getattr(df_cum, operation)().ffill(), columns)
```
This is byte-identical for `cumsum` (the existing `test_cum_with_gap`
expectation `[1.0, 3.0, 3.0, 7.0]` passes unchanged) and corrects the other
three. One deliberate difference: a *leading* gap stays empty instead of
becoming `0`, so no point is plotted before the series has any data.
### How to reproduce the bug
1. Send a chart-data request whose `post_processing` contains `{"operation":
"cum", "options": {"operator": "prod", "columns": {"y": "y"}}}`.
2. Use a series with at least one missing value.
3. Every value after the first gap is `0`.
### Screenshots/recordings
_No response_
### Superset version
master / latest-dev
### Python version
3.11
### Node version
Not applicable
### Browser
Not applicable
### Additional context
_No response_
### Checklist
- [x] I have searched Superset docs and Slack and didn't find a solution to
my problem.
- [x] I have searched the GitHub issue tracker and didn't find a similar bug
report.
- [x] I have checked Superset's logs for errors and if I found a relevant
Python stacktrace, I included it here as text in the "additional context"
section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
