codeant-ai-for-open-source[bot] commented on code in PR #40005:
URL: https://github.com/apache/superset/pull/40005#discussion_r3215798003
##########
superset/utils/pandas_postprocessing/pivot.py:
##########
@@ -27,6 +28,35 @@
)
+def _restore_dropped_metric_columns(
+ df: DataFrame, expected_metrics: list[str]
+) -> DataFrame:
+ """Re-add metric columns that pivot_table dropped due to all-NaN values.
+
+ When drop_missing_columns=True, pandas pivot_table silently removes columns
+ whose entries are all NaN. This breaks downstream post-processing steps
+ (rename, rolling) that use validate_column_args to assert the columns
exist.
+ Restoring the columns as all-NaN preserves the expected schema.
+ """
+ if isinstance(df.columns, pd.MultiIndex):
+ existing_metrics = set(df.columns.get_level_values(0))
+ missing = [m for m in expected_metrics if m not in existing_metrics]
+ if missing:
+ category_values = (
+ df.columns.get_level_values(-1).unique()
+ if len(df.columns) > 0
+ else [None]
+ )
+ for metric in missing:
+ for cat in category_values:
+ df[(metric, cat)] = float("nan")
Review Comment:
**🟠Architect Review — HIGH**
_restore_dropped_metric_columns assumes MultiIndex columns are 2-level
(metric, category), rebuilding missing metrics using only the last level
(get_level_values(-1)). For pivots with multiple `columns` dimensions (3+
MultiIndex levels), this cannot reconstruct the full (metric, col1, col2, ...)
keys and will either create malformed/incomplete column labels or fail to
re-add the expected combinations, so all-NaN metrics are not correctly restored
in those valid multi-column pivot configurations.
**Suggestion:** When restoring missing metrics for MultiIndex columns, build
new column keys by copying the full non-metric part of existing tuples (levels
1..n), e.g. for each existing column tuple `col` create `(metric, *col[1:])`,
matching how `series_set` is constructed. Add a unit test where `columns` has
2+ dimensions and one metric is all-NaN to verify the restored schema works
with downstream post-processing.
[Fix in
Cursor](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt=This%20is%20an%20%2A%2AArchitect%20%2F%20Logical%20Review%2A%2A%20comment%20left%20during%20a%20code%20review.%20These%20reviews%20are%20first-class%2C%20important%20findings%20%E2%80%94%20not%20optional%20suggestions.%20Do%20NOT%20dismiss%20this%20as%20a%20%27big%20architectural%20change%27%20just%20because%20the%20title%20says%20architect%20review%3B%20most%20of%20these%20can%20be%20resolved%20with%20a%20small%2C%20localized%20fix%20once%20the%20intent%20is%20understood.%0A%0A%2A%2APath%3A%2A%2A%20superset%2Futils%2Fpandas_postprocessing%2Fpivot.py%0A%2A%2ALine%3A%2A%2A%2041%3A52%0A%2A%2AComment%3A%2A%2A%0A%09%2AHIGH%3A%20_restore_dropped_metric_columns%20assumes%20MultiIndex%20columns%20are%202-level%20%28metric%2C%20category%29%2C%20rebuilding%20missing%20metrics%20using%20only%20the%20last%20level%20%28get_level_values%28-1%29%29.%20For%20pivots%20with%20multiple%20%60columns%60%20dimensions%20%283%2B%20Mult
iIndex%20levels%29%2C%20this%20cannot%20reconstruct%20the%20full%20%28metric%2C%20col1%2C%20col2%2C%20...%29%20keys%20and%20will%20either%20create%20malformed%2Fincomplete%20column%20labels%20or%20fail%20to%20re-add%20the%20expected%20combinations%2C%20so%20all-NaN%20metrics%20are%20not%20correctly%20restored%20in%20those%20valid%20multi-column%20pivot%20configurations.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20issue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0AIf%20a%20suggested%20approach%20is%20provided%20above%2C%20use%20it%20as%20the%20authoritative%20instruction.%20If%20no%20explicit%20code%20suggestion%20is%20given%2C%20you%20MUST%20still%20draft%20and%20apply%20your%20own%20minimal%2C%20localized%20fix%20%E2%80%94%20do%20not%20punt%20back%20with%20%27no%20suggestion%20provided%2C%20review%20manually%27.%20Keep%20the%20change%20as%20small%20as%20possible%3A%20ad
d%20a%20guard%20clause%2C%20gate%20on%20a%20loading%20state%2C%20reorder%20an%20await%2C%20wrap%20in%20a%20conditional%2C%20etc.%20Do%20not%20refactor%20surrounding%20code%20or%20expand%20scope%20beyond%20the%20finding.%0AOnce%20fix%20is%20implemented%2C%20also%20check%20other%20comments%20on%20the%20same%20PR%2C%20and%20ask%20user%20if%20the%20user%20wants%20to%20fix%20the%20rest%20of%20the%20comments%20as%20well.%20if%20said%20yes%2C%20then%20fetch%20all%20the%20comments%20validate%20the%20correctness%20and%20implement%20a%20minimal%20fix%0A)
| [Fix in VSCode
Claude](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt=This%20is%20an%20%2A%2AArchitect%20%2F%20Logical%20Review%2A%2A%20comment%20left%20during%20a%20code%20review.%20These%20reviews%20are%20first-class%2C%20important%20findings%20%E2%80%94%20not%20optional%20suggestions.%20Do%20NOT%20dismiss%20this%20as%20a%20%27big%20architectural%20change%27%20just%20because%20the%20title%20says%20architect%20review%3B%20most
%20of%20these%20can%20be%20resolved%20with%20a%20small%2C%20localized%20fix%20once%20the%20intent%20is%20understood.%0A%0A%2A%2APath%3A%2A%2A%20superset%2Futils%2Fpandas_postprocessing%2Fpivot.py%0A%2A%2ALine%3A%2A%2A%2041%3A52%0A%2A%2AComment%3A%2A%2A%0A%09%2AHIGH%3A%20_restore_dropped_metric_columns%20assumes%20MultiIndex%20columns%20are%202-level%20%28metric%2C%20category%29%2C%20rebuilding%20missing%20metrics%20using%20only%20the%20last%20level%20%28get_level_values%28-1%29%29.%20For%20pivots%20with%20multiple%20%60columns%60%20dimensions%20%283%2B%20MultiIndex%20levels%29%2C%20this%20cannot%20reconstruct%20the%20full%20%28metric%2C%20col1%2C%20col2%2C%20...%29%20keys%20and%20will%20either%20create%20malformed%2Fincomplete%20column%20labels%20or%20fail%20to%20re-add%20the%20expected%20combinations%2C%20so%20all-NaN%20metrics%20are%20not%20correctly%20restored%20in%20those%20valid%20multi-column%20pivot%20configurations.%0A%0AValidate%20the%20correctness%20of%20the%20flagged%20is
sue.%20If%20correct%2C%20How%20can%20I%20resolve%20this%3F%20If%20you%20propose%20a%20fix%2C%20implement%20it%20and%20please%20make%20it%20concise.%0AIf%20a%20suggested%20approach%20is%20provided%20above%2C%20use%20it%20as%20the%20authoritative%20instruction.%20If%20no%20explicit%20code%20suggestion%20is%20given%2C%20you%20MUST%20still%20draft%20and%20apply%20your%20own%20minimal%2C%20localized%20fix%20%E2%80%94%20do%20not%20punt%20back%20with%20%27no%20suggestion%20provided%2C%20review%20manually%27.%20Keep%20the%20change%20as%20small%20as%20possible%3A%20add%20a%20guard%20clause%2C%20gate%20on%20a%20loading%20state%2C%20reorder%20an%20await%2C%20wrap%20in%20a%20conditional%2C%20etc.%20Do%20not%20refactor%20surrounding%20code%20or%20expand%20scope%20beyond%20the%20finding.%0AOnce%20fix%20is%20implemented%2C%20also%20check%20other%20comments%20on%20the%20same%20PR%2C%20and%20ask%20user%20if%20the%20user%20wants%20to%20fix%20the%20rest%20of%20the%20comments%20as%20well.%20if%20said%2
0yes%2C%20then%20fetch%20all%20the%20comments%20validate%20the%20correctness%20and%20implement%20a%20minimal%20fix%0A)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is an **Architect / Logical Review** comment left during a code review.
These reviews are first-class, important findings — not optional suggestions.
Do NOT dismiss this as a 'big architectural change' just because the title says
architect review; most of these can be resolved with a small, localized fix
once the intent is understood.
**Path:** superset/utils/pandas_postprocessing/pivot.py
**Line:** 41:52
**Comment:**
*HIGH: _restore_dropped_metric_columns assumes MultiIndex columns are
2-level (metric, category), rebuilding missing metrics using only the last
level (get_level_values(-1)). For pivots with multiple `columns` dimensions (3+
MultiIndex levels), this cannot reconstruct the full (metric, col1, col2, ...)
keys and will either create malformed/incomplete column labels or fail to
re-add the expected combinations, so all-NaN metrics are not correctly restored
in those valid multi-column pivot configurations.
Validate the correctness of the flagged issue. If correct, How can I resolve
this? If you propose a fix, implement it and please make it concise.
If a suggested approach is provided above, use it as the authoritative
instruction. If no explicit code suggestion is given, you MUST still draft and
apply your own minimal, localized fix — do not punt back with 'no suggestion
provided, review manually'. Keep the change as small as possible: add a guard
clause, gate on a loading state, reorder an await, wrap in a conditional, etc.
Do not refactor surrounding code or expand scope beyond the finding.
Once fix is implemented, also check other comments on the same PR, and ask
user if the user wants to fix the rest of the comments as well. if said yes,
then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
