rusackas commented on code in PR #43693:
URL: https://github.com/apache/superset/pull/43693#discussion_r4161987387
##########
superset/utils/pandas_postprocessing/pivot.py:
##########
@@ -190,6 +190,44 @@ def _restore_dropped_metric_columns(
return df
+def _fill_dimension_column(df: DataFrame, col: str, fill_value: str) -> None:
+ """Fill missing values in a groupby dimension column before pivoting.
+
+ Handles categorical dtypes (adding fill_value to categories), datetime
+ dtypes (converting to string representation with fill_value for NaT), and
+ nullable extension dtypes (casting to object so a string fill value is
+ valid) to prevent dtype errors and preserve NULL/NaN/NaT keys through
+ pivot_table().
+ """
+ s = df[col]
+ if isinstance(s.dtype, pd.CategoricalDtype):
+ if s.isna().any():
+ # Only add the category when there is a NULL to fill: adding it
+ # unconditionally leaves pivot_table()'s default observed=False
+ # to materialize an unobserved "<NULL>" group/column even for
+ # dimensions that never had a missing value.
+ if fill_value not in s.cat.categories:
+ s = s.cat.add_categories([fill_value])
+ df[col] = s.fillna(value=fill_value)
+ elif pd.api.types.is_datetime64_any_dtype(s.dtype) or (
+ getattr(s.dtype, "kind", None) == "M"
+ ):
+ if s.isna().any():
+ df[col] = s.astype(str).where(~s.isna(), other=fill_value)
Review Comment:
Confirmed and fixed in 2492275 — switched to
`s.astype(object).where(~s.isna(), other=fill_value)`, which boxes values
without stringifying, so non-null entries stay real `Timestamp` objects and
only the NaT positions become the `NULL_STRING` sentinel. Verified the type
difference directly:
```python
>>> s.astype(str).where(~s.isna(), other='<NULL>')
['2019-01-01', '<NULL>', '2020-06-15'] # all str
>>> s.astype(object).where(~s.isna(), other='<NULL>')
[Timestamp('2019-01-01 00:00:00'), '<NULL>', Timestamp('2020-06-15
00:00:00')] # Timestamps preserved
```
Added a regression test asserting non-null index values stay `Timestamp`
instances.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
