luth2 commented on code in PR #45823:
URL: https://github.com/apache/superset/pull/45823#discussion_r4453041856
##########
superset/commands/dataset/importers/v1/utils.py:
##########
@@ -986,32 +986,31 @@ def _convert_temporal_columns(df: pd.DataFrame, dtype:
dict[str, Any]) -> None:
for column_name, sqla_type in dtype.items():
if isinstance(sqla_type, (Date, DateTime)):
try:
- df[column_name] = pd.to_datetime(df[column_name])
- except OutOfBoundsDatetime:
- # Row-level fallback: coerce only OOB values; re-raise for
malformed
- # strings. Whole-column errors="coerce" would silently swallow
- # malformed values that happen to share a column with an OOB
date.
- original = df[column_name].copy()
- result = []
- for val in original:
- if pd.isna(val):
- result.append(pd.NaT)
- continue
- try:
- result.append(pd.to_datetime(val))
- except OutOfBoundsDatetime:
- result.append(pd.NaT)
- # Other exceptions (e.g. malformed strings) propagate
- converted = pd.Series(result, index=original.index)
- n_coerced = int(converted.isna().sum() - original.isna().sum())
- if n_coerced > 0:
- logger.warning(
- "Coerced %d out-of-bounds datetime value(s) "
- "in column '%s' to NaT",
- n_coerced,
- column_name,
- )
- df[column_name] = converted
+ converted = pd.to_datetime(df[column_name])
+ if pd.api.types.is_datetime64_any_dtype(converted):
+ df[column_name] = converted
+ continue
+ except ValueError:
Review Comment:
Fixed in 39312c304d and bac69d3c76. The vectorized parser only falls back
for OutOfBoundsDatetime or the specific Mixed timezones error; all other
ValueErrors are re-raised. A regression using ["01/02/2020", "05/03/2020",
"13/02/2020"] requires the format mismatch to raise and verifies the frame
remains unchanged. Mixed-offset timestamps, far-future dates and genuine
numeric overflow remain covered. After merging current master, 634 focused
tests passed on Python 3.15/pandas 3.1.0; 118 importer/pivot/date tests passed
on Python 3.11/pandas 3.1.0. Relevant hooks and serial Pylint pass.
##########
superset/utils/pandas_postprocessing/pivot.py:
##########
@@ -403,10 +403,9 @@ def pivot( # pylint: disable=too-many-arguments # noqa:
C901
if percent_mode is not None:
df = _apply_show_values_as(df, percent_mode)
- if combine_value_with_metric:
- # stack() no longer introduces rows of NA values to drop (pandas
>=2.1's
- # "future" stack implementation, the only one since pandas 3.0), so the
- # restored all-NaN metric rows are preserved without a dropna=
argument.
- df = df.stack(level=0).unstack()
+ if combine_value_with_metric and isinstance(df.columns, pd.MultiIndex):
+ df = df.reorder_levels([*range(1, df.columns.nlevels), 0],
axis=1).sort_index(
Review Comment:
Fixed in 39312c304d and bac69d3c76. Removed the lexicographic sort of all
column levels. Existing columns are now grouped stably by the first-seen
non-metric tuple from pivot_table before reordering levels, preserving category
order and keeping All last without recreating absent metric/category pairs. The
new test_pivot_combine_keeps_totals_last asserts the exact unsorted column
sequence (AU,m1), (AU,m2), (US,m1), (US,m2), (All,m1), (All,m2). The complete
pivot module and temporal conversion tests pass (60 cases); post-merge
validation also passed 634 focused tests on Python 3.15 and 118
importer/pivot/date tests on Python 3.11, both with pandas 3.1.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
