luth2 commented on code in PR #45823:
URL: https://github.com/apache/superset/pull/45823#discussion_r4431341874
##########
superset/commands/dataset/importers/v1/utils.py:
##########
@@ -986,32 +986,31 @@ def _convert_temporal_columns(df: pd.DataFrame, dtype:
dict[str, Any]) -> None:
for column_name, sqla_type in dtype.items():
if isinstance(sqla_type, (Date, DateTime)):
try:
- df[column_name] = pd.to_datetime(df[column_name])
- except OutOfBoundsDatetime:
- # Row-level fallback: coerce only OOB values; re-raise for
malformed
- # strings. Whole-column errors="coerce" would silently swallow
- # malformed values that happen to share a column with an OOB
date.
- original = df[column_name].copy()
- result = []
- for val in original:
- if pd.isna(val):
- result.append(pd.NaT)
- continue
- try:
- result.append(pd.to_datetime(val))
- except OutOfBoundsDatetime:
- result.append(pd.NaT)
- # Other exceptions (e.g. malformed strings) propagate
- converted = pd.Series(result, index=original.index)
- n_coerced = int(converted.isna().sum() - original.isna().sum())
- if n_coerced > 0:
- logger.warning(
- "Coerced %d out-of-bounds datetime value(s) "
- "in column '%s' to NaT",
- n_coerced,
- column_name,
- )
- df[column_name] = converted
+ converted = pd.to_datetime(df[column_name])
+ if pd.api.types.is_datetime64_any_dtype(converted):
+ df[column_name] = converted.dt.as_unit("ns")
Review Comment:
Confirmed and fixed in 6b78fdb989. Removed both forced nanosecond casts,
keeping the resolution returned by pandas in the vectorized and per-value
paths. The existing far-future regression now passes; the mixed-offset fallback
test also requires the year-3118 timestamp to survive. Genuine numeric overflow
still becomes NaT with a warning, and malformed dates still raise. All 9
temporal conversion cases pass on Python 3.11/pandas 3.0.6 and Python
3.15/pandas 3.1.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
