SEPURI-SAI-KRISHNA commented on code in PR #45410:
URL: https://github.com/apache/superset/pull/45410#discussion_r4333965527
##########
superset/charts/client_processing.py:
##########
@@ -1304,6 +1390,34 @@ def _read_excel_for_client_processing(
return df.set_index(first_column)
+def payload_coltypes(
+ df: pd.DataFrame,
+ datasource: Optional[Union["BaseDatasource", "Query"]],
+) -> list[GenericDataType]:
+ """The payload's column types, reporting a text-rendered column as text.
+
+ ``extract_dataframe_dtypes`` lets the dataset's ``is_dttm`` flag decide the
+ type, so a temporal column a post-processor has rendered as text is still
+ reported as ``TEMPORAL``. ``chart_data_to_dataframe`` casts every
+ ``TEMPORAL`` column back to ``datetime64[ms]`` to build the notification
+ table, which would re-parse those strings and lose the configured format,
+ and would read ``02/01/2024`` month-first while reading ``15/01/2024``
+ day-first.
+
+ Only a column whose values really are strings is reported differently. The
+ XLSX path hands these types to ``apply_column_types`` to get date cells out
+ of a frame whose dates are still dates, and that frame is untouched here.
+ """
+ coltypes = extract_dataframe_dtypes(df, datasource)
+ return [
+ GenericDataType.STRING
+ if coltype == GenericDataType.TEMPORAL
+ and pd.api.types.infer_dtype(df.iloc[:, index], skipna=True) ==
"string"
Review Comment:
Confirmed, and the wrong date reproduces. On pandas 2.3.3 it is the `NaT`
specifically:
infer_dtype(["02/01/2024", pd.NaT], skipna=True) -> "mixed"
infer_dtype(["02/01/2024", None], skipna=True) -> "string"
`skipna` drops `None` and `NaN` but counts a `NaT` as a value of its own
type,
so the column stays `TEMPORAL` and the cast reads the string month-first:
pd.Series(["02/01/2024", pd.NaT]).astype("datetime64[ms]")
-> [Timestamp("2024-02-01 00:00:00"), NaT]
With `%d/%m/%Y` on 2 January that is 1 February in the report, and any null
in
the column was enough to undo the format.
Fixed by reading only the non-null values:
pd.api.types.infer_dtype(df.iloc[:, index].dropna(), skipna=False)
I kept it keyed on the values rather than tracking which columns `table()`
formatted, because the type has to follow the values for the CSV branch too,
where `read_csv` hands back text this module never wrote. A column of nothing
but nulls reads as empty rather than text and keeps the type the dataset
gives
it, which is what the XLSX branch needs to get date cells.
New test `test_table_report_time_format_survives_a_null_in_the_column`. It
fails on the previous head with `TEMPORAL`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
