jesperct opened a new pull request, #41060: URL: https://github.com/apache/superset/pull/41060
### SUMMARY Temporal columns with a configured datetime format (`python_date_format`, for example `epoch_s`) were only normalized to datetimes for aggregated charts. In a raw/unaggregated query the column is selected as a plain physical column name, which is never a base-axis column, so `normalize_df` skipped it and returned the raw stored value. In a table chart the unconverted epoch integer then rendered as a wrong date (the frontend reads a bare number as epoch milliseconds), so `epoch_s` looked broken for unaggregated tables while it worked for aggregated ones. `normalize_df` now also collects the raw/unaggregated temporal columns from the query's selected columns, in addition to the base-axis and granularity columns it already handled. The raw columns are gated on a declared `python_date_format` rather than `is_dttm` alone, so a plain integer column that is flagged temporal but has no format is left untouched instead of being misread as nanosecond timestamps. The logic lives in the shared query pipeline, so other non-aggregating chart types benefit as well. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF Raw-mode table chart over a column stored as epoch seconds, with the dataset column's datetime format set to `epoch_s`: **Before:** the column renders as incorrect 1970 dates (e.g. `1970-01-19 06:17:16`), because the raw epoch-seconds integer reaches the chart unconverted. **After:** the same column renders as the correct dates (`2020-01-01`, `2021-01-01`, ...), matching what an aggregated table already shows. ### TESTING INSTRUCTIONS 1. Create a table whose temporal column stores epoch seconds (values like `1577836800`). 2. Create a dataset on it and mark the column as temporal with the datetime format `epoch_s`. 3. Build a Table chart, set Query mode to "Raw records", and add the column. 4. The column should render as proper dates (e.g. `2020-01-01`), matching the aggregated table. Unit tests: `pytest tests/unit_tests/models/helpers_test.py -k normalize_df` covers `epoch_s`, `epoch_ms`, and a guard that an unconfigured integer column is left untouched. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [ ] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
