linrrzqqq opened a new pull request, #70345:
URL: https://github.com/apache/doris/pull/70345
### Background
Python UDFs, UDAFs, and UDTFs exchange column batches with BE through Arrow
Flight. Doris builds Arrow schemas from the declared SQL argument and return
types, converts inputs for Python, and converts returned arrays back into Doris
columns.
DATETIME stores calendar fields without a timezone. TIMESTAMPTZ represents
an instant displayed using the session timezone.
### What problem does this PR solve?
**Issue Number:** None
**Related PR:** None
**PR Type:** Bugfix
**Problem Summary:**
Python temporal conversions can shift DATETIME results, expose TIMESTAMPTZ
inputs in the wrong timezone, reject valid TIME results, or turn the valid
TIMESTAMP_NS minimum into an error or NULL.
This PR makes the Arrow schemas and value conversions follow each SQL type’s
semantics, including normalizing UTC/GMT-prefixed offsets into names supported
by PyArrow. Most changed lines are tests and expected results.
### Details
- **Change map:** Counts include additions and deletions: **261 production
lines**, **849 test-code lines**, **159 generated regression-result lines**,
and **no documentation changes**.
- **208 production lines:** DATETIME/TIMESTAMPTZ schemas, session timezone
propagation, timezone-name normalization, and the internal compatibility field.
- **41 production lines:** Reading TIME results from Arrow float64
microseconds.
- **12 production lines:** Preserving the TIMESTAMP_NS minimum in
vectorized Python inputs.
- **Frequency:** Deterministic for naive DATETIME returns, TIMESTAMPTZ
sessions using other timezones or UTC/GMT-prefixed offsets, TIME roundtrips,
and vectorized TIMESTAMP_NS inputs containing the minimum value.
- **Root cause:**
| Type | Cause |
| --- | --- |
| DATETIME | Inherited a timezone-bearing Arrow timestamp schema. |
| TIMESTAMPTZ | Python conversions used a fixed `+08:00`; some valid Doris
timezone names are unsupported by PyArrow. |
| TIME | The BE reader rejected the float64 representation used by the
Python protocol. |
| TIMESTAMP_NS | pandas reserves the valid minimum integer for `NaT`. |
- **Why existing tests missed it:** Existing coverage did not verify all
affected schemas and Python roundtrips together. TIMESTAMP_NS tests missed the
minimum value in list and Series inputs, while timezone tests missed
UTC/GMT-prefixed session names.
- **Fix:**
| Type | Change |
| --- | --- |
| DATETIME | Use timezone-free Arrow timestamps to preserve calendar
fields. |
| TIMESTAMPTZ | Use the session timezone and normalize prefixed offsets
with the existing Doris utility. |
| TIME | Accept float64 microseconds, validate values, and round to the
declared precision. |
| TIMESTAMP_NS | Reuse scalar normalization in vectorized inputs to
preserve the minimum separately from NULL. |
- **Other paths and branches:** Apply consistent schemas to UDF, UDAF, UDTF,
and nested types. Preserve IANA timezone names for daylight-saving rules. An
optional `TFunction` field preserves the legacy DATETIME mapping when absent;
new FE sends it automatically for Python functions. No session variable is
added.
- **Behavior change:** Python receives timezone-free DATETIME values and
session-local TIMESTAMPTZ values. TIME returns support negative values, values
exceeding 24 hours, and declared-precision rounding. The TIMESTAMP_NS minimum
remains valid and distinct from NULL.
**TIMESTAMP_NS Series:** A Series containing the minimum uses `object`
dtype, with an Arrow scalar representing that value. `.dt` is unavailable for
that Series. Ordinary timestamp Series retain their existing dtype.
### Testing
All validation ran **locally**.
| Validation | Result |
| --- | --- |
| Full ASAN FE/BE build | Passed |
| Schema and Arrow conversion unit tests | 42 passed |
| TIME conversion unit tests | 5 passed |
| `test_python_arrow_time_types` | Passed |
| `test_pythonudf_timestamp_ns` | Passed |
| Formatting, header hygiene, and static analysis | Passed |
Regression coverage includes:
- Scalar, list, and Series calls.
- Aggregates, windows, and table functions.
- Nested types, NULLs, boundaries, and precision rounding.
- `UTC+08:00`, `GMT+08:00`, non-hour offsets, zero offset, and a
daylight-saving timezone.
Before-and-after verification:
- Direct `AdaptivePythonUDF` checks reproduced the TIMESTAMP_NS list
exception and Series data loss before the fix, then passed afterward.
- The timezone-schema test failed before prefix normalization and passed
afterward.
### Release note
Fix Python temporal Arrow conversions so DATETIME preserves calendar fields,
TIMESTAMPTZ follows the session timezone, TIME accepts float64 microsecond
results, and vectorized TIMESTAMP_NS calls preserve the valid minimum
separately from NULL. Series inputs containing the TIMESTAMP_NS minimum use
`object` dtype and cannot directly use `.dt`.
### Check List (For Author)
- **Behavior changed:** Yes (see Details).
- **Does this need documentation:** No.
- **Pre-submission status:** Ready.
### AI Disclosure
- **Tool:** OpenAI Codex (GPT-6).
- **Generated:** Implementation, unit and regression tests.
- **Human review:** Author reviewed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
