[ 
https://issues.apache.org/jira/browse/FLINK-41484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144359#comment-18144359
 ] 

Sangkyoon Nam commented on FLINK-41484:
---------------------------------------

I reproduced the {{python.execution-mode=thread}} part on master @ b703f069c56 
(Nov 3), which already includes FLINK-41196, with {{table.local-time-zone}} set 
to Asia/Shanghai. On the same build, process mode handles epoch 0 correctly in 
both directions (input arrives as {{08:00+08:00}}, and returning 
{{08:00+08:00}} casts back to 08:00:00), as a scalar and inside 
{{ARRAY<TIMESTAMP_LTZ>}}.

I'd like to take the thread-mode part. Could you assign this ticket to me, or 
create a thread-mode subtask if you'd rather keep the pandas part separate?

> Fix TIMESTAMP_LTZ handling in nested Pandas conversions and thread-mode 
> Python UDFs
> -----------------------------------------------------------------------------------
>
>                 Key: FLINK-41484
>                 URL: https://issues.apache.org/jira/browse/FLINK-41484
>             Project: Flink
>          Issue Type: Bug
>          Components: API / Python
>            Reporter: Dian Fu
>            Priority: Major
>
> TIMESTAMP_LTZ conversion remains incomplete or incorrect in two areas:
> 1. Nested values in PyFlink pandas/Arrow conversion paths, including 
> Table.to_pandas() and pandas UDFs.
> 2. Both top-level and nested values in regular Python UDFs running with 
> python.execution-mode=thread.
> These paths should preserve the represented instant and expose supported 
> TIMESTAMP_LTZ inputs as timezone-aware Python values in the session time zone.
> 1. Nested TIMESTAMP_LTZ in pandas/Arrow paths
> The shared pandas conversion logic handles top-level TIMESTAMP_LTZ columns 
> but does not consistently apply the same conversion recursively to nested 
> values.
> Depending on the API and schema, nested TIMESTAMP_LTZ is either explicitly 
> rejected or accepted without the required time-zone conversion.
> For example, with table.local-time-zone=Asia/Shanghai and both values 
> representing epoch 0, decoding an Arrow batch for a pandas UDF produces:
> Top-level TIMESTAMP_LTZ:
> Timestamp('1970-01-01 08:00:00+0800', tz='Asia/Shanghai')
> Value inside MAP<STRING, TIMESTAMP_LTZ>:
> [('k', datetime.datetime(1970, 1, 1, 0, 0))]
> The nested value has no tzinfo and retains UTC calendar fields. A UDF reading 
> the hour therefore observes 8 for the top-level value and 0 for the nested 
> value, although both represent the same instant and have the same logical 
> timestamp type.
> In the pandas UDF path:
> - ARRAY<TIMESTAMP_LTZ> and ROW fields directly containing TIMESTAMP_LTZ are 
> explicitly rejected.
> - MAP keys/values containing TIMESTAMP_LTZ can bypass these checks and reach 
> Python without time-zone conversion.
> - The same gap can occur when such a MAP is nested inside another ARRAY or 
> ROW.
> The updated PR #29297 adds rejection checks for certain nested LTZ schemas in 
> Table.to_pandas(). This prevents those schemas from being silently 
> misconverted on that path, but does not provide recursive LTZ support or fix 
> the accepted MAP schemas in pandas UDFs.
> The confirmed silent-conversion defect is on the pandas input/decode side. 
> Encoding must also be covered by regression tests; some timezone-aware nested 
> outputs are already normalized correctly by PyArrow.
> 2. TIMESTAMP_LTZ in regular Python UDF thread mode
> With python.execution-mode=thread, top-level and nested LTZ values follow 
> different conversion paths, both of which have problems.
> For top-level TIMESTAMP_LTZ, with session time zone Asia/Shanghai:
> - Input epoch 0 becomes datetime.datetime(1970, 1, 1, 0, 0), with tzinfo=None.
> - The expected input is 1970-01-01 08:00:00+08:00.
> - Returning datetime.datetime(1970, 1, 1, 8, 
> tzinfo=datetime.timezone(datetime.timedelta(hours=8))) produces epoch 
> milliseconds 28800000 instead of 0.
> For nested TIMESTAMP_LTZ:
> - ARRAY elements, ROW fields, and MAP keys/values reach Python as 
> pemja.PyJObject wrappers around java.time.Instant.
> - Returning a newly constructed Python datetime inside ARRAY<TIMESTAMP_LTZ> 
> fails with ArrayStoreException: java.sql.Timestamp.
> - Returning a Python datetime inside ROW or a MAP value fails with 
> ClassCastException: java.sql.Timestamp cannot be cast to java.time.Instant.
> Identity round trips can still pass. A top-level identity UDF can preserve 
> epoch 0, and a nested identity UDF can return the original Java Instant 
> wrappers unchanged. Input representation and output instant preservation must 
> therefore be tested independently.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to