[
https://issues.apache.org/jira/browse/FLINK-41245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-41245:
----------------------------
Description:
Since the FLINK-41032 backport 79907771be7, the python leg on release-2.3 fails
mypy. It passes allow_nested to create_arrow_schema, which only exists from
FLINK-40472 (2.4.0) on:
https://github.com/apache/flink/actions/runs/40486299256/job/122134071075
https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=80592&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602
{code}
pyflink/table/table.py:952: error: Unexpected keyword argument "allow_nested"
for "create_arrow_schema" [call-arg]
Found 1 error in 1 file (checked 73 source files)
{code}
Before #29634 the FLINK-34625 compile break hid it.
h3. Timestamp precision in empty pandas results
After restoring the missing {{allow_nested}} support, the same backport also
exposes a timestamp precision issue on release-2.3. {{_from_java_data_type()}}
hardcodes {{TIMESTAMP}} precision to 3 and uses the default precision of 6 for
{{TIMESTAMP_LTZ}}. As a result, empty {{Table.to_pandas()}} results use
milliseconds instead of seconds for {{TIMESTAMP(0)}}, and microseconds instead
of milliseconds for {{TIMESTAMP_LTZ(3)}}. This causes
{{test_empty_to_pandas_with_timestamps}} and
{{test_to_pandas_with_fixed_offset_timezones}} to fail in both batch and
streaming modes.
The fix preserves {{logical_type.getPrecision()}} for both timestamp types
while retaining their nullability. These two changes match the implementation
already introduced on master by
[FLINK-40187|https://github.com/apache/flink/commit/0910fc925dbe28965f1e15aff8aba5faa651630a],
which is missing from release-2.3.
Regression coverage includes Python-Java-Python round trips for {{TIMESTAMP}}
and {{TIMESTAMP_LTZ}}, precisions 0-9, and both nullable and non-nullable types
(40 combinations). Local validation passed all 10 {{DataTypeConvertTests}} and
all 22 pandas conversion tests across batch and streaming modes, using a
2.3-SNAPSHOT runtime and 2.3.0 test helper JARs.
PR: https://github.com/apache/flink/pull/29641
was:
Since the FLINK-41032 backport 79907771be7, the python leg on release-2.3 fails
mypy. It passes allow_nested to create_arrow_schema, which only exists from
FLINK-40472 (2.4.0) on:
https://github.com/apache/flink/actions/runs/40486299256/job/122134071075
https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=80592&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602
{code}
pyflink/table/table.py:952: error: Unexpected keyword argument "allow_nested"
for "create_arrow_schema" [call-arg]
Found 1 error in 1 file (checked 73 source files)
{code}
Before #29634 the FLINK-34625 compile break hid it.
> PyFlink mypy check fails on release-2.3: create_arrow_schema has no
> allow_nested parameter
> ------------------------------------------------------------------------------------------
>
> Key: FLINK-41245
> URL: https://issues.apache.org/jira/browse/FLINK-41245
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Affects Versions: 2.3.1
> Reporter: Martijn Visser
> Assignee: Dian Fu
> Priority: Blocker
> Labels: pull-request-available, test-stability
>
> Since the FLINK-41032 backport 79907771be7, the python leg on release-2.3
> fails mypy. It passes allow_nested to create_arrow_schema, which only exists
> from FLINK-40472 (2.4.0) on:
> https://github.com/apache/flink/actions/runs/40486299256/job/122134071075
> https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=80592&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602
> {code}
> pyflink/table/table.py:952: error: Unexpected keyword argument "allow_nested"
> for "create_arrow_schema" [call-arg]
> Found 1 error in 1 file (checked 73 source files)
> {code}
> Before #29634 the FLINK-34625 compile break hid it.
> h3. Timestamp precision in empty pandas results
> After restoring the missing {{allow_nested}} support, the same backport also
> exposes a timestamp precision issue on release-2.3.
> {{_from_java_data_type()}} hardcodes {{TIMESTAMP}} precision to 3 and uses
> the default precision of 6 for {{TIMESTAMP_LTZ}}. As a result, empty
> {{Table.to_pandas()}} results use milliseconds instead of seconds for
> {{TIMESTAMP(0)}}, and microseconds instead of milliseconds for
> {{TIMESTAMP_LTZ(3)}}. This causes {{test_empty_to_pandas_with_timestamps}}
> and {{test_to_pandas_with_fixed_offset_timezones}} to fail in both batch and
> streaming modes.
> The fix preserves {{logical_type.getPrecision()}} for both timestamp types
> while retaining their nullability. These two changes match the implementation
> already introduced on master by
> [FLINK-40187|https://github.com/apache/flink/commit/0910fc925dbe28965f1e15aff8aba5faa651630a],
> which is missing from release-2.3.
> Regression coverage includes Python-Java-Python round trips for {{TIMESTAMP}}
> and {{TIMESTAMP_LTZ}}, precisions 0-9, and both nullable and non-nullable
> types (40 combinations). Local validation passed all 10
> {{DataTypeConvertTests}} and all 22 pandas conversion tests across batch and
> streaming modes, using a 2.3-SNAPSHOT runtime and 2.3.0 test helper JARs.
> PR: https://github.com/apache/flink/pull/29641
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
