1fanwang opened a new pull request, #29297: URL: https://github.com/apache/flink/pull/29297
## What is the purpose of the change PyFlink can execute a query returning `ARRAY<ROW>`, but collecting the same query with `Table.to_pandas()` raises `ValueError: ROW is not supported to be used as the element type of ArrayType.` This fixes that collection path for nested rows. Nested `TIMESTAMP_LTZ` remains rejected because pandas conversion does not localize timestamps inside arrays or rows. https://issues.apache.org/jira/browse/FLINK-34923 ## Brief change log - Permit nested rows when building the Arrow schema for `to_pandas()` without changing the default schema conversion or the Arrow-native transport. - Cover the reported SQL, a row inside a row, and nested zoned timestamps. ## Verifying this change The reported SQL failed before the fix and now returns its array of rows through the real Table API collection path. The focused regression and existing Arrow schema tests pass on Python 3.12 with the Flink 2.2.0 runtime wheel and Java 21. The repository's MiniCluster-based pandas tests need the Maven test-utils JAR; that suite was not run here. <details><summary>Reproducer and raw logs</summary> ```sql SELECT `value` FROM (VALUES (CAST(ARRAY[ROW(1, 2), ROW(2, 2)] AS ARRAY<ROW<`a` INT, `b` INT>>))) AS `t`(`value`) ``` Before, `table_env.sql_query(query).to_pandas()`: ``` ValueError: ROW is not supported to be used as the element type of ArrayType. FAILED flink-python/pyflink/table/tests/test_pandas_conversion.py::NestedPandasConversionTests::test_to_pandas_with_nested_rows ``` After, using the same query in the regression test: ``` $ python -m pytest -q flink-python/pyflink/table/tests/test_pandas_conversion.py flink-python/pyflink/fn_execution/tests/test_coders.py -k 'NestedPandasConversionTests or nested_types_require_arrow_mode or pandas_collection_schema_and_round_trip' --disable-warnings --maxfail=1 .... [100%] 4 passed, 52 deselected, 8 warnings in 6.23s ``` </details> ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable; the `to_pandas()` docstring now states the nested-type boundary. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: GitHub Copilot CLI 1.0.87-0 (Auto) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
