dianfu commented on code in PR #29297:
URL: https://github.com/apache/flink/pull/29297#discussion_r4545662222
##########
flink-python/pyflink/table/types.py:
##########
@@ -2348,25 +2351,33 @@ def to_arrow_type(data_type: DataType, *,
allow_nested=False):
else:
return pa.timestamp('ns')
elif isinstance(data_type, MapType):
- key_type = to_arrow_type(data_type.key_type, allow_nested=allow_nested)
- value_type = to_arrow_type(data_type.value_type,
allow_nested=allow_nested)
+ key_type = to_arrow_type(data_type.key_type, allow_nested=allow_nested,
+ allow_nested_rows=allow_nested_rows)
+ value_type = to_arrow_type(data_type.value_type,
allow_nested=allow_nested,
+ allow_nested_rows=allow_nested_rows)
# PyArrow 5 only accepts data types and always makes map values
nullable.
if hasattr(pa.MapType, 'item_field'):
value_type = pa.field("value", value_type,
nullable=data_type.value_type._nullable)
return pa.map_(key_type, value_type)
elif isinstance(data_type, ArrayType):
Review Comment:
With allow_nested_rows=True, newly accepted schemas such as ARRAY<ROW<m
MAP<STRING NOT NULL, TIMESTAMP_LTZ(3)>>> bypass the nested timestamp
restriction through the MapType branch. However, to_pandas() only applies
timezone conversion to top-level TIMESTAMP_LTZ columns.
With Asia/Shanghai and epoch 0, I get 08:00+08:00 at the top level but a
timezone-naive 00:00 inside the map.
##########
flink-python/pyflink/table/tests/test_pandas_conversion.py:
##########
@@ -17,16 +17,47 @@
################################################################################
import datetime
import decimal
+import unittest
from pandas.testing import assert_frame_equal
from pyflink.common import Row
+from pyflink.table import EnvironmentSettings, TableEnvironment
from pyflink.table.types import DataTypes
from pyflink.testing import source_sink_utils
from pyflink.testing.test_case_utils import PyFlinkBatchTableTestCase, \
PyFlinkStreamTableTestCase
+class NestedPandasConversionTests(unittest.TestCase):
Review Comment:
Could we put these cases in the existing PyFlink integration-test hierarchy
instead of inheriting directly from unittest.TestCase?
When these tests start the shared gateway before
PyFlinkTestCase.setUpClass(), FLINK_TESTING has not been set, so the JVM starts
without the test JARs. Later tests reuse that gateway and cannot load
MiniClusterResourceConfiguration.Builder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
