Yibo Dong created FLINK-41554:
---------------------------------

             Summary: Parquet reader fails on reordered nested ROW fields with 
different physical types
                 Key: FLINK-41554
                 URL: https://issues.apache.org/jira/browse/FLINK-41554
             Project: Flink
          Issue Type: Bug
         Environment: Apache Flink 2.3.0 (stock), batch Table API, local 
filesystem connector with CSV format. No optimizer rules disabled.
            Reporter: Yibo Dong


h2. Description

Flink's filesystem Parquet sink writes nested ROW fields successfully, but the 
Parquet reader fails when the physical field order differs from the declared 
read schema and the reordered fields have different physical types.

Both files contain the same named nested fields and values. The file written 
with ROW<x INT, y STRING> can be read successfully, whereas the file written 
with ROW<y STRING, x INT> fails when read using ROW<x INT, y STRING>.

h2. How to Reproduce

Run the following SQL in a fresh batch TableEnvironment, using initially 
nonexistent filesystem paths:

{code:sql}
CREATE VIEW input_rows AS
SELECT * FROM (VALUES
(1, 11, 'a'),
(2, 12, 'b')
) AS t(id, x, y);

CREATE TABLE write_ordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/ordered',
'format' = 'parquet'
);

CREATE TABLE write_reordered (
id INT,
r ROW<y STRING, x INT>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/reordered',
'format' = 'parquet'
);

INSERT INTO write_ordered
SELECT id, CAST(ROW(x, y) AS ROW<x INT, y STRING>)
FROM input_rows;

INSERT INTO write_reordered
SELECT id, CAST(ROW(y, x) AS ROW<y STRING, x INT>)
FROM input_rows;

CREATE TABLE read_ordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/ordered',
'format' = 'parquet'
);

CREATE TABLE read_reordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/reordered',
'format' = 'parquet'
);

SELECT id, x, y FROM input_rows ORDER BY id;
SELECT id, r.x, r.y FROM read_ordered ORDER BY id;
SELECT id, r.x, r.y FROM read_reordered ORDER BY id;
{code}

h2. Expected Result

All three queries should return the same values when accessing nested fields by 
name:

{code:text}
+I[1, 11, a]
+I[2, 12, b]
{code}

h2. Actual Result

The input view and the ordered Parquet file return the expected rows.

The reordered Parquet file is written successfully, but reading it fails 
without returning any rows:

{code:text}
IllegalArgumentException: Unexpected type: BINARY
{code}

The exception originates from 
{{ParquetSplitReaderUtil.createWritableColumnVector}}. The outer query reports 
{{Failed to fetch next result}}.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to