[
https://issues.apache.org/jira/browse/FLINK-41554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibo Dong updated FLINK-41554:
------------------------------
Description:
h2. Description
Flink's filesystem Parquet sink writes nested ROW fields successfully, but the
Parquet reader fails when the physical field order differs from the declared
read schema and the reordered fields have different physical types.
Both files contain the same named nested fields and values. The file written
with ROW<x INT, y STRING> can be read successfully, whereas the file written
with ROW<y STRING, x INT> fails when read using ROW<x INT, y STRING>.
h2. How to Reproduce
Run the following SQL in a fresh batch TableEnvironment, using initially
nonexistent filesystem paths:
{code:sql}
CREATE VIEW input_rows AS
SELECT * FROM (VALUES
(1, 11, 'a'),
(2, 12, 'b')
) AS t(id, x, y);
CREATE TABLE write_ordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/ordered',
'format' = 'parquet'
);
CREATE TABLE write_reordered (
id INT,
r ROW<y STRING, x INT>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/reordered',
'format' = 'parquet'
);
INSERT INTO write_ordered
SELECT id, CAST(ROW(x, y) AS ROW<x INT, y STRING>)
FROM input_rows;
INSERT INTO write_reordered
SELECT id, CAST(ROW(y, x) AS ROW<y STRING, x INT>)
FROM input_rows;
CREATE TABLE read_ordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/ordered',
'format' = 'parquet'
);
CREATE TABLE read_reordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/reordered',
'format' = 'parquet'
);
SELECT id, x, y FROM input_rows ORDER BY id;
SELECT id, r.x, r.y FROM read_ordered ORDER BY id;
SELECT id, r.x, r.y FROM read_reordered ORDER BY id;
{code}
h2. Expected Result
All three queries should return the same values when accessing nested fields by
name:
{code}
+I[1, 11, a]
+I[2, 12, b]
{code}
h2. Actual Result
The input view and the ordered Parquet file return the expected rows.
The reordered Parquet file is written successfully, but reading it fails
without returning any rows:
{code}
IllegalArgumentException: Unexpected type: BINARY
{code}
The exception originates from
{{{}ParquetSplitReaderUtil.createWritableColumnVector{}}}. The outer query
reports {{{}Failed to fetch next result{}}}.
was:
h2. Description
Flink's filesystem Parquet sink writes nested ROW fields successfully, but the
Parquet reader fails when the physical field order differs from the declared
read schema and the reordered fields have different physical types.
Both files contain the same named nested fields and values. The file written
with ROW<x INT, y STRING> can be read successfully, whereas the file written
with ROW<y STRING, x INT> fails when read using ROW<x INT, y STRING>.
h2. How to Reproduce
Run the following SQL in a fresh batch TableEnvironment, using initially
nonexistent filesystem paths:
{code:sql}
CREATE VIEW input_rows AS
SELECT * FROM (VALUES
(1, 11, 'a'),
(2, 12, 'b')
) AS t(id, x, y);
CREATE TABLE write_ordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/ordered',
'format' = 'parquet'
);
CREATE TABLE write_reordered (
id INT,
r ROW<y STRING, x INT>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/reordered',
'format' = 'parquet'
);
INSERT INTO write_ordered
SELECT id, CAST(ROW(x, y) AS ROW<x INT, y STRING>)
FROM input_rows;
INSERT INTO write_reordered
SELECT id, CAST(ROW(y, x) AS ROW<y STRING, x INT>)
FROM input_rows;
CREATE TABLE read_ordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/ordered',
'format' = 'parquet'
);
CREATE TABLE read_reordered (
id INT,
r ROW<x INT, y STRING>
) WITH (
'connector' = 'filesystem',
'path' = 'file:///tmp/flink-parquet-reorder/reordered',
'format' = 'parquet'
);
SELECT id, x, y FROM input_rows ORDER BY id;
SELECT id, r.x, r.y FROM read_ordered ORDER BY id;
SELECT id, r.x, r.y FROM read_reordered ORDER BY id;
{code}
h2. Expected Result
All three queries should return the same values when accessing nested fields by
name:
{code:text}
+I[1, 11, a]
+I[2, 12, b]
{code}
h2. Actual Result
The input view and the ordered Parquet file return the expected rows.
The reordered Parquet file is written successfully, but reading it fails
without returning any rows:
{code:text}
IllegalArgumentException: Unexpected type: BINARY
{code}
The exception originates from
{{ParquetSplitReaderUtil.createWritableColumnVector}}. The outer query reports
{{Failed to fetch next result}}.
> Parquet reader fails on reordered nested ROW fields with different physical
> types
> ---------------------------------------------------------------------------------
>
> Key: FLINK-41554
> URL: https://issues.apache.org/jira/browse/FLINK-41554
> Project: Flink
> Issue Type: Bug
> Environment: Apache Flink 2.3.0 (stock), batch Table API, local
> filesystem connector with CSV format. No optimizer rules disabled.
> Reporter: Yibo Dong
> Priority: Major
>
> h2. Description
> Flink's filesystem Parquet sink writes nested ROW fields successfully, but
> the Parquet reader fails when the physical field order differs from the
> declared read schema and the reordered fields have different physical types.
> Both files contain the same named nested fields and values. The file written
> with ROW<x INT, y STRING> can be read successfully, whereas the file written
> with ROW<y STRING, x INT> fails when read using ROW<x INT, y STRING>.
> h2. How to Reproduce
> Run the following SQL in a fresh batch TableEnvironment, using initially
> nonexistent filesystem paths:
> {code:sql}
> CREATE VIEW input_rows AS
> SELECT * FROM (VALUES
> (1, 11, 'a'),
> (2, 12, 'b')
> ) AS t(id, x, y);
> CREATE TABLE write_ordered (
> id INT,
> r ROW<x INT, y STRING>
> ) WITH (
> 'connector' = 'filesystem',
> 'path' = 'file:///tmp/flink-parquet-reorder/ordered',
> 'format' = 'parquet'
> );
> CREATE TABLE write_reordered (
> id INT,
> r ROW<y STRING, x INT>
> ) WITH (
> 'connector' = 'filesystem',
> 'path' = 'file:///tmp/flink-parquet-reorder/reordered',
> 'format' = 'parquet'
> );
> INSERT INTO write_ordered
> SELECT id, CAST(ROW(x, y) AS ROW<x INT, y STRING>)
> FROM input_rows;
> INSERT INTO write_reordered
> SELECT id, CAST(ROW(y, x) AS ROW<y STRING, x INT>)
> FROM input_rows;
> CREATE TABLE read_ordered (
> id INT,
> r ROW<x INT, y STRING>
> ) WITH (
> 'connector' = 'filesystem',
> 'path' = 'file:///tmp/flink-parquet-reorder/ordered',
> 'format' = 'parquet'
> );
> CREATE TABLE read_reordered (
> id INT,
> r ROW<x INT, y STRING>
> ) WITH (
> 'connector' = 'filesystem',
> 'path' = 'file:///tmp/flink-parquet-reorder/reordered',
> 'format' = 'parquet'
> );
> SELECT id, x, y FROM input_rows ORDER BY id;
> SELECT id, r.x, r.y FROM read_ordered ORDER BY id;
> SELECT id, r.x, r.y FROM read_reordered ORDER BY id;
> {code}
> h2. Expected Result
> All three queries should return the same values when accessing nested fields
> by name:
> {code}
> +I[1, 11, a]
> +I[2, 12, b]
> {code}
> h2. Actual Result
> The input view and the ordered Parquet file return the expected rows.
> The reordered Parquet file is written successfully, but reading it fails
> without returning any rows:
> {code}
> IllegalArgumentException: Unexpected type: BINARY
> {code}
> The exception originates from
> {{{}ParquetSplitReaderUtil.createWritableColumnVector{}}}. The outer query
> reports {{{}Failed to fetch next result{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)