[
https://issues.apache.org/jira/browse/FLINK-31811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712390#comment-17712390
]
Krzysztof Chmielewski commented on FLINK-31811:
-----------------------------------------------
I think this might be a duplicate of
https://issues.apache.org/jira/browse/FLINK-31202 that manifest in SQL API.
P.S.
Are you sure that this was working in 1.15.1?
I know that writing "simple" complex types like Array of Intigers or Strings
bBut not sure about this one -> Arrays Of Map.
> Unsupported complex data type for Flink SQL
> -------------------------------------------
>
> Key: FLINK-31811
> URL: https://issues.apache.org/jira/browse/FLINK-31811
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.16.1
> Reporter: jirawech.s
> Priority: Major
> Fix For: 1.16.2
>
>
> I found this issue when I tried to write data on local filesystem using Flink
> SQL
> {code:java}
> 19:51:32,966 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph
> [] - compact-operator (1/4)
> (4f2a09b638c786f74262c675d248afd9_80fe6c4f32f605d447b391cdb16cc1ff_0_4)
> switched from RUNNING to FAILED on 69ed2306-371b-4bfc-a98e-bf75fb41748f @
> localhost (dataPort=-1).
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:659) ~[?:1.8.0_301]
> at java.util.ArrayList.get(ArrayList.java:435) ~[?:1.8.0_301]
> at org.apache.parquet.schema.GroupType.getType(GroupType.java:216)
> ~[parquet-column-1.12.2.jar:1.12.2]
> at
> org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createWritableColumnVector(ParquetSplitReaderUtil.java:523)
> ~[flink-parquet-1.16.1.jar:1.16.1]
> at
> org.apache.flink.formats.parquet.vector.ParquetSplitReaderUtil.createWritableColumnVector(ParquetSplitReaderUtil.java:503)
> ~[flink-parquet-1.16.1.jar:1.16.1]
> at
> org.apache.flink.formats.parquet.ParquetVectorizedInputFormat.createWritableVectors(ParquetVectorizedInputFormat.java:281)
> ~[flink-parquet-1.16.1.jar:1.16.1]
> at
> org.apache.flink.formats.parquet.ParquetVectorizedInputFormat.createReaderBatch(ParquetVectorizedInputFormat.java:270)
> ~[flink-parquet-1.16.1.jar:1.16.1]
> at
> org.apache.flink.formats.parquet.ParquetVectorizedInputFormat.createPoolOfBatches(ParquetVectorizedInputFormat.java:260)
> ~[flink-parquet-1.16.1.jar:1.16.1]
> {code}
> What i tried to do is writing complex data type to parquet file
> Here is the schema of sink table. The problematic data type is
> ARRAY<MAP<STRING,STRING>>
> {code:java}
> CREATE TEMPORARY TABLE local_table (
> `user_id` STRING, `order_id` STRING, `amount` INT, `restaurant_id` STRING,
> `experiment` ARRAY<MAP<STRING,STRING>>, `dt` STRING
> ) PARTITIONED BY (`dt`) WITH (
> 'connector'='filesystem',
> 'path'='file:///tmp/test_hadoop_write',
> 'format'='parquet',
> 'auto-compaction'='true',
> 'sink.partition-commit.policy.kind'='success-file'
> ) {code}
> PS. It is used to work in Flink version 1.15.1
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
