[
https://issues.apache.org/jira/browse/FLINK-23074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372601#comment-17372601
]
Ada Wong edited comment on FLINK-23074 at 7/1/21, 10:41 AM:
------------------------------------------------------------
[~luoyuxia]
I custom a HDFS connector, it use ExampleParquetWriter class included in
flink-parquet.
!screenshot-4.png!
when call ExampleParquetWriter.Builder#build and already loaded
hive-connector-2.3.6, it will throw the following error.
{code:java}
Caused by: java.lang.NoSuchMethodError:
org.apache.parquet.hadoop.example.GroupWriteSupport.<init>(Lorg/apache/parquet/schema/MessageType;Ljava/util/Map;)V
at
org.apache.parquet.hadoop.example.ExampleParquetWriter$Builder.getWriteSupport(ExampleParquetWriter.java:114)
at
org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:489)
{code}
So I want to shade hive-connector all version. Because hive is often used as
the Flink catalog.
GroupWriteSupport in hive-connector
!image-2021-07-01-18-23-47-105.png!
GroupWriteSupport in flink-parquet
!image-2021-07-01-18-40-00-991.png!
!image-2021-07-01-18-40-31-729.png!
was (Author: ana4):
[~luoyuxia]
I custom a HDFS connector, it use ExampleParquetWriter class included in
flink-parquet.
!screenshot-4.png!
when call ExampleParquetWriter.Builder#build and already loaded
hive-connector-2.3.6, it will throw the following error.
{code:java}
Caused by: java.lang.NoSuchMethodError:
org.apache.parquet.hadoop.example.GroupWriteSupport.<init>(Lorg/apache/parquet/schema/MessageType;Ljava/util/Map;)V
at
org.apache.parquet.hadoop.example.ExampleParquetWriter$Builder.getWriteSupport(ExampleParquetWriter.java:114)
at
org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:489)
{code}
So I want to shade hive-connector all version. Because hive is often used as
the Flink catalog.
!image-2021-07-01-18-23-47-105.png!
> There is a class conflict between flink-connector-hive and flink-parquet
> ------------------------------------------------------------------------
>
> Key: FLINK-23074
> URL: https://issues.apache.org/jira/browse/FLINK-23074
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
> Affects Versions: 1.13.1, 1.12.4
> Reporter: Ada Wong
> Assignee: Ada Wong
> Priority: Major
> Attachments: E8C394D1-F970-4825-82CD-3EFA74C65B27.png,
> image-2021-06-23-17-26-32-559.png, image-2021-07-01-18-23-47-105.png,
> image-2021-07-01-18-40-00-991.png, image-2021-07-01-18-40-31-729.png,
> screenshot-1.png, screenshot-3.png, screenshot-4.png
>
>
> flink-connector-hive 2.3.6 include parquet-hadoop 1.8.1 version but
> flink-parquet include 1.11.1.
> org.apache.parquet.hadoop.example.GroupWriteSupport
> is different.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
