[
https://issues.apache.org/jira/browse/FLINK-32760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17758570#comment-17758570
]
luoyuxia edited comment on FLINK-32760 at 8/24/23 12:50 PM:
------------------------------------------------------------
master: 6bb5a12eb0fd106f14538bb716650417a46adf58
todo:
1.18:
1.17:
1.16:
was (Author: luoyuxia):
master: 6bb5a12eb0fd106f14538bb716650417a46adf58
todo:
1.17:
1.18:
1.16:
> Version Conflict in flink-sql-connector-hive for shaded.parquet prefix
> packages
> -------------------------------------------------------------------------------
>
> Key: FLINK-32760
> URL: https://issues.apache.org/jira/browse/FLINK-32760
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
> Affects Versions: 1.17.1
> Reporter: dongwoo.kim
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-08-05-14-50-47-806.png
>
>
> h2. Summary
> In https://issues.apache.org/jira/browse/FLINK-23074 it seems like shading
> parquet dependency from *hive-exec* is done.
> But I think this is not enough and causing errors like below when I try to
> read parquet file using sql-gateway which requires both *flink-parquet* and
> *flink-sql-connector-hive* dependencies.
> !image-2023-08-05-14-50-47-806.png|width=1392,height=909!
>
> h2. {color:#172b4d}Cause{color}
> {color:#172b4d}Parquet dependency not only includes *org.apache.parquet* but
> also *shaded.parquet* prefix dependencies.
> ([ref|https://github.com/apache/parquet-mr/blob/515734c373f69b5250e8b63eb3d1c973da893b63/pom.xml#L72]){color}
> {color:#172b4d}So we need to shade both.{color}
> {color:#172b4d}- flink-parquet depends on Parquet 1.12.3 with shaded Thrift
> 0.16.0 (prefix: {{{}shaded.parquet{}}}){color}
> {color:#172b4d}- flink-sql-connector-hive depends on hive-exec 3.1.3 with
> Parquet 1.10.0 and shaded Thrift 0.9.3 (prefix: {{{}shaded.parquet{}}}){color}
> {color:#172b4d}- Code compiled against Thrift 0.16.0 attempts to run against
> 0.9.3, causing the error.{color}
> h2. {color:#172b4d}Proposed solution{color}
> Adding new shading rule to flink-sql-connector-hive project.
> I have confirmed that adding this rule could resolve the above error.
> {code:xml}
> <relocation>
> <pattern>shaded.parquet</pattern>
> <shadedPattern>shaded.parquet.flink.hive.shaded</shadedPattern>
> </relocation>{code}
>
> I would be happy to implement it if the proposal is accepted. Thanks
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
