[
https://issues.apache.org/jira/browse/HIVE-27475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshat Mathur updated HIVE-27475:
---------------------------------
Description:
It was needed to shade a version of zstd-jni that is compatible with parquet in
hive-ql after upgrading parquet version dowstream, otherwise Tez couldn't work
with parquet and zstd encryption. Parquet and kafka-client both use zstd-jni.
In hive-ql, zstd-jni is coming as a transitive dependency from kafka-client.
The zstd-jni version in kafka-client 2.5.0 was not compatible with parquet, so
in PR-4082 kafka-client was upgraded to latest version which used zstd-jni
version close to the version in parquet.
While the upgrade fixed the zstd-ini issue, It introduced multiple
compatibility issue in the Kafka-handler.
It makes more sense to downgrade kafka versions back to 2.5.0, stabalize
kafka-handler, enable test and then upgrade.
was:
Based on the discussion on [4436|https://github.com/apache/hive/pull/4436]
It makes more sense to downgrade kafka versions back to 2.5.0, stabalize
kafka-handler, enable test and then upgrade.
> Revert Kafka version to stabilise Kafka handler
> ------------------------------------------------
>
> Key: HIVE-27475
> URL: https://issues.apache.org/jira/browse/HIVE-27475
> Project: Hive
> Issue Type: Task
> Components: kafka integration
> Affects Versions: 4.0.0-alpha-2
> Reporter: Akshat Mathur
> Assignee: Akshat Mathur
> Priority: Major
> Labels: pull-request-available
>
> It was needed to shade a version of zstd-jni that is compatible with parquet
> in hive-ql after upgrading parquet version dowstream, otherwise Tez couldn't
> work with parquet and zstd encryption. Parquet and kafka-client both use
> zstd-jni. In hive-ql, zstd-jni is coming as a transitive dependency from
> kafka-client. The zstd-jni version in kafka-client 2.5.0 was not compatible
> with parquet, so in PR-4082 kafka-client was upgraded to latest version which
> used zstd-jni version close to the version in parquet.
> While the upgrade fixed the zstd-ini issue, It introduced multiple
> compatibility issue in the Kafka-handler.
> It makes more sense to downgrade kafka versions back to 2.5.0, stabalize
> kafka-handler, enable test and then upgrade.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
