[
https://issues.apache.org/jira/browse/HIVE-27475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akshat Mathur updated HIVE-27475:
---------------------------------
Description:
It was needed to shade a version of zstd-jni that is compatible with parquet in
hive-ql after upgrading parquet version dowstream, otherwise Tez couldn't work
with parquet and zstd encryption. Parquet and kafka-client both use zstd-jni.
In hive-ql, zstd-jni is coming as a transitive dependency from kafka-client.
The zstd-jni version in kafka-client 2.5.0 was not compatible with parquet, so
in [PR-4082|https://github.com/apache/hive/pull/4082] kafka-client was upgraded
to latest version which used zstd-jni version close to the version in parquet.
While the upgrade fixed the zstd-ini issue, It introduced multiple
compatibility issue in the Kafka-handler and due to lack to test cases and
disabled tests these issues went unidentified.
With the recent refactor in [HIVE-27402|http://example.com/], Kafka-client
dependency was moved out of hive-ql, Which fixed the original zstd-ini
issue.(confirmed by [~difin])
Hence , It makes more sense to downgrade kafka versions back to 2.5.0,
stabalize kafka-handler, enable test and then upgrade.
Ref to discussion can be found here:
[#4436|https://github.com/apache/hive/pull/4436]
was:
It was needed to shade a version of zstd-jni that is compatible with parquet in
hive-ql after upgrading parquet version dowstream, otherwise Tez couldn't work
with parquet and zstd encryption. Parquet and kafka-client both use zstd-jni.
In hive-ql, zstd-jni is coming as a transitive dependency from kafka-client.
The zstd-jni version in kafka-client 2.5.0 was not compatible with parquet, so
in [PR-4082|https://github.com/apache/hive/pull/4082] kafka-client was upgraded
to latest version which used zstd-jni version close to the version in parquet.
While the upgrade fixed the zstd-ini issue, It introduced multiple
compatibility issue in the Kafka-handler.
With the recent refactor in [HIVE-27402|http://example.com/], Kafka-client
dependency was moved out of hive-ql, Which fixed the original zstd-ini
issue.(confirmed by [~difin])
Hence , It makes more sense to downgrade kafka versions back to 2.5.0,
stabalize kafka-handler, enable test and then upgrade.
Ref to discussion can be found here:
[#4436|https://github.com/apache/hive/pull/4436]
> Revert Kafka version to stabilise Kafka handler
> ------------------------------------------------
>
> Key: HIVE-27475
> URL: https://issues.apache.org/jira/browse/HIVE-27475
> Project: Hive
> Issue Type: Task
> Components: kafka integration
> Affects Versions: 4.0.0-alpha-2
> Reporter: Akshat Mathur
> Assignee: Akshat Mathur
> Priority: Major
> Labels: pull-request-available
>
> It was needed to shade a version of zstd-jni that is compatible with parquet
> in hive-ql after upgrading parquet version dowstream, otherwise Tez couldn't
> work with parquet and zstd encryption. Parquet and kafka-client both use
> zstd-jni. In hive-ql, zstd-jni is coming as a transitive dependency from
> kafka-client. The zstd-jni version in kafka-client 2.5.0 was not compatible
> with parquet, so in [PR-4082|https://github.com/apache/hive/pull/4082]
> kafka-client was upgraded to latest version which used zstd-jni version close
> to the version in parquet.
> While the upgrade fixed the zstd-ini issue, It introduced multiple
> compatibility issue in the Kafka-handler and due to lack to test cases and
> disabled tests these issues went unidentified.
> With the recent refactor in [HIVE-27402|http://example.com/], Kafka-client
> dependency was moved out of hive-ql, Which fixed the original zstd-ini
> issue.(confirmed by [~difin])
> Hence , It makes more sense to downgrade kafka versions back to 2.5.0,
> stabalize kafka-handler, enable test and then upgrade.
> Ref to discussion can be found here:
> [#4436|https://github.com/apache/hive/pull/4436]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
