[
https://issues.apache.org/jira/browse/HIVE-29728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Turoczy updated HIVE-29728:
----------------------------------
Description:
Remove Hive’s dependency on ZooKeeper by introducing a replaceable
service-discovery mechanism for LLAP coordinators and executors.
Hive currently uses ZooKeeper for LLAP coordinator and executor discovery. The
stored information is relatively lightweight and mainly consists of small
metadata entries describing the currently available coordinators.
Maintaining ZooKeeper solely for this purpose introduces additional
infrastructure complexity, operational cost, and security overhead. These
concerns are particularly relevant in cloud environments, where native
service-discovery capabilities may provide a simpler and more efficient
alternative.
h2. Goal
Design and implement a service-discovery abstraction that removes the direct
dependency between Hive LLAP and ZooKeeper.
The solution should:
* Support coordinator and executor registration and discovery.
* Provide a well-defined and replaceable interface.
* Allow different implementations depending on the deployment environment.
* Support AWS, Azure, and GCP deployments.
* Allow the use of cloud-native services or another suitable backend behind
the common interface.
* Preserve the required availability, consistency, and failure-detection
characteristics of the current ZooKeeper-based implementation.
h2. Expected Benefits
* Reduced infrastructure and operational costs.
* Simplified deployment and maintenance.
* Smaller security and compliance footprint.
* Better alignment with cloud-native architectures.
* Reduced dependency on an additional distributed system.
h2. Design Considerations and Potential Pitfalls
h3. On-Premises Deployments
A replacement must also be defined for on-premises environments where
cloud-native service-discovery solutions are unavailable.
Potential options may include:
* Retaining ZooKeeper as one supported implementation.
* Using Kubernetes-native service discovery where applicable.
* Providing another lightweight, self-managed implementation.
* Backward compatibility
h3. HiveServer2 Active/Active
The design must evaluate any impact on HiveServer2 Active/Active deployments,
including:
* Instance registration and discovery.
* Leader election or coordination requirements, where applicable.
* Failure detection and stale registration cleanup.
* Compatibility with existing client-routing and fail-over mechanisms.
was:
We should remove Hive's dependency on Zookeeper. This move offers significant
benefits, cutting costs, simplifying our infrastructure, and boosting security,
especially in Cloud environments.
Currently the zookeper is used for LLAP coordinator / executor discorvery. This
mainly contains a small text entry about the currently available coordinators.
For cloud we should have some kind of native service discovery solution. The
solution should be replaceable. Aka should have a support for AWS, Azure or
GCP. Can be native service or anything behind a well definde interface.
Possible pitholes:
On-Prem how it would look like
HS2 Active / Active
> Zookeeper removal from Hive stack
> ---------------------------------
>
> Key: HIVE-29728
> URL: https://issues.apache.org/jira/browse/HIVE-29728
> Project: Hive
> Issue Type: New Feature
> Reporter: Attila Turoczy
> Priority: Major
> Labels: cloud
>
> Remove Hive’s dependency on ZooKeeper by introducing a replaceable
> service-discovery mechanism for LLAP coordinators and executors.
> Hive currently uses ZooKeeper for LLAP coordinator and executor discovery.
> The stored information is relatively lightweight and mainly consists of small
> metadata entries describing the currently available coordinators.
> Maintaining ZooKeeper solely for this purpose introduces additional
> infrastructure complexity, operational cost, and security overhead. These
> concerns are particularly relevant in cloud environments, where native
> service-discovery capabilities may provide a simpler and more efficient
> alternative.
> h2. Goal
> Design and implement a service-discovery abstraction that removes the direct
> dependency between Hive LLAP and ZooKeeper.
> The solution should:
> * Support coordinator and executor registration and discovery.
> * Provide a well-defined and replaceable interface.
> * Allow different implementations depending on the deployment environment.
> * Support AWS, Azure, and GCP deployments.
> * Allow the use of cloud-native services or another suitable backend behind
> the common interface.
> * Preserve the required availability, consistency, and failure-detection
> characteristics of the current ZooKeeper-based implementation.
> h2. Expected Benefits
> * Reduced infrastructure and operational costs.
> * Simplified deployment and maintenance.
> * Smaller security and compliance footprint.
> * Better alignment with cloud-native architectures.
> * Reduced dependency on an additional distributed system.
> h2. Design Considerations and Potential Pitfalls
> h3. On-Premises Deployments
> A replacement must also be defined for on-premises environments where
> cloud-native service-discovery solutions are unavailable.
> Potential options may include:
> * Retaining ZooKeeper as one supported implementation.
> * Using Kubernetes-native service discovery where applicable.
> * Providing another lightweight, self-managed implementation.
> * Backward compatibility
> h3. HiveServer2 Active/Active
> The design must evaluate any impact on HiveServer2 Active/Active deployments,
> including:
> * Instance registration and discovery.
> * Leader election or coordination requirements, where applicable.
> * Failure detection and stale registration cleanup.
> * Compatibility with existing client-routing and fail-over mechanisms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
