[
https://issues.apache.org/jira/browse/HIVE-29728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Turoczy updated HIVE-29728:
----------------------------------
Summary: Zookeeper removal from Hive LLAP Service-discovery (was:
Zookeeper removal from Hive stack)
> Zookeeper removal from Hive LLAP Service-discovery
> ---------------------------------------------------
>
> Key: HIVE-29728
> URL: https://issues.apache.org/jira/browse/HIVE-29728
> Project: Hive
> Issue Type: New Feature
> Reporter: Attila Turoczy
> Priority: Major
> Labels: cloud
>
> Remove Hive’s dependency on ZooKeeper by introducing a replaceable
> service-discovery mechanism for LLAP coordinators and executors.
> Hive currently uses ZooKeeper for LLAP coordinator and executor discovery.
> The stored information is relatively lightweight and mainly consists of small
> metadata entries describing the currently available coordinators.
> Maintaining ZooKeeper solely for this purpose introduces additional
> infrastructure complexity, operational cost, and security overhead. These
> concerns are particularly relevant in cloud environments, where native
> service-discovery capabilities may provide a simpler and more efficient
> alternative.
> h2. Goal
> Design and implement a service-discovery abstraction that removes the direct
> dependency between Hive LLAP and ZooKeeper.
> The solution should:
> * Support coordinator and executor registration and discovery.
> * Provide a well-defined and replaceable interface.
> * Allow different implementations depending on the deployment environment.
> * Support AWS, Azure, and GCP deployments.
> * Allow the use of cloud-native services or another suitable backend behind
> the common interface.
> * Preserve the required availability, consistency, and failure-detection
> characteristics of the current ZooKeeper-based implementation.
> h2. Expected Benefits
> * Reduced infrastructure and operational costs.
> * Simplified deployment and maintenance.
> * Smaller security and compliance footprint.
> * Better alignment with cloud-native architectures.
> * Reduced dependency on an additional distributed system.
> h2. Design Considerations and Potential Pitfalls
> h3. On-Premises Deployments
> A replacement must also be defined for on-premises environments where
> cloud-native service-discovery solutions are unavailable.
> Potential options may include:
> * Retaining ZooKeeper as one supported implementation.
> * Using Kubernetes-native service discovery where applicable.
> * Providing another lightweight, self-managed implementation.
> * Backward compatibility
> h3. HiveServer2 Active/Active
> The design must evaluate any impact on HiveServer2 Active/Active deployments,
> including:
> * Instance registration and discovery.
> * Leader election or coordination requirements, where applicable.
> * Failure detection and stale registration cleanup.
> * Compatibility with existing client-routing and fail-over mechanisms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
