[
https://issues.apache.org/jira/browse/HIVE-29728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097810#comment-18097810
]
László Bodor edited comment on HIVE-29728 at 7/21/26 10:15 AM:
---------------------------------------------------------------
[~tanishqchugh] has been working on an excellent patch that makes the zookeeper
/ k8s-based LLAP discovery completely pluggable, having them share the same
interface, I believe that's exactly what we need here
regarding "HiveServer2 Active/Active": be careful while mentioning it here; HS2
is not supposed to handle LLAP daemons at all
a single exception I know is the proactive LLAP cache eviction (on table drop),
but it's supposed to be a best-effort call without the need for sophisticated
error handling, like: 3 HS2 instances call the cache eviction -> just do it;
some of them are going to succeed
"Providing another lightweight, self-managed implementation." for LLAP
discovery? I would definitely not do it for a simple service discovery, because
it ultimately means a new hive "always-running" component, leading to
unnecessary complexity
was (Author: abstractdog):
[~tanishqchugh] has been working on an excellent patch that makes the zookeeper
/ k8s-based LLAP discovery completely pluggable, having them share the same
interface, I believe that's exactly what we need here
regarding "HiveServer2 Active/Active": be careful while mentioning it here; HS2
is not supposed to handle LLAP daemons at all
a single exception I know is the proactive LLAP cache eviction (on table drop),
but it's supposed to be a best-effort call without the need for sophisticated
error handling, like: 3 HS2 instances call the cache eviction -> just do it;
some of them are going to succeed
> Zookeeper removal from Hive LLAP Service-discovery
> ---------------------------------------------------
>
> Key: HIVE-29728
> URL: https://issues.apache.org/jira/browse/HIVE-29728
> Project: Hive
> Issue Type: New Feature
> Reporter: Attila Turoczy
> Priority: Major
> Labels: cloud
>
> Remove Hive’s dependency on ZooKeeper by introducing a replaceable
> service-discovery mechanism for LLAP coordinators and executors.
> Hive currently uses ZooKeeper for LLAP coordinator and executor discovery.
> The stored information is relatively lightweight and mainly consists of small
> metadata entries describing the currently available coordinators.
> Maintaining ZooKeeper solely for this purpose introduces additional
> infrastructure complexity, operational cost, and security overhead. These
> concerns are particularly relevant in cloud environments, where native
> service-discovery capabilities may provide a simpler and more efficient
> alternative.
> h2. Goal
> Design and implement a service-discovery abstraction that removes the direct
> dependency between Hive LLAP and ZooKeeper.
> The solution should:
> * Support coordinator and executor registration and discovery.
> * Provide a well-defined and replaceable interface.
> * Allow different implementations depending on the deployment environment.
> * Support AWS, Azure, and GCP deployments.
> * Allow the use of cloud-native services or another suitable backend behind
> the common interface.
> * Preserve the required availability, consistency, and failure-detection
> characteristics of the current ZooKeeper-based implementation.
> h2. Expected Benefits
> * Reduced infrastructure and operational costs.
> * Simplified deployment and maintenance.
> * Smaller security and compliance footprint.
> * Better alignment with cloud-native architectures.
> * Reduced dependency on an additional distributed system.
> h2. Design Considerations and Potential Pitfalls
> h3. On-Premises Deployments
> A replacement must also be defined for on-premises environments where
> cloud-native service-discovery solutions are unavailable.
> Potential options may include:
> * Retaining ZooKeeper as one supported implementation.
> * Using Kubernetes-native service discovery where applicable.
> * Providing another lightweight, self-managed implementation.
> * Backward compatibility
> h3. HiveServer2 Active/Active
> The design must evaluate any impact on HiveServer2 Active/Active deployments,
> including:
> * Instance registration and discovery.
> * Leader election or coordination requirements, where applicable.
> * Failure detection and stale registration cleanup.
> * Compatibility with existing client-routing and fail-over mechanisms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
