samad created HBASE-30245:
-----------------------------
Summary: RPC connection pinned to stale IP after cross-pod IP
reuse: NSRE storm persists indefinitely because pooled channel is reused
without re-resolving DNS
Key: HBASE-30245
URL: https://issues.apache.org/jira/browse/HBASE-30245
Project: HBase
Issue Type: Bug
Environment: * HBase 2.5.12 client (present in other client versions
as well)
* HBase deployed on Kubernetes.
* RegionServer and DataNode are co-hosted in the same pod.
* Multiple HBase clusters run in the same Kubernetes environment.
Reporter: samad
Summary: Under a narrow but real Kubernetes condition — a RegionServer pod's IP
is reassigned to a different live pod (sometimes a RegionServer of an entirely
different HBase cluster) — the async client gets stuck issuing requests to that
wrong-but-live server and receives a continuous stream of
{{NotServingRegionException}} (NSRE) for the affected regions. The condition
does not self-heal: only a client process restart fixes it.
We run HBase on Kubernetes, where pod cohost both a RegionServer and a
DataNode. Multiple independent HBase clusters share the same Kubernetes
environment.
We observed a failure scenario during node maintenance where an HBase async
client can become permanently stuck talking to the wrong RegionServer after
Kubernetes pod IP reuse.
Consider the following example:
* *Pod A* hosts *RegionServer A* belonging to {*}HBase Cluster A{*}.
* *Pod B* hosts *RegionServer B* belonging to {*}HBase Cluster B{*}.
* Both pods are running on the same Kubernetes node.
During a maintenance activity (node reboot, drain, upgrade, etc.), all pods on
the node restart.
A possible sequence is:
# Pod A goes down.
# Kubernetes later reassigns Pod A's old IP address to Pod B.
# The client already has an established TCP connection to Pod A's old IP.
# Because the connection remains alive through the networking/service-mesh
layer, the client does not see a transport failure.
# Requests intended for RegionServer A are now delivered to RegionServer B,
which belongs to a completely different HBase cluster and has never hosted the
requested regions.
# RegionServer B correctly responds with {{{}NotServingRegionException
(NSRE){}}}.
# The client continues reusing the same underlying RPC connection and it gets
following continuously
2026-06-09T06:03:15.226Z, org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:
app_ns:my-table,rowprefix-0001,1761217611582.65e37b957f2a12c0c710d3866bece520.
is not online on
hbase-B-dn-4.hbase-B.k8s-namespace.svc.cluster.local,16020,1780977222833
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3552)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3530)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1486)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2972)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Note: The table `app_ns:my-table` belongs to HBase cluster-A. The RegionServer
named in the NSRE (`hbase-B-dn-4`) belongs to a completely different HBase
cluster
(cluster-B). This table has never existed on cluster-B. The client is reaching
cluster-B's RegionServer only because cluster-B's pod acquired the IP address
that
previously belonged to cluster-A's pod (hbase-A-dn-7). After a Kubernetes node
maintenance event, both pods restarted on the same node. Cluster-B's pod came
up
first and was assigned cluster-A's old pod IP. The client's existing TCP
channel
was pinned to that IP and never re-resolved DNS, so all RPCs intended for
cluster-A's RegionServer are landing on cluster-B's RegionServer instead.
# Requests continue hitting RegionServer B and receive NSREs indefinitely.
In production, we observed this condition persisting for approximately an hour
and generating tens of thousands of NSREs. Recovery occurred only after
restarting the hbase client process
*Observed Behavior*
* Continuous NSREs for the same regions.
* The same RegionServer appears in all NSRE responses.
* The responding RegionServer belongs to a different HBase cluster than the
target table.
* No transport errors or connection failures are observed.
* Client restart immediately restores normal operation.
*Expected Behavior*
When the client receives repeated NSREs from a RegionServer that does not match
the expected destination, it should eventually drop the existing connection and
establish a fresh one, allowing DNS re-resolution and recovery without
requiring a client restart.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
