[
https://issues.apache.org/jira/browse/HDFS-15756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hbprotoss updated HDFS-15756:
-----------------------------
Description:
Affected version: all version with rbf
When RBF work with spark 2.4 client mode, there will be a chance that token is
missing across different nodes in RBF cluster. The root cause is that spark
renew the token(via resource manager) immediately after got one, as zookeeper
don't have a strong consistency guarantee after an update in cluster, zookeeper
client may read a stale value in some followers not synced with other nodes.
We apply a patch in spark, but it is still the problem of RBF. Is it possible
for RBF to replace the delegation token store using some other datasource(redis
for example)?
was:
When RBF work with spark 2.4 client mode, there will be a chance that token is
missing across different nodes in RBF cluster. The root cause is that spark
renew the token(via resource manager) immediately after got one, as zookeeper
don't have a strong consistency guarantee after an update in cluster, zookeeper
client may read a stale value in some followers not synced with other nodes.
We apply a patch in spark, but it is still the problem of RBF. Is it possible
for RBF to replace the delegation token store using some other datasource(redis
for example)?
> [RBF]Cannot get updated delegation token from zookeeper
> -------------------------------------------------------
>
> Key: HDFS-15756
> URL: https://issues.apache.org/jira/browse/HDFS-15756
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
> Affects Versions: 3.0.0
> Reporter: hbprotoss
> Priority: Major
>
> Affected version: all version with rbf
> When RBF work with spark 2.4 client mode, there will be a chance that token
> is missing across different nodes in RBF cluster. The root cause is that
> spark renew the token(via resource manager) immediately after got one, as
> zookeeper don't have a strong consistency guarantee after an update in
> cluster, zookeeper client may read a stale value in some followers not synced
> with other nodes.
>
> We apply a patch in spark, but it is still the problem of RBF. Is it possible
> for RBF to replace the delegation token store using some other
> datasource(redis for example)?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
