[
https://issues.apache.org/jira/browse/HDFS-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062059#comment-18062059
]
ASF GitHub Bot commented on HDFS-16842:
---------------------------------------
balodesecurity opened a new pull request, #8291:
URL: https://github.com/apache/hadoop/pull/8291
## Problem
When `DataNode.refreshNamenodes()` is called to add a new nameservice it
reloads configuration via `setConf(new Configuration())`, but
`DatanodeHttpServer` kept holding a stale reference to the old `Configuration`.
Subsequent WebHDFS requests for blocks on the newly-added nameservice would
fail because the `DFSClient` created per-request could not resolve the namenode
addresses for that nameservice from the stale config.
Root cause: `DatanodeHttpServer.conf` and `confForCreate` were `final`
fields set once at construction time. `DataNode.refreshNamenodes()` never
called any update method on `httpServer`.
## Fix
- Make `conf` and `confForCreate` in `DatanodeHttpServer` `volatile`
(non-final) so Netty I/O threads see updates atomically.
- Add `DatanodeHttpServer.updateConf(Configuration newConf)` that refreshes
both fields and the embedded Jetty `infoServer`'s servlet-context attributes
(`CONF_CONTEXT_ATTRIBUTE`, `CURRENT_CONF`).
- Call `httpServer.updateConf(getConf())` at the end of
`DataNode.refreshNamenodes()` after the new configuration has been loaded.
The Netty `ChannelInitializer` creates a fresh `URLDispatcher` per
connection and reads `DatanodeHttpServer.conf` at that point, so new
connections automatically pick up the updated configuration without requiring a
pipeline or server restart.
## Testing
- Added `TestDatanodeHttpServerUpdateConf.testUpdateConfRefreshesFields` — a
unit test (no cluster required) that:
- Creates a `DatanodeHttpServer` with an initial configuration.
- Calls `updateConf(newConf)`.
- Asserts (via reflection) that the `conf` and `confForCreate` instance
fields now reference the updated configuration.
- Asserts that the Jetty `infoServer` servlet-context attributes have been
updated.
```
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
```
## Related
- JIRA: https://issues.apache.org/jira/browse/HDFS-16842
🤖 Generated with [Claude Code](https://claude.com/claude-code)
> DatanodeHttpServer cannot get the new configuration of the new nameservice
> even after refreshNamenodes is run
> -------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-16842
> URL: https://issues.apache.org/jira/browse/HDFS-16842
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
> Affects Versions: 3.1.2
> Reporter: Jihye Seo
> Priority: Major
> Labels: patch-available
> Fix For: 3.1.2
>
> Attachments: HDFS-16842.001.patch
>
>
> 'refreshNamenodes' should be run when a new nameservice is added.
> However, a 'DatanodeHttpServer' cannot get the new configuration of the new
> nameservice even after 'refreshNamenodes' is run.
> Therefore, when a client tries to access the new namespace's data with a
> swebhdfs or REST API request, an error occurs as follows.
> {code:java}
> Does not contain a valid host:port authority: NEW_NAMESPACE{code}
> When 'refreshNamenodes' is executed, a Datanode object reads the new
> configuration file and updates its configuration with 'setConf(new
> Configuration)'.
> {code:java}
> public void refreshNamenodes() throws IOException {
> checkSuperuserPrivilege();
> setConf(new Configuration());
> refreshNamenodes(getConf());
> }{code}
> On the other hand, the configuration of a DatanodeHttpServer object remains
> unchanged after the DatanodeHttpServer object is created.
> When a Datanode object is created, a DatanodeHttpServer is created by copying
> the Datanode's configuration. Since then, there's no way to update its
> configuration.
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.restCsrfPreventionFilter = createRestCsrfPreventionFilter(conf);
> this.conf = conf;
> ...
> } {code}
> This causes a DatanodeHttpServer to be unable to interpret a request for the
> new name service.
> Therefore, a DatanodeHttpServer also needs to update its configuration to
> handle a request for a new name service.
> I attached a patch based on the trunk.
>
> Thanks!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
