dannytbecker opened a new pull request, #6018:
URL: https://github.com/apache/hadoop/pull/6018
### Description of PR
During rollingUpgrade, bootstrapStandby will fail with an exception due to
different NameNodeLayoutVersions. We can ignore this safely during
RollingUpgrade because different NameNodeLayoutVersions are expected.
NameNodes will not be able to recover with BootstrapStandby if they go
through destructive repair before the rollingUpgrade has been finalized.
Error during BootstrapStandby before change:
```
=====================================================
About to bootstrap Standby ID nn2 from:
Nameservice ID: MTPrime-MWHE01-0
Other Namenode ID: nn1
Other NN's HTTP address: https://MWHEEEAP002D9A2:81
Other NN's IPC address: MWHEEEAP002D9A2.ap.gbl/10.59.208.18:8020
Namespace ID: 895912530
Block pool ID: BP-1556042256-10.99.154.61-1663325602669
Cluster ID: MWHE01
Layout version: -64
isUpgradeFinalized: true
=====================================================
2023-08-28T19:35:06,940 ERROR [main] namenode.NameNode: Failed to start
namenode.
java.io.IOException: java.lang.RuntimeException:
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException:
Image transfer servlet at
https://MWHEEEAP002D9A2:81/imagetransfer?getimage=1&txid=25683470&storageInfo=-64:895912530:1663325602669:MWHE01&bootstrapstandby=true
failed with status code 403
Response message:
This namenode has storage info -63:895912530:1663325602669:MWHE01 but the
secondary expected -64:895912530:1663325602669:MWHE01
at
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:583)
~[hadoop-hdfs-2.9.2-MT-SNAPSHOT.jar:?]
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1717)
~[hadoop-hdfs-2.9.2-MT-SNAPSHOT.jar:?]
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1819)
[hadoop-hdfs-2.9.2-MT-SNAPSHOT.jar:?]
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpGetFailedException:
Image transfer servlet at https://MWHEEEAP002D9A2:81
```
This is caused because the namespaceInfo sent from the proxy node does not
include the effective layout version, which causes BootstrapStandby to send a
request with a storageinfo param using the service layout version. This causes
the proxy node to refuse the request, because it compares the storageinfo param
against its storage info, which uses the effective layout version, not the
service layout version.
To fix this we can modify the proxy.versionRequest() call stack to set the
layout version using the effective layout version on the proxy node. We can
then add logic to BootstrapStandby to properly handle the case where the proxy
node is in rolling upgrade.
### How was this patch tested?
Tested in environment doing a rolling upgrade from Hadoop 2.9 to Hadoop 3.3
before finalizing the environment. BootstrapStandby would not work for Hadoop
3.3 NameNodes because the proxy node was expecting -63, but the
BootstrapStandby tool was sending -66 layout version in the transfer image
request.
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
