[
https://issues.apache.org/jira/browse/HDFS-17945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiang he updated HDFS-17945:
----------------------------
Description:
With an invalid checkpoint directory / user home configuration, HDFS client
initialization fails with a bare NullPointerException.
The error message does not identify the invalid configuration key or value.
Reproduction config
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.datanode.data.read.bandwidthPerSec</name>
<value>0</value>
</property>
<property>
<name>dfs.namenode.block.deletion.lock.threshold.ms</name>
<value>-1</value>
</property>
<property>
<name>dfs.user.home.dir.prefix</name>
<value>@name@</value>
</property>
<property>
<name>dfs.namenode.checkpoint.edits.dir</name>
<value>valid/dir</value>
</property>
<property>
<name>dfs.namenode.block.deletion.unlock.interval.ms</name>
<value>99999999</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file://${hadoop.tmp.dir}/dfs/namesecondary</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>@name@</value>
</property>
</configuration>
Observed
Failed to initialize filesystem hdfs://127.0.0.1:9000:
java.lang.NullPointerException
Expected
HDFS should reject the invalid configuration with a clear error message
that identifies the invalid key/value, instead of exposing a bare
NullPointerException.
Control test
Using normal path values passes successfully:
dfs.user.home.dir.prefix=/user
dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}
dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary
was:
Several HDFS client integer configuration keys fail with a raw
NumberFormatException when set to an out-of-range negative value.
Reproduced with Hadoop 3.4.3.
Examples:
1. dfs.client.block.write.retries=-4294967296
Observed:
Failed to initialize filesystem hdfs://127.0.0.1:9000:
java.lang.NumberFormatException: For input string: "-4294967296"
2. dfs.client.pipeline.recovery.max-retries=-4294967296
Observed:
Failed to initialize filesystem hdfs://127.0.0.1:9000:
java.lang.NumberFormatException: For input string: "-4294967296"
3. dfs.client.retry.max.attempts=-4294967296
Observed:
Failed to initialize filesystem hdfs://127.0.0.1:9000:
java.lang.NumberFormatException: For input string: "-4294967296"
4. dfs.client.congestion.backoff.mean.time=-4294967296
Observed:
java.lang.NumberFormatException: For input string: "-4294967296"
Expected:
HDFS should report a clear configuration validation error that identifies
the invalid key and explains the valid range, instead of exposing a raw
NumberFormatException.
Control test:
The following normal values pass successfully:
dfs.client.block.write.retries=3
dfs.client.pipeline.recovery.max-retries=5
dfs.client.retry.max.attempts=10
dfs.client.congestion.backoff.mean.time=5000
> Invalid checkpoint directory configuration can cause NullPointerException
> during HDFS FileSystem initialization
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17945
> URL: https://issues.apache.org/jira/browse/HDFS-17945
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
> Affects Versions: 3.4.3
> Reporter: jiang he
> Priority: Major
>
> With an invalid checkpoint directory / user home configuration, HDFS client
> initialization fails with a bare NullPointerException.
> The error message does not identify the invalid configuration key or value.
> Reproduction config
> <configuration>
> <property>
> <name>dfs.replication</name>
> <value>1</value>
> </property>
> <property>
> <name>dfs.datanode.data.read.bandwidthPerSec</name>
> <value>0</value>
> </property>
> <property>
> <name>dfs.namenode.block.deletion.lock.threshold.ms</name>
> <value>-1</value>
> </property>
> <property>
> <name>dfs.user.home.dir.prefix</name>
> <value>@name@</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.edits.dir</name>
> <value>valid/dir</value>
> </property>
> <property>
> <name>dfs.namenode.block.deletion.unlock.interval.ms</name>
> <value>99999999</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
> <value>file://${hadoop.tmp.dir}/dfs/namesecondary</value>
> </property>
> <property>
> <name>dfs.namenode.checkpoint.dir</name>
> <value>@name@</value>
> </property>
> </configuration>
> Observed
> Failed to initialize filesystem hdfs://127.0.0.1:9000:
> java.lang.NullPointerException
> Expected
> HDFS should reject the invalid configuration with a clear error message
> that identifies the invalid key/value, instead of exposing a bare
> NullPointerException.
> Control test
> Using normal path values passes successfully:
> dfs.user.home.dir.prefix=/user
> dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}
> dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
