[
https://issues.apache.org/jira/browse/HADOOP-18802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HADOOP-18802:
---------------------------
Description:
h2. What happened:
When manually set fs.viewfs.mounttable.default.name.key to default (the same as
default value) in HCommon, test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of InodeTree, the tree attempts to get all the mount table
entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry<String, String> si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here mountTablePrefix="fs.viewfs.mounttable.default.". However, it just so
happens that the name of the configuration users use to specify the default
mount table is fs.viewfs.mounttable.default.name.key. Thus, if a user specifies
the default mount table and uses InodeTree the name.key would be falsely parsed
as the entry point to one of the mount tables, which would cause InodeTree to
throw an exception since name.key is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set fs.viewfs.mounttable.default.name.key to default
(2) Run test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
You can use the reproduce.sh in the attachment to easily reproduce the bug.
We are happy to provide a patch if this issue is confirmed.
was:
h2. What happened:
When manually set fs.viewfs.mounttable.default.name.key to default (the same as
default value) in HCommon, test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
would fail.
But the test can pass if this parameter is not manually set in the
configuration file.
h2. Where's the bug:
In the constructor of InodeTree, the tree attempts to get all the mount table
entry points set by user in the configuration and process them one by one:
{code:java}
for (Entry<String, String> si : config) {
final String key = si.getKey();
if (!key.startsWith(mountTablePrefix)) {
continue;
}
gotMountTableEntry = true;
LinkType linkType;
String src = key.substring(mountTablePrefix.length());
...
{code}
Here mountTablePrefix="fs.viewfs.mounttable.default.". However, it just so
happens that the name of the configuration users use to specify the default
mount table is fs.viewfs.mounttable.default.name.key. Thus, if a user specifies
the default mount table and uses InodeTree the name.key would be falsely parsed
as the entry point to one of the mount tables, which would cause InodeTree to
throw an exception since name.key is not a valid entry.
h2. Stack trace:
{code:java}
java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
Invalid entry in Mount table in config: name.key
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
at
org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
at
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
h2. How to reproduce:
(1) Set fs.viewfs.mounttable.default.name.key to default
(2) Run test
org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
You can use the reproduce.sh in the attachment to easily reproduce the bug:
We are happy to provide a patch if this issue is confirmed.
> Collision of config key name fs.viewfs.mounttable.default.name.key to other
> keys that specify the entry point to mount tables
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18802
> URL: https://issues.apache.org/jira/browse/HADOOP-18802
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, conf, fs
> Reporter: ConfX
> Priority: Critical
> Attachments: reproduce.sh
>
>
> h2. What happened:
> When manually set fs.viewfs.mounttable.default.name.key to default (the same
> as default value) in HCommon, test
> org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
> would fail.
> But the test can pass if this parameter is not manually set in the
> configuration file.
> h2. Where's the bug:
> In the constructor of InodeTree, the tree attempts to get all the mount table
> entry points set by user in the configuration and process them one by one:
> {code:java}
> for (Entry<String, String> si : config) {
> final String key = si.getKey();
> if (!key.startsWith(mountTablePrefix)) {
> continue;
> }
>
> gotMountTableEntry = true;
> LinkType linkType;
> String src = key.substring(mountTablePrefix.length());
> ...
> {code}
> Here mountTablePrefix="fs.viewfs.mounttable.default.". However, it just so
> happens that the name of the configuration users use to specify the default
> mount table is fs.viewfs.mounttable.default.name.key. Thus, if a user
> specifies the default mount table and uses InodeTree the name.key would be
> falsely parsed as the entry point to one of the mount tables, which would
> cause InodeTree to throw an exception since name.key is not a valid entry.
> h2. Stack trace:
> {code:java}
> java.lang.RuntimeException: java.io.IOException: ViewFs: Cannot initialize:
> Invalid entry in Mount table in config: name.key
> at
> org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:470)
> at
> org.apache.hadoop.fs.viewfs.ViewFsTestSetup.setupForViewFsLocalFs(ViewFsTestSetup.java:88)
> at
> org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs.setUp(TestFcMainOperationsLocalFs.java:38){code}
> h2. How to reproduce:
> (1) Set fs.viewfs.mounttable.default.name.key to default
> (2) Run test
> org.apache.hadoop.fs.viewfs.TestFcMainOperationsLocalFs#testGlobStatusWithMultipleWildCardMatches
> You can use the reproduce.sh in the attachment to easily reproduce the bug.
> We are happy to provide a patch if this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
