[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15026195#comment-15026195
]
Hudson commented on HDFS-8807:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #640 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/640/])
HDFS-8807. dfs.datanode.data.dir does not handle spaces between (szetszwo: rev
78ec38b2ede8bdf3874b2ae051af9580007a9ba1)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch, HDFS-8807.002.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15026141#comment-15026141
]
Hudson commented on HDFS-8807:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2577 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2577/])
HDFS-8807. dfs.datanode.data.dir does not handle spaces between (szetszwo: rev
78ec38b2ede8bdf3874b2ae051af9580007a9ba1)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch, HDFS-8807.002.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15026069#comment-15026069
]
Hudson commented on HDFS-8807:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #729 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/729/])
HDFS-8807. dfs.datanode.data.dir does not handle spaces between (szetszwo: rev
78ec38b2ede8bdf3874b2ae051af9580007a9ba1)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch, HDFS-8807.002.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025920#comment-15025920
]
Hudson commented on HDFS-8807:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1451 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1451/])
HDFS-8807. dfs.datanode.data.dir does not handle spaces between (szetszwo: rev
78ec38b2ede8bdf3874b2ae051af9580007a9ba1)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch, HDFS-8807.002.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025914#comment-15025914
]
Hudson commented on HDFS-8807:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #719 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/719/])
HDFS-8807. dfs.datanode.data.dir does not handle spaces between (szetszwo: rev
78ec38b2ede8bdf3874b2ae051af9580007a9ba1)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch, HDFS-8807.002.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025889#comment-15025889
]
Hudson commented on HDFS-8807:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2660 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2660/])
HDFS-8807. dfs.datanode.data.dir does not handle spaces between (szetszwo: rev
78ec38b2ede8bdf3874b2ae051af9580007a9ba1)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch, HDFS-8807.002.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025841#comment-15025841
]
Hudson commented on HDFS-8807:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8882 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8882/])
HDFS-8807. dfs.datanode.data.dir does not handle spaces between (szetszwo: rev
78ec38b2ede8bdf3874b2ae051af9580007a9ba1)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StorageLocation.java
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch, HDFS-8807.002.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025575#comment-15025575
]
Hadoop QA commented on HDFS-8807:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} trunk passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_85 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 52m 30s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 35s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_85. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.security.TestPermission |
| | hadoop.hdfs.TestWriteReadStripedFile |
| JDK v1.7.0_85 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.TestDatanodeDeath |
| | hadoop.security.TestPermission |
\\
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025454#comment-15025454
]
Anu Engineer commented on HDFS-8807:
sure thing, looks like my editor auto fixed the spaces in license header. I
will fix it and post a new patch.
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025418#comment-15025418
]
Tsz Wo Nicholas Sze commented on HDFS-8807:
---
{code}
- * http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
{code}
Indeed, we should not change the license header. Please revert it.
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639158#comment-14639158
]
Anu Engineer commented on HDFS-8807:
Failure does not seem to be related to this patch. These tests are passing with
this patch on my local machine.
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14638048#comment-14638048
]
Hadoop QA commented on HDFS-8807:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 20m 48s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 46s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 21s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 25s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 9s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 160m 32s | Tests failed in hadoop-hdfs. |
| | | 208m 19s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDistributedFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12746654/HDFS-8807.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 06e5dd2 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11792/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11792/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11792/console |
This message was automatically generated.
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.j
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637715#comment-14637715
]
Anu Engineer commented on HDFS-8807:
bq. If we had a DTD for the XML, what would it look like? We've basically built
an untestable monster.
I agree that we have a pretty complex config. I am not even sure if all the
assumptions we have about configs can be encoded in a DTD, especially things
like cross-key value dependencies.
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
> Attachments: HDFS-8807.001.patch
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637687#comment-14637687
]
Allen Wittenauer commented on HDFS-8807:
Fundamentally, this is the problem:
bq. The [StorageType][URI] notation
If we had a DTD for the XML, what would it look like? We've basically built an
untestable monster. If my app is not Hadoop and want to parse or build this
file, how is it supposed to be coded?
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637669#comment-14637669
]
Anu Engineer commented on HDFS-8807:
[~aw] agree, but this is not part of the URI. This is just a white space
*before* URI. I ran into this myself, and it is a trivial patch to fix. The
\[StorageType\]\[URI\] notation makes it tempting for users to put a space
between the type and URI.
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8807) dfs.datanode.data.dir does not handle spaces between storageType and URI correctly
[
https://issues.apache.org/jira/browse/HDFS-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637655#comment-14637655
]
Allen Wittenauer commented on HDFS-8807:
Technically, throwing an error on the space is correct; spaces aren't safe in
URIs and should be encoded.
> dfs.datanode.data.dir does not handle spaces between storageType and URI
> correctly
> --
>
> Key: HDFS-8807
> URL: https://issues.apache.org/jira/browse/HDFS-8807
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS
>Affects Versions: 2.8.0
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Fix For: 2.8.0
>
>
> if you add a space between the storage type and file URI then datanodes fail
> during startup.
> Here is an example of "mis-configration" that leads to datanode failure.
> {code}
>
> dfs.datanode.data.dir
>
> [DISK] file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> Here is the "fixed" version. Please *note* the lack of space between \[DISK\]
> and file URI
> {code}
>
> dfs.datanode.data.dir
>
> [DISK]file://tmp/hadoop-aengineer/disk1/dfs/data
>
>
> {code}
> we fail with a parsing error, here is the info from the datanode logs.
> {code}
> 2015-07-22 13:34:36,812 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in secureMain
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Illegal
> character in scheme name at index 0:
> file://tmp/hadoop-aengineer/disk1/dfs/data
> at org.apache.hadoop.fs.Path.initialize(Path.java:204)
> at org.apache.hadoop.fs.Path.(Path.java:170)
> at
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.parse(StorageLocation.java:97)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getStorageLocations(DataNode.java:2314)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2298)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2349)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2529)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2553)
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 0: file://tmp/hadoop-aengineer/disk1/dfs/data
> at java.net.URI$Parser.fail(URI.java:2829)
> at java.net.URI$Parser.checkChars(URI.java:3002)
> at java.net.URI$Parser.checkChar(URI.java:3012)
> at java.net.URI$Parser.parse(URI.java:3028)
> at java.net.URI.(URI.java:753)
> at org.apache.hadoop.fs.Path.initialize(Path.java:201)
> ... 7 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
