[
https://issues.apache.org/jira/browse/HDFS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yixiaohua updated HDFS-3307:
----------------------------
Component/s: name-node
Description:
this the log information of the exception from the SecondaryNameNode:
2012-03-28 00:48:42,553 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.IOException:
Found lease for
non-existent file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/????@???????????????
??????????tor.qzone.qq.com/keypart-00174
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
at java.lang.Thread.run(Thread.java:619)
this is the log information about the file from namenode:
2012-03-28 00:32:26,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34 cmd=create
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@? tor.qzone.qq.com/keypart-00174 dst=null
perm=boss:boss:rw-r--r--
2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@? tor.qzone.qq.com/keypart-00174.
blk_2751836614265659170_184668759
2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@? tor.qzone.qq.com/keypart-00174 is closed by
DFSClient_attempt_201203271849_0016_r_000174_0
2012-03-28 00:37:50,315 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
ip=/10.131.16.34 cmd=rename
src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
@? tor.qzone.qq.com/keypart-00174
dst=/user/boss/pgv/fission/task16/split/ @?
tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
after check the code that save FSImage,I found there are a problem that maybe a
bug of HDFS Code,I past below:
-------------this is the saveFSImage method in FSImage.java, I make some mark
at the problem code------------
/**
* Save the contents of the FS image to the file.
*/
void saveFSImage(File newFile) throws IOException {
FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
FSDirectory fsDir = fsNamesys.dir;
long startTime = FSNamesystem.now();
//
// Write out data
//
DataOutputStream out = new DataOutputStream(
new BufferedOutputStream(
new
FileOutputStream(newFile)));
try {
.........
// save the rest of the nodes
saveImage(strbuf, 0, fsDir.rootDir, out);------------------problem
fsNamesys.saveFilesUnderConstruction(out);------------------problem
detail is below
strbuf = null;
} finally {
out.close();
}
LOG.info("Image file of size " + newFile.length() + " saved in "
+ (FSNamesystem.now() - startTime)/1000 + " seconds.");
}
/**
* Save file tree image starting from the given root.
* This is a recursive procedure, which first saves all children of
* a current directory and then moves inside the sub-directories.
*/
private static void saveImage(ByteBuffer parentPrefix,
int prefixLength,
INodeDirectory current,
DataOutputStream out) throws IOException {
int newPrefixLength = prefixLength;
if (current.getChildrenRaw() == null)
return;
for(INode child : current.getChildren()) {
// print all children first
parentPrefix.position(prefixLength);
parentPrefix.put(PATH_SEPARATOR).put(child.getLocalNameBytes());------------------problem
saveINode2Image(parentPrefix, child, out);
}
..........
}
// Helper function that writes an INodeUnderConstruction
// into the input stream
//
static void writeINodeUnderConstruction(DataOutputStream out,
INodeFileUnderConstruction cons,
String path)
throws IOException {
writeString(path, out);------------------problem
..........
}
static private final UTF8 U_STR = new UTF8();
static void writeString(String str, DataOutputStream out) throws IOException {
U_STR.set(str);
U_STR.write(out);------------------problem
}
/**
* Converts a string to a byte array using UTF8 encoding.
*/
static byte[] string2Bytes(String str) {
try {
return str.getBytes("UTF8");------------------problem
} catch(UnsupportedEncodingException e) {
assert false : "UTF8 encoding is not supported ";
}
return null;
}
------------------------------------------below is the
explain------------------------
in saveImage method: child.getLocalNameBytes(),the bytes use the method of
str.getBytes("UTF8");
but in writeINodeUnderConstruction, the bytes user the method of Class UTF8 to
get the bytes.
I make a test use our messy code file name , found the the two bytes arrsy are
not equal. so I both use the class UTF8,then the problem desappare.
I think this is a bug of HDFS or UTF8.
Environment: SUSE LINUX
Target Version/s: 0.20.1
Affects Version/s: 0.20.1
Tags: messy code,SecondaryNameNode,FSImage
Remaining Estimate: 12h
Original Estimate: 12h
Summary: when save FSImage ，HDFS（ or SecondaryNameNode or
FSImage）can't handle some file whose file name has some special messy code(乱码)
(was: when a file)
> when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some
> file whose file name has some special messy code(乱码)
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-3307
> URL: https://issues.apache.org/jira/browse/HDFS-3307
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.20.1
> Environment: SUSE LINUX
> Reporter: yixiaohua
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> this the log information of the exception from the SecondaryNameNode:
> 2012-03-28 00:48:42,553 ERROR
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
> java.io.IOException: Found lease for
> non-existent file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/????@???????????????
> ??????????tor.qzone.qq.com/keypart-00174
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
> at java.lang.Thread.run(Thread.java:619)
> this is the log information about the file from namenode:
> 2012-03-28 00:32:26,528 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34 cmd=create
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @? tor.qzone.qq.com/keypart-00174 dst=null
> perm=boss:boss:rw-r--r--
> 2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @? tor.qzone.qq.com/keypart-00174.
> blk_2751836614265659170_184668759
> 2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @? tor.qzone.qq.com/keypart-00174 is closed by
> DFSClient_attempt_201203271849_0016_r_000174_0
> 2012-03-28 00:37:50,315 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34 cmd=rename
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @? tor.qzone.qq.com/keypart-00174
> dst=/user/boss/pgv/fission/task16/split/ @?
> tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
> after check the code that save FSImage,I found there are a problem that maybe
> a bug of HDFS Code,I past below:
> -------------this is the saveFSImage method in FSImage.java, I make some
> mark at the problem code------------
> /**
> * Save the contents of the FS image to the file.
> */
> void saveFSImage(File newFile) throws IOException {
> FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
> FSDirectory fsDir = fsNamesys.dir;
> long startTime = FSNamesystem.now();
> //
> // Write out data
> //
> DataOutputStream out = new DataOutputStream(
> new BufferedOutputStream(
> new
> FileOutputStream(newFile)));
> try {
> .........
>
> // save the rest of the nodes
> saveImage(strbuf, 0, fsDir.rootDir, out);------------------problem
> fsNamesys.saveFilesUnderConstruction(out);------------------problem
> detail is below
> strbuf = null;
> } finally {
> out.close();
> }
> LOG.info("Image file of size " + newFile.length() + " saved in "
> + (FSNamesystem.now() - startTime)/1000 + " seconds.");
> }
> /**
> * Save file tree image starting from the given root.
> * This is a recursive procedure, which first saves all children of
> * a current directory and then moves inside the sub-directories.
> */
> private static void saveImage(ByteBuffer parentPrefix,
> int prefixLength,
> INodeDirectory current,
> DataOutputStream out) throws IOException {
> int newPrefixLength = prefixLength;
> if (current.getChildrenRaw() == null)
> return;
> for(INode child : current.getChildren()) {
> // print all children first
> parentPrefix.position(prefixLength);
>
> parentPrefix.put(PATH_SEPARATOR).put(child.getLocalNameBytes());------------------problem
> saveINode2Image(parentPrefix, child, out);
> }
> ..........
> }
> // Helper function that writes an INodeUnderConstruction
> // into the input stream
> //
> static void writeINodeUnderConstruction(DataOutputStream out,
> INodeFileUnderConstruction cons,
> String path)
> throws IOException {
> writeString(path, out);------------------problem
> ..........
> }
>
> static private final UTF8 U_STR = new UTF8();
> static void writeString(String str, DataOutputStream out) throws
> IOException {
> U_STR.set(str);
> U_STR.write(out);------------------problem
> }
> /**
> * Converts a string to a byte array using UTF8 encoding.
> */
> static byte[] string2Bytes(String str) {
> try {
> return str.getBytes("UTF8");------------------problem
> } catch(UnsupportedEncodingException e) {
> assert false : "UTF8 encoding is not supported ";
> }
> return null;
> }
> ------------------------------------------below is the
> explain------------------------
> in saveImage method: child.getLocalNameBytes(),the bytes use the method of
> str.getBytes("UTF8");
> but in writeINodeUnderConstruction, the bytes user the method of Class UTF8
> to get the bytes.
> I make a test use our messy code file name , found the the two bytes arrsy
> are not equal. so I both use the class UTF8,then the problem desappare.
> I think this is a bug of HDFS or UTF8.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
