Hi All, We're using hadoop-2.0.0-cdh4.3.2. Recently we relocated our namenode by shutting it down, changing its ip address and restarting it. During that process, the hdfs clients were still trying to write to hdfs.
Some clients failed due to an java.util.ConcurrentModificationException throw out from the LeaseRenewer. Looks like it is the problem addressed by the https://issues.apache.org/jira/browse/HDFS-5028. The other clients didn't complain anything. However some of the files they wrote at hdfs has 0 size although they did write something into the DFSOutputstream. To make it clear, when the client writes file, it firstly writes to a temp file at hdfs. Once it's completed, the temp file is rename under the same directory. Later some time, the file will be renamed again to be put to a different directory. We found after the namenode relocation, these clients keeping generating empty files at hdfs sometimes but not all the time for a few days. We finally restarted all the clients. No new empty files anymore. But the strange thing is, gradually some files which were empty before became none-empty. Here's my question. There's a bug in our program: we didn't handle the exception when we close the FSDataOutputstream. So it's reasonable to see empty files. 1. But why did our clients write empty file after the relocation was done. There should be no connection issue to hadoop since they can write some files right. I don't understand why sometimes they cannot. 2. After restarting clients, why do some empty files became non-empty? 3. Why didn't all the clients get the ConcurrentModificationException from LeaseRenewer? And for those clients that don't have such exceptions, what happened to their lease, outputstream? Please let me know if you need more information. Thanks a lot! Lei
