Thanks for the digging. This concurs with my suspicion in the beginning. I am copying Jesse who wrote the code. He should have more insight on this.
After his confirmation, you can log a JIRA. Cheers On Sun, Dec 30, 2012 at 10:59 AM, Jean-Marc Spaggiari < [email protected]> wrote: > So. Looking deeper I found few things. > > First, why checkAndDeleteDirectory is not "simply" calling > FSUtils.delete (fs, toCheck, true)? I guess it's doing the same thing? > > Also, FSUtils.listStatus(fs, toCheck, null); will return null if there > is no status. Not just an empty array. And it's returning null, we > will exit without calling the delete methode. > > I tried to manually create a file on one of those directories. The > exception disapears for 300 seconds because of the TTL for the newly > created file. After 300 seconds, the file I pushed AND the directory > got removed. So the issue is really with empty directories. > > I will take a look at what is in the trunk and in 0.94.4 to see if > it's the same issue. But I think we can simple change all this code by > a call to FSUtils.delete. > > I can open a JIRA and submit a patch for that. Just let me know. > > JM > > 2012/12/30, Jean-Marc Spaggiari <[email protected]>: > > Regargind the logcleaner settings, I have not changed anything. It's > > what came with the initial install. So I don't have anything setup for > > this plugin in my configuration files. > > > > For the files on the FS, here is what I have: > > hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls > > /hbase/.archive/entry_duplicate > > Found 30 items > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/00c185bc44b6dcf85a90b83bdda4ec2e > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/0ddf0d1802c6afd97d032fd09ea9e37d > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/18cf7c5c946ddf33e49b227feedfb688 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/2353f10e79dacc5cf201be6a1eb63607 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > > /hbase/.archive/entry_duplicate/243f4007cf05415062010a5650598bff > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > > /hbase/.archive/entry_duplicate/287682333698e36cea1670f5479fbf18 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/3742da9bd798342e638e1ce341f27537 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > > /hbase/.archive/entry_duplicate/435c9c08bc08ed7248a013b6ffaa163b > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/45346b4b4248d77d45e031ea71a1fb63 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/4afe48fe6d8defe569f8632dd2514b07 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/68a4e364fe791a0d1f47febbb41e8112 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/7673d718962535c7b54cef51830f22a5 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > > /hbase/.archive/entry_duplicate/7df6845ae9d052f4eae4a01e39313d61 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/8c5a263167d1b09f645af8efb4545554 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/8c98d9c635ba30d467d127a2ec1c69f8 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/8dfa96393e18ecca826fd9200e6bf68b > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/8e8f532e91a7197cd53b7626130be698 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/8eca1a325fe442a8546e43ac2f00cfef > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/9ad4c0551b90ea7717d7e3aaec76dc26 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/a135ccbc6f61ce544dbd537dc12489e9 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/a3d0332a6d51a8b15b99d1caca3f355a > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/bd2b8c942af27e541e20e430d506d2c0 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/c10c3a66948bde75fc41349108d86cf9 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > > /hbase/.archive/entry_duplicate/cbf2f178691bfca8a7e9825115629b8e > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/d14a2546eaceede73b282e444ad1bb40 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:38 > > /hbase/.archive/entry_duplicate/d570a21a39e04ba2ec896bbe7166423c > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/e943bda56acd6beb35bdd56f0560f87f > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/ef5692ba83aba48d9e7a6b9c2cd0661e > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/fd85dd319c289959a790faed32ef1530 > > drwxr-xr-x - hbase supergroup 0 2012-12-10 14:39 > > /hbase/.archive/entry_duplicate/ffcdf6554accda1800e74838b67d3004 > > hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls > > /hbase/.archive/entry_duplicate/00c185bc44b6dcf85a90b83bdda4ec2e > > hadoop@node3:~/hadoop-1.0.3$ > > > > I have not lookeqd into ALL the subdirectories, but the 10 first are > > empty. > > > > I see that there is some traces on checkAndDeleteDirectory... I will > > try to activate that and see if there is more details. > > > > > > JM > > > > 2012/12/30, Ted Yu <[email protected]>: > >> The exception came from this line: > >> if (file.isDir()) checkAndDeleteDirectory(file.getPath()); > >> Looking at checkAndDeleteDirectory(), it recursively deletes files and > >> directories under the specified path. > >> > >> Does /hbase/.archive/entry_duplicate only contain empty directories > >> underneath it ? > >> > >> You didn't modify the logcleaner plugin setting, right ? > >> <property> > >> <name>hbase.master.logcleaner.plugins</name> > >> > >> > <value>org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner</value> > >> > >> </property> > >> > >> Cheers > >> > >> On Sun, Dec 30, 2012 at 9:53 AM, Jean-Marc Spaggiari < > >> [email protected]> wrote: > >> > >>> I was going to move to 0.94.4 today ;) And yes I'm using 0.94.3. I > >>> might wait a bit in case some testing is required with my version. > >>> > >>> Is this what you are looking for? http://pastebin.com/N8Q0FMba > >>> > >>> I will keep the files for now since it seems it's not causing any > >>> major issue. That will allow some more testing if required. > >>> > >>> JM > >>> > >>> > >>> 2012/12/30, Ted Yu <[email protected]>: > >>> > Looks like you're using 0.94.3 > >>> > > >>> > The archiver is backport of: > >>> > HBASE-5547, Don't delete HFiles in backup mode > >>> > > >>> > Can you provide more the log where the IOE was reported using > pastebin > >>> > ? > >>> > > >>> > Thanks > >>> > > >>> > On Sun, Dec 30, 2012 at 9:08 AM, Jean-Marc Spaggiari < > >>> > [email protected]> wrote: > >>> > > >>> >> Hi, > >>> >> > >>> >> I have a "IOException" /hbase/.archive/table_name is non empty > >>> >> exception every minute on my logs. > >>> >> > >>> >> There is 30 directories under this directory. the main directory is > >>> >> from yesterday, but all sub directories are from December 10th, all > >>> >> the same time. > >>> >> > >>> >> What does this .archive directory is used for, and what should I do? > >>> >> > >>> >> Thanks, > >>> >> > >>> >> JM > >>> >> > >>> > > >>> > >> > > >
