On Tue, Jan 19, 2010 at 9:25 PM, Matthew Ahrens <matthew.ahr...@sun.com> wrote:
> Michael Schuster wrote:
>>
>> Mike Gerdts wrote:
>>>
>>> On Tue, Jan 5, 2010 at 4:34 AM, Mikko Lammi <mikko.la...@lmmz.net> wrote:
>>>>
>>>> Hello,
>>>>
>>>> As a result of one badly designed application running loose for some
>>>> time,
>>>> we now seem to have over 60 million files in one directory. Good thing
>>>> about ZFS is that it allows it without any issues. Unfortunatelly now
>>>> that
>>>> we need to get rid of them (because they eat 80% of disk space) it seems
>>>> to be quite challenging.
>>>>
>>>> Traditional approaches like "find ./ -exec rm {} \;" seem to take
>>>> forever
>>>> - after running several days, the directory size still says the same.
>>>> The
>>>> only way how I've been able to remove something has been by giving "rm
>>>> -rf" to problematic directory from parent level. Running this command
>>>> shows directory size decreasing by 10,000 files/hour, but this would
>>>> still
>>>> mean close to ten months (over 250 days) to delete everything!
>>>>
>>>> I also tried to use "unlink" command to directory as a root, as a user
>>>> who
>>>> created the directory, by changing directory's owner to root and so
>>>> forth,
>>>> but all attempts gave "Not owner" error.
>>>>
>>>> Any commands like "ls -f" or "find" will run for hours (or days) without
>>>> actually listing anything from the directory, so I'm beginning to
>>>> suspect
>>>> that maybe the directory's data structure is somewhat damaged. Is there
>>>> some diagnostics that I can run with e.g "zdb" to investigate and
>>>> hopefully fix for a single directory within zfs dataset?
>>>
>>> In situations like this, ls will be exceptionally slow partially
>>> because it will sort the output.
>>
>> that's what '-f' was supposed to avoid, I'd guess.
>
> Yes, but unfortunately, the typical reason ls is slow with huge directories
> is that it requires a huge amount of memory. Even when not sorting (with
> -f), it still allocates a huge amount of memory for each entry listed, and
> buffers the output until the directory is entirely read. So typically -f
> doesn't help performance much. Improving this would be a great small
> project for an OpenSolaris contributor! I filed a couple of bugs for this
> several years ago, I can dig them up if anyone is interested.
>
> --matt
After a few days of deliberation, I've decided to start working on
this (in addition to adding the 256 color ls support Danek was
interested in as well as addressing a number of other bugs).
I suspect it's going to require a significant overhaul of the existing
code to get it where it can behave better with large directories
(though the current code could probably use the cleanup).
If anyone's interested in testing once I've got it to a point worth
testing (will probably be a few weeks, depending on how much time I
can commit to it)... let me know...
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
