Re: reading an empty directory after reboot is very slow

2015-04-27 Thread Bob Proulx
Vincent Lefevre wrote: In short, ext3 is bad for renames, but if Bob Proulx's suggestion were implemented (without a workaround such that caching the whole directory after it is opened[*]), it would be much worse as arbitrary entries would be missed in a readdir sequence. To be clear I didn't

Re: reading an empty directory after reboot is very slow

2015-04-27 Thread Vincent Lefevre
On 2015-04-26 13:26:41 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-24 22:01:37 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): So, I would say that this is a bug. POSIX says in

Re: reading an empty directory after reboot is very slow

2015-04-27 Thread Vincent Lefevre
On 2015-04-27 11:04:51 -0500, David Wright wrote: Exactly. Saying But this matters in a POSIX-compliant filesystem is very different from But with the current solution. To me, a current solution is an concrete implementation, not a POSIX specification; the latter is a target. I really meant

Re: reading an empty directory after reboot is very slow

2015-04-27 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-26 13:26:41 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-24 22:01:37 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): So, I would say that this is a bug. POSIX

Re: reading an empty directory after reboot is very slow

2015-04-27 Thread Bob Proulx
David Wright wrote: Quoting Kushal Kumaran: Bob Proulx writes: If a directory became full it was easy to extend it by writing the array longer. But if an early entry in the array was deleted the system would zero it out rather than move each and every entry in the file system down

Re: reading an empty directory after reboot is very slow

2015-04-26 Thread Vincent Lefevre
On 2015-04-24 22:01:37 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): So, I would say that this is a bug. POSIX says in http://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html It may well be. But I'm just presenting facts about the filesystems that

Re: reading an empty directory after reboot is very slow

2015-04-26 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-24 22:01:37 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): So, I would say that this is a bug. POSIX says in http://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html It may well be. But

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread Vincent Lefevre
On 2015-04-23 11:22:11 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-23 11:31:27 +0200, Nicolas George wrote: Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : David's test shows that the renamed file is missed. No, it shows that

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread Vincent Lefevre
On 2015-04-24 09:35:29 +0200, Vincent Lefevre wrote: Anyway remember that the main slowness I have is when I retag messages. I could improve my script to cache the message-id - filename mapping, which would make it much faster. But what I like to understand is why my current script is so much

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread Vincent Lefevre
On 2015-04-24 19:41:03 -0500, David Wright wrote: Well, that's what I thought, because of the cacheing. But Nicolas asked me to try using thousands of files, and so here we are, ie, your new test: ~ $ for j in `seq 1` ; do mkdir /tmp/testdir/file$j ; done ~ $ ./a.out /tmp/testdir/

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-23 11:22:11 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-23 11:31:27 +0200, Nicolas George wrote: Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : David's test shows that the

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-24 19:41:03 -0500, David Wright wrote: Well, that's what I thought, because of the cacheing. But Nicolas asked me to try using thousands of files, and so here we are, ie, your new test: ~ $ for j in `seq 1` ; do mkdir

Re: reading an empty directory after reboot is very slow

2015-04-24 Thread Vincent Lefevre
On 2015-04-23 10:10:54 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-22 23:28:46 -0500, David Wright wrote: No, I wasn't expecting mutt to use mairix. But I thought you might be using it. Otherwise, why do you index them? I use mairix when I need

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-23 11:31:27 +0200, Nicolas George wrote: Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : David's test shows that the renamed file is missed. No, it shows that the renamed file is NOT missed: he renamed the entry for

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-22 23:28:46 -0500, David Wright wrote: No, I wasn't expecting mutt to use mairix. But I thought you might be using it. Otherwise, why do you index them? I use mairix when I need a body search first, otherwise such a search would be

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread David Wright
Quoting Nicolas George (geo...@nsup.org): Le tridi 3 floréal, an CCXXIII, David Wright a écrit : OK. Here's a demonstration of a file going AWOL by moving *up* the directory listing. Because of read-ahead, readdir still sees the old name and the stat() fails. What are you trying to prove

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-22 23:06:47 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-21 11:05:58 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-20 13:04:41 -0500, David Wright

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Nicolas George
Le tridi 3 floréal, an CCXXIII, David Wright a écrit : OK. Here's a demonstration of a file going AWOL by moving *up* the directory listing. Because of read-ahead, readdir still sees the old name and the stat() fails. What are you trying to prove with that test? You would get the same failure

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-22 23:28:46 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-21 12:47:14 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): This mailbox is constantly open in a Mutt running in screen (in read-only mode). I often

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-23 09:13:02 +0200, Nicolas George wrote: Le tridi 3 floréal, an CCXXIII, David Wright a écrit : OK. Here's a demonstration of a file going AWOL by moving *up* the directory listing. Because of read-ahead, readdir still sees the old name and the stat() fails. What are you

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Nicolas George
Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : David's test was to show what could happen in practice after some simple change, such as removing objects from the directory, not to take care of all possible race conditions Simple changes are the bricks for race conditions.

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-22 23:30:49 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-21 10:35:12 -0600, Bob Proulx wrote: Vincent Lefevre wrote: So, there's still something I don't understand: after dropping the caches, why is Mutt fast to read the mailbox

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-22 23:06:47 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-21 11:05:58 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-20 13:04:41 -0500, David Wright wrote: Quoting Vincent Lefevre

Re: reading an empty directory after reboot is very slow

2015-04-23 Thread Vincent Lefevre
On 2015-04-23 11:31:27 +0200, Nicolas George wrote: Le quartidi 4 floréal, an CCXXIII, Vincent Lefevre a écrit : David's test shows that the renamed file is missed. No, it shows that the renamed file is NOT missed: he renamed the entry for inode 497003 from file2 into a long name, and that

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-21 11:05:58 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-20 13:04:41 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): But with the current solution (no automatic

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-21 10:35:12 -0600, Bob Proulx wrote: Vincent Lefevre wrote: So, there's still something I don't understand: after dropping the caches, why is Mutt fast to read the mailbox (about 1 minute), but not my tool (about 30 minutes)?

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-21 12:47:14 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): This mailbox is constantly open in a Mutt running in screen (in read-only mode). I often read it, and I modify it from time to time, either by

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread Vincent Lefevre
On 2015-04-21 10:35:12 -0600, Bob Proulx wrote: Vincent Lefevre wrote: So, there's still something I don't understand: after dropping the caches, why is Mutt fast to read the mailbox (about 1 minute), but not my tool (about 30 minutes)? Are you using mutt's header_cache feature? It

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread Vincent Lefevre
On 2015-04-21 12:47:14 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): This mailbox is constantly open in a Mutt running in screen (in read-only mode). I often read it, and I modify it from time to time, either by adding new messages in the usual way, or by

Re: reading an empty directory after reboot is very slow

2015-04-22 Thread Vincent Lefevre
On 2015-04-21 11:05:58 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-20 13:04:41 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): But with the current solution (no automatic moving of an entry), you can't miss an entry

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-20 15:59:22 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): Possibly, but individual modifications would take much more time than with Maildir (such modifications, consisting in retagging, occur from time

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread Vincent Lefevre
On 2015-04-20 13:04:41 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): But with the current solution (no automatic moving of an entry), you can't miss an entry that hasn't been removed. $ mkdir /tmp/testdir ~ $ mkdir /tmp/testdir/file1 ~ $ mkdir

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread Vincent Lefevre
On 2015-04-20 18:03:01 -0600, Bob Proulx wrote: Vincent Lefevre wrote: With an uncompressed mbox file, using the Content-Length, it could be faster, but there's still the problem with individual changes. Have you run across Jamie Zawinski's rant against Content-Length?

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread Vincent Lefevre
On 2015-04-20 15:59:22 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): Possibly, but individual modifications would take much more time than with Maildir (such modifications, consisting in retagging, occur from time to time). I take it these are real emails being

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-20 13:04:41 -0500, David Wright wrote: Quoting Vincent Lefevre (vinc...@vinc17.net): But with the current solution (no automatic moving of an entry), you can't miss an entry that hasn't been removed. [...] ...so if you happen to

Re: reading an empty directory after reboot is very slow

2015-04-21 Thread Bob Proulx
Vincent Lefevre wrote: So, there's still something I don't understand: after dropping the caches, why is Mutt fast to read the mailbox (about 1 minute), but not my tool (about 30 minutes)? Are you using mutt's header_cache feature? It keeps a cache of mail files off to the side. It really

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread Bob Proulx
Vincent Lefevre wrote: With an uncompressed mbox file, using the Content-Length, it could be faster, but there's still the problem with individual changes. Have you run across Jamie Zawinski's rant against Content-Length? http://www.jwz.org/doc/content-length.html But let's say that all

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-15 13:41:20 -0600, Bob Proulx wrote: Vincent Lefevre wrote: I also notice slowness with a large maildir directory: drwx-- 2 vlefevre vlefevre 8409088 2015-03-24 14:04:33 Mail/oldarc/cur/ In this one, the files are

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-15 14:13:43 -0500, David Wright wrote: Quoting Kushal Kumaran (kus...@locationd.net): Moving entries around breaks ongoing readdir operations. If a readdir has gone past the file being removed, and you moved the last entry

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread Vincent Lefevre
On 2015-04-15 13:41:20 -0600, Bob Proulx wrote: Vincent Lefevre wrote: I also notice slowness with a large maildir directory: drwx-- 2 vlefevre vlefevre 8409088 2015-03-24 14:04:33 Mail/oldarc/cur/ In this one, the files are real (145400 files), but I have a Perl script that

Re: reading an empty directory after reboot is very slow

2015-04-20 Thread Vincent Lefevre
On 2015-04-15 14:13:43 -0500, David Wright wrote: Quoting Kushal Kumaran (kus...@locationd.net): Moving entries around breaks ongoing readdir operations. If a readdir has gone past the file being removed, and you moved the last entry there, the entry being moved would be missed, despite

Re: reading an empty directory after reboot is very slow

2015-04-16 Thread Vincent Lefevre
On 2015-04-15 14:19:40 -0500, David Wright wrote: Good to see people testing their tools. Actually it is not a test of a tool, but a test to find for various double-precision functions, the arguments that are the hardest to round, in order to solve the Table Maker's Dilemma. Such values can also

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 23:18, Vincent Lefevre wrote: On 2015-04-14 13:26:16 -0300, Henrique de Moraes Holschuh wrote: Yeah, that's a bad habit to have as it slows down way too many utilities (lots of stuff benefit for extremely lightweight ultra-fast tmpfs in /tmp and $TMPDIR, from sort

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Vincent Lefevre
On 2015-04-15 09:32:24 -0300, Henrique de Moraes Holschuh wrote: On Tue, Apr 14, 2015, at 23:18, Vincent Lefevre wrote: Can't disk caching be as fast as tmpfs (or almost)? Never subestimate the needs of a persistent filesystem to care for metadata and data safety, unless it has an eatmydata

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Henrique de Moraes Holschuh
On Wed, Apr 15, 2015, at 16:57, Bob Proulx wrote: Henrique de Moraes Holschuh wrote: 2. unmount filesystem and run e2fsck -D on it every so often. Every so often? That phrase roused my interest. Wouldn't exactly once be enough? Yes, as long as nobody ever creates a massive number of

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Bob Proulx
Henrique de Moraes Holschuh wrote: Bob Proulx wrote: Henrique de Moraes Holschuh wrote: 2. unmount filesystem and run e2fsck -D on it every so often. Every so often? That phrase roused my interest. Wouldn't exactly once be enough? Yes, as long as nobody ever creates a massive

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread David Wright
Quoting Kushal Kumaran (kus...@locationd.net): Bob Proulx b...@proulx.com writes: Petter Adsen wrote: Can someone please enlighten me as to why the entry for this directory is so large, even though it is empty? Since it's apparently obvious to everyone else, I would very much like to know

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Bob Proulx
Cam Hutchison wrote: I don't think dir_index has anything to do with it. An index speeds up lookups. You are not doing lookups; you are traversing the entire data structure. I think you might be right about Vincent's problem. It seems there is a large amount of uncached data that needs to be

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Bob Proulx
Henrique de Moraes Holschuh wrote: 2. unmount filesystem and run e2fsck -D on it every so often. Every so often? That phrase roused my interest. Wouldn't exactly once be enough? I think there are two issues under discussion. One is if dir_index has been applied. For that once would be

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): On 2015-04-13 15:50:40 -0500, David Wright wrote: That's staggering. My /var/lib/dpkg/info has ~8900 files and occupies 462848 bytes. So that would be over ½million files in your case. Does eftests stand for excessive files tests! It means

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Bob Proulx
Vincent Lefevre wrote: Filesystem created: Mon Jan 4 16:26:16 2010 My machine is old, but I've never changed anything concerning the file system. 2010 isn't that old. Just a baby! :-) After a quick look on my network I located these. desolation: Filesystem created: Tue Feb

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Kushal Kumaran
Bob Proulx b...@proulx.com writes: Petter Adsen wrote: Can someone please enlighten me as to why the entry for this directory is so large, even though it is empty? Since it's apparently obvious to everyone else, I would very much like to know :) snipped If a directory became full it was

Re: reading an empty directory after reboot is very slow

2015-04-15 Thread Vincent Lefevre
On 2015-04-15 12:29:49 +0530, Kushal Kumaran wrote: Moving entries around breaks ongoing readdir operations. If a readdir has gone past the file being removed, and you moved the last entry there, the entry being moved would be missed, despite *it* not being the entry added or removed.

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Petter Adsen
On Mon, 13 Apr 2015 15:41:03 +0200 Vincent Lefevre vinc...@vinc17.net wrote: On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: 2015-04-13 14:39 GMT+02:00 Vincent Lefevre vinc...@vinc17.net: The problem is that this operation is (always?) very slow: something like 100 seconds (1 minute and

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Reco
Hi. On Tue, Apr 14, 2015 at 08:58:20AM +0200, Petter Adsen wrote: On Mon, 13 Apr 2015 15:41:03 +0200 Vincent Lefevre vinc...@vinc17.net wrote: On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: 2015-04-13 14:39 GMT+02:00 Vincent Lefevre vinc...@vinc17.net: The problem is that this

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Petter Adsen
On Tue, 14 Apr 2015 10:12:28 +0300 Reco recovery...@gmail.com wrote: Hi. On Tue, Apr 14, 2015 at 08:58:20AM +0200, Petter Adsen wrote: On Mon, 13 Apr 2015 15:41:03 +0200 Vincent Lefevre vinc...@vinc17.net wrote: On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: 2015-04-13 14:39

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Reco
Hi. On Tue, Apr 14, 2015 at 09:22:15AM +0200, Petter Adsen wrote: On Tue, 14 Apr 2015 10:12:28 +0300 Reco recovery...@gmail.com wrote: Hi. On Tue, Apr 14, 2015 at 08:58:20AM +0200, Petter Adsen wrote: On Mon, 13 Apr 2015 15:41:03 +0200 Vincent Lefevre vinc...@vinc17.net wrote:

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 04:22, Petter Adsen wrote: But if you create new files in that directory after deleting them, I expect the inodes get reallocated? They do, but a directory does not store inodes. http://en.wikipedia.org/wiki/Inode For ext4:

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Mon, Apr 13, 2015, at 19:28, Bob Proulx wrote: Without dir_index an ext filesystem with large directories is slow due to the linear nature of directories. But with dir_index it should be using a B-tree data structure and should be much faster. This can be turned off for a migration if it

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Gene Heskett
On Tuesday 14 April 2015 02:58:20 Petter Adsen wrote: On Mon, 13 Apr 2015 15:41:03 +0200 Vincent Lefevre vinc...@vinc17.net wrote: On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: 2015-04-13 14:39 GMT+02:00 Vincent Lefevre vinc...@vinc17.net: The problem is that this operation is

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 08:47:15 -0300, Henrique de Moraes Holschuh wrote: Also, some of the Unix best-practices do address this kind of filesystem issue. One such best-practice is that you don't remove just the files in ephemeral directories: you either use an ephemeral filesystem (tmpfs) in the first

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Bob Proulx
Petter Adsen wrote: Can someone please enlighten me as to why the entry for this directory is so large, even though it is empty? Since it's apparently obvious to everyone else, I would very much like to know :) In the old days directories were simply an array of fixed sized integers and fixed

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 13:11, Bob Proulx wrote: Oh! I was unaware of the e2fsck -D option. Good deal! I didn't recall it existed either, or I'd have mentioned it much earlier in the thread... but I noticed it in the manpage while doing some research due to my participation on this thread.

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Doug
On 04/14/2015 05:15 AM, Reco wrote: Hi. On Tue, Apr 14, 2015 at 09:22:15AM +0200, Petter Adsen wrote: /snip/ 2015-04-13 14:39 GMT+02:00 Vincent Lefevre vinc...@vinc17.net: The problem is that this operation is (always?) very slow: something like 100 seconds (1 minute and 40 seconds). It

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 11:04:06 -0300, Henrique de Moraes Holschuh wrote: On Tue, Apr 14, 2015, at 10:06, Vincent Lefevre wrote: Without dir_index an ext filesystem with large directories is slow due to the linear nature of directories. But with dir_index it should be using a B-tree data

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Bob Proulx
Henrique de Moraes Holschuh wrote: Bob Proulx wrote: tune2fs -O dir_index /dev/sda5 But existing directories are not converted. Only new directories are Do a forced e2fsck run on the filesystem, and it should add the indexes to all directories that lack it. It can also change the

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 13:26:16 -0300, Henrique de Moraes Holschuh wrote: Yeah, that's a bad habit to have as it slows down way too many utilities (lots of stuff benefit for extremely lightweight ultra-fast tmpfs in /tmp and $TMPDIR, from sort to gcc without -pipe), but it is indeed widespread. Can't

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 23:01:19 -, Cam Hutchison wrote: I don't think dir_index has anything to do with it. An index speeds up lookups. You are not doing lookups; you are traversing the entire data structure. A B-tree data structure can take longer to traverse than a contiguous array data structure

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 10:55:12 -0600, Bob Proulx wrote: Other file systems such as xfs designed for large files and large numbers of files DO shrink when files are removed. That is one of the reasons why xfs is recommended for industrial strength use. It was designed to handle those kinds of

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread The Wanderer
On 04/14/2015 at 12:55 PM, Bob Proulx wrote: Petter Adsen wrote: Can someone please enlighten me as to why the entry for this directory is so large, even though it is empty? Since it's apparently obvious to everyone else, I would very much like to know :) In the old days directories

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Reco
On Tue, Apr 14, 2015 at 03:15:02PM +0200, Vincent Lefevre wrote: On 2015-04-14 12:15:03 +0300, Reco wrote: On Tue, Apr 14, 2015 at 09:22:15AM +0200, Petter Adsen wrote: On Tue, 14 Apr 2015 10:12:28 +0300 Reco recovery...@gmail.com wrote: On Tue, Apr 14, 2015 at 08:58:20AM +0200,

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 10:06, Vincent Lefevre wrote: Without dir_index an ext filesystem with large directories is slow due to the linear nature of directories. But with dir_index it should be using a B-tree data structure and should be much faster. So, why is it slow? What kernel?

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Henrique de Moraes Holschuh
On Tue, Apr 14, 2015, at 14:52, Doug wrote: Removing files from the directory does not change directory's inode size. If using ext4, at least. What is the best way of solving this? Since the directory is empty, can you just do an rmdir dirname ? Now if there are one or two files in it,

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-13 15:50:40 -0500, David Wright wrote: That's staggering. My /var/lib/dpkg/info has ~8900 files and occupies 462848 bytes. So that would be over ½million files in your case. Does eftests stand for excessive files tests! It means elementary function tests, but what this doesn't say

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-13 16:28:27 -0600, Bob Proulx wrote: Vincent Lefevre wrote: Loïc Grenié wrote: Vincent Lefevre wrote: The problem is that this operation is (always?) very slow: something like 100 seconds (1 minute and 40 seconds). It has been reproducible for several months. The logs

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Vincent Lefevre
On 2015-04-14 12:15:03 +0300, Reco wrote: On Tue, Apr 14, 2015 at 09:22:15AM +0200, Petter Adsen wrote: On Tue, 14 Apr 2015 10:12:28 +0300 Reco recovery...@gmail.com wrote: On Tue, Apr 14, 2015 at 08:58:20AM +0200, Petter Adsen wrote: Removing files from the directory does not change

Re: reading an empty directory after reboot is very slow

2015-04-14 Thread Cam Hutchison
Vincent Lefevre vinc...@vinc17.net writes: On 2015-04-13 16:28:27 -0600, Bob Proulx wrote: Without dir_index an ext filesystem with large directories is slow due to the linear nature of directories. But with dir_index it should be using a B-tree data structure and should be much faster. So,

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Henrique de Moraes Holschuh
On Mon, Apr 13, 2015, at 13:48, Gene Heskett wrote: I wonder if our sort utilities also do that file shrink? No. It is something the VFS (generic filesystem layer inside the kernel) or filesystem itself would have to do, due to atomicity requirements. Unrelated of course, but of all the

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread David Wright
Quoting Vincent Lefevre (vinc...@vinc17.net): Thanks! I didn't know that (I thought that the file system would automatically optimize directories when files are removed, so I've never looked closely at their size). Indeed: ypig:~/eftests ls -ld tmp drwxr-xr-x 2 vlefevre vlefevre 29655040

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Bob Proulx
Vincent Lefevre wrote: Loïc Grenié wrote: Vincent Lefevre wrote: The problem is that this operation is (always?) very slow: something like 100 seconds (1 minute and 40 seconds). It has been reproducible for several months. The logs show nothing during this operation. Any idea?

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Gene Heskett
On Monday 13 April 2015 16:08:44 Henrique de Moraes Holschuh wrote: On Mon, Apr 13, 2015, at 13:48, Gene Heskett wrote: I wonder if our sort utilities also do that file shrink? No. It is something the VFS (generic filesystem layer inside the kernel) or filesystem itself would have to do,

reading an empty directory after reboot is very slow

2015-04-13 Thread Vincent Lefevre
I have some regular directory ~/eftests/tmp, and after a reboot, I often check that it is empty (as I remove its contents before the reboot): from the ~/eftests directory, I do ypig:~/eftests ll tmp which corresponds to ls -bF --color -l tmp after alias expansion. The problem is that this

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Loïc Grenié
2015-04-13 14:39 GMT+02:00 Vincent Lefevre vinc...@vinc17.net: I have some regular directory ~/eftests/tmp, and after a reboot, I often check that it is empty (as I remove its contents before the reboot): from the ~/eftests directory, I do ypig:~/eftests ll tmp which corresponds to ls

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Vincent Lefevre
On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: 2015-04-13 14:39 GMT+02:00 Vincent Lefevre vinc...@vinc17.net: The problem is that this operation is (always?) very slow: something like 100 seconds (1 minute and 40 seconds). It has been reproducible for several months. The logs show nothing

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Darac Marjal
On Mon, Apr 13, 2015 at 02:39:40PM +0200, Vincent Lefevre wrote: I have some regular directory ~/eftests/tmp, and after a reboot, I often check that it is empty (as I remove its contents before the reboot): from the ~/eftests directory, I do ypig:~/eftests ll tmp which corresponds to

Re: reading an empty directory after reboot is very slow

2015-04-13 Thread Gene Heskett
On Monday 13 April 2015 09:41:03 Vincent Lefevre wrote: On 2015-04-13 14:45:25 +0200, Loïc Grenié wrote: 2015-04-13 14:39 GMT+02:00 Vincent Lefevre vinc...@vinc17.net: The problem is that this operation is (always?) very slow: something like 100 seconds (1 minute and 40 seconds). It has