Ignore the last comment, the bug is only in this newer version of the
patch... Here're some additional comments and measurements from Eero
about the new patch.
-----
Here are patches for reducing the huge dosfsck memory usage to something
slightly more reasonable. Except for the dosfsck.h bitfield change,
they should be easy/straightforward to verify/review. As far as I could
see, the bitfield change is also safe.
In common.c:
* By using special block allocator, the 8 byte qalloc() and 8 byte
C-library malloc() overhead can be avoided. According to Valgrind
Massif, this reduces dosfsck memory usage from over 120MB[1] to
slightly over 80MB with a 8GB FAT file system full of files
* Changed common.h function prototypes to use size_t like malloc does
[1] full 8GB file system, before:
n time(i) total(B) useful-heap(B) extra-heap(B)
------------------------------------------------------------------
79 22,339,620,705 121,351,024 91,858,713 29,492,311
80 22,368,083,813 121,487,936 91,958,709 29,529,227
75.69% (91,958,709B) (heap allocation functions) malloc/new/new[],
--alloc-fns, etc.
->75.69% (91,958,002B) 0x804CC40: alloc (common.c:52)
| ->64.91% (78,855,324B) 0x804CC8E: qalloc (common.c:65)
| | ->50.09% (60,851,840B) 0x804C1B2: add_file (check.c:787)
| | | ->50.09% (60,851,784B) 0x804C9EA: subdirs (check.c:824)
| | | | ->50.08% (60,840,584B) 0x804CA5F: subdirs (check.c:833)
| | | | | ->49.65% (60,316,424B) 0x804CA5F: subdirs (check.c:833)
| | | | | | ->35.85% (43,554,504B) 0x804CA5F: subdirs (check.c:833)
| | | | | | | ->35.85% (43,554,504B) 0x804CB67: scan_root (check.c:869)
| | | | | | | ->35.85% (43,554,504B) 0x804D2AC: main (dosfsck.c:166)
After optimizations:
n time(i) total(B) useful-heap(B) extra-heap(B)
-----------------------------------------------------------------
55 22,411,187,114 81,472,072 81,203,278 268,794
99.67% (81,203,278B) (heap allocation functions) malloc/new/new[],
--alloc-fns, etc.
->99.67% (81,203,278B) 0x804CC40: alloc (common.c:63)
->84.08% (68,498,528B) 0x804CCB5: qalloc (common.c:86)
| ->82.07% (66,863,328B) 0x804C1B2: add_file (check.c:787)
| | ->82.06% (66,859,240B) 0x804C9EA: subdirs (check.c:824)
| | | ->82.05% (66,851,064B) 0x804CA5F: subdirs (check.c:833)
| | | | ->81.43% (66,340,064B) 0x804CA5F: subdirs (check.c:833)
| | | | | ->54.36% (44,289,392B) 0x804CA5F: subdirs (check.c:833)
| | | | | | ->54.36% (44,289,392B) 0x804CB67: scan_root (check.c:869)
| | | | | | ->54.36% (44,289,392B) 0x804D1FD: main (dosfsck.c:166)
For another full, 2GB filesystem:
* Before:
alloc size: count: total: + overhead:
8 248917 1991336 1991336
56 186687 10454472 1493496
94 4427 416138 35416
95 43824 4163280 350592
96 13978 1341888 111824
752556 1 752556 8
total of all allocations + allocations overhead:
= 18672kB + 3889kB = 22561kB
* After common.c patching:
alloc size: count: total: + overhead:
8 4023 32184 32184
4088 4022 16441936 32176
752556 1 752556 8
total of all allocations + allocations overhead:
= 16823kB + 63kB = 16886kB
In dosfsck.h:
* Used bitfields to reduce cluster entry struct size 1/4th. This saves
nearly 4MB of RAM with 2GB file system, more with larger ones
In fat.c:
* Instead of allocating two FATs in the beginning of read_fat(),
having pointer to more correct one and freeing both only at the end of
the function, copy the correct one to memory allocated for the first
one and free second FAT allocation before doing more allocations.
This saves nearly 4MB of RAM with an 2GB file system, more with
larger ones (with empty file system this peak is more significant
than the qalloc() memory overhead fixed in common.c)
Before (on empty 2GB file system):
->100.00% (23,990,528B) 0x804CBC0: alloc (common.c:52)
| ->66.66% (15,993,680B) 0x804CC0E: qalloc (common.c:65)
| | ->66.66% (15,993,680B) 0x804D77B: read_fat (fat.c:101)
| | ->66.66% (15,993,680B) 0x804D224: main (dosfsck.c:166)
| |
| ->16.67% (3,998,420B) 0x804D6E3: read_fat (fat.c:59)
| | ->16.67% (3,998,420B) 0x804D224: main (dosfsck.c:166)
| |
| ->16.67% (3,998,420B) 0x804D720: read_fat (fat.c:63)
| | ->16.67% (3,998,420B) 0x804D224: main (dosfsck.c:166)
After:
->99.99% (15,993,688B) 0x804CC40: alloc (common.c:63)
| ->75.00% (11,995,260B) 0x804CCFC: qalloc (common.c:111)
| | ->75.00% (11,995,260B) 0x804D91D: read_fat (fat.c:108)
| | ->75.00% (11,995,260B) 0x804D324: main (dosfsck.c:166)
| |
| ->25.00% (3,998,420B) 0x804D86B: read_fat (fat.c:59)
| | ->25.00% (3,998,420B) 0x804D324: main (dosfsck.c:166)
- Eero
--
memory saving patch to dosfsck
https://bugs.launchpad.net/bugs/189184
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
