Op 10-05-11 06:56, Edward Ned Harvey schreef:
>> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-
>> boun...@opensolaris.org] On Behalf Of Edward Ned Harvey
>>
>> BTW, here's how to tune it:
>>
>> echo "arc_meta_limit/Z 0x30000000" | sudo mdb -kw
>>
>> echo "::arc" | sudo mdb -k | grep meta_limit
>> arc_meta_limit = 768 MB
>
> Well ... I don't know what to think yet. I've been reading these numbers
> for like an hour, finding interesting things here and there, but nothing to
> really solidly point my finger at.
>
> The one thing I know for sure... The free mem drops at an unnatural rate.
> Initially the free mem disappears at a rate approx 2x faster than the sum of
> file size and metadata combined. Meaning the system could be caching the
> entire file and all the metadata, and that would only explain half of the
> memory disappearance.
I'm seeing similar things. Yesterday I first rebooted with set
zfs:zfs_arc_meta_limit=0x100000000 (that's 4 GiB) set in /etc/system and
monitored while the box was doing its regular job (taking backups).
zfs_arc_min is also set to 4 GiB. What I noticed is that shortly after
the reboot, the arc started filling up rapidly, mostly with metadata. It
shot up to:
arc_meta_max = 3130 MB
afterwards, the number for arc_meta_used steadily dropped. Some 12 hours
ago, I started deleting files, it has deleted about 6000000 files since
then. Now at the moment the arc size stays right at the minimum of 2
GiB, of which metadata fluctuates around 1650 MB.
This is the output of the getmemstats.sh script you posted.
Memory: 6135M phys mem, 539M free mem, 6144M total swap, 6144M free swap
zfs:0:arcstats:c 2147483648 = 2 GiB target size
zfs:0:arcstats:c_max 5350862848 = 5 GiB
zfs:0:arcstats:c_min 2147483648 = 2 GiB
zfs:0:arcstats:data_size 829660160 = 791 MiB
zfs:0:arcstats:hdr_size 93396336 = 89 MiB
zfs:0:arcstats:other_size 411215168 = 392 MiB
zfs:0:arcstats:size 1741492896 = 1661 Mi
arc_meta_used = 1626 MB
arc_meta_limit = 4096 MB
arc_meta_max = 3130 MB
I get way more cache misses then I'd like:
Time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
10:01:13 3K 380 10 166 7 214 15 259 7 1G 2G
10:02:13 2K 340 16 37 2 302 46 323 16 1G 2G
10:03:13 2K 368 18 47 3 321 46 347 17 1G 2G
10:04:13 1K 348 25 44 4 303 63 335 24 1G 2G
10:05:13 2K 420 15 87 4 332 36 383 14 1G 2G
10:06:13 3K 489 16 132 6 357 35 427 14 1G 2G
10:07:13 2K 405 15 49 2 355 39 401 15 1G 2G
10:08:13 2K 366 13 40 2 326 37 366 13 1G 2G
10:09:13 1K 364 20 18 1 345 58 363 20 1G 2G
10:10:13 4K 370 8 59 2 311 21 369 8 1G 2G
10:11:13 4K 351 8 57 2 294 21 350 8 1G 2G
10:12:13 3K 378 10 59 2 319 26 372 10 1G 2G
10:13:13 3K 393 11 53 2 339 28 393 11 1G 2G
10:14:13 2K 403 13 40 2 363 35 402 13 1G 2G
10:15:13 3K 365 11 48 2 317 30 365 11 1G 2G
10:16:13 2K 374 15 40 2 334 40 374 15 1G 2G
10:17:13 3K 385 12 43 2 341 28 383 12 1G 2G
10:18:13 4K 343 8 64 2 279 19 343 8 1G 2G
10:19:13 3K 391 10 59 2 332 23 391 10 1G 2G
So, one explanation I can think of is that the "rest" of the memory are
l2arc pointers, supposing they are not actually counted in the arc
memory usage totals (AFAIK l2arc pointers are considered to be part of
arc). Then again my l2arc is still growing (slowly) and I'm only caching
metadata at the moment, so you'd think it'd shrink if there's no more
room for l2arc pointers. Besides I'm getting very little reads from ssd:
capacity operations bandwidth
pool alloc free read write read write
------------ ----- ----- ----- ----- ----- -----
backups 5.49T 1.57T 415 121 3.13M 1.58M
raidz1 5.49T 1.57T 415 121 3.13M 1.58M
c0t0d0s1 - - 170 16 2.47M 551K
c0t1d0s1 - - 171 16 2.46M 550K
c0t2d0s1 - - 170 16 2.53M 552K
c0t3d0s1 - - 170 16 2.44M 550K
cache - - - - - -
c1t5d0 63.4G 48.4G 20 0 2.45M 0
------------ ----- ----- ----- ----- ----- -----
(typical statistic over 1 minute)
I might try the "windows solution" and reboot the machine to free up
memory and let it fill the cache all over again and see if I get more
cache hits... hmmm...
> I set the arc_meta_limit to 768 as mentioned above. I ran all these tests,
> and here are the results:
> (sorry it's extremely verbose)
> http://dl.dropbox.com/u/543241/dedup%20tests/runtest-output.xlsx
>
> BTW, here are all the scripts etc that I used to produce those results:
> http://dl.dropbox.com/u/543241/dedup%20tests/datagenerate.c
> http://dl.dropbox.com/u/543241/dedup%20tests/getmemstats.sh
> http://dl.dropbox.com/u/543241/dedup%20tests/runtest.sh
--
No part of this copyright message may be reproduced, read or seen,
dead or alive or by any means, including but not limited to telepathy
without the benevolence of the author.
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
