> A couple more questions here. > > [mpstat] > > > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl > > 0 0 0 3109 3616 316 196 5 17 48 45 245 0 85 0 15 > > 1 0 0 3127 3797 592 217 4 17 63 46 176 0 84 0 15 > > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl > > 0 0 0 3051 3529 277 201 2 14 25 48 216 0 83 0 17 > > 1 0 0 3065 3739 606 195 2 14 37 47 153 0 82 0 17 > > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl > > 0 0 0 3011 3538 316 242 3 26 16 52 202 0 81 0 19 > > 1 0 0 3019 3698 578 269 4 25 23 56 309 0 83 0 17 ... > The largest numbers from mpstat are for interrupts and cross calls. > What does intrstat(1M) show? > > Have you run dtrace to determine the most frequent cross-callers?
As far as I understand it, we have these frequent cross calls because 1. the test was run on an x86 MP machine 2. the kernel zmod / gzip code allocates and frees four big chunks of memory (4 * 65544 bytes) per zio_write_compress ( gzip ) call [1] Freeing these big memory chunks generates lots of cross calls, because page table entries for that memory are invalidated on all cpus (cores). Of cause this effect cannot be observed on an uniprocessor machine (one cpu / core). And apparently it isn't the root cause for the bad interactive performance with this test; the bad interactive performance can also be observed on single cpu/single core x86 machines. A possible optimization for MP machines: use some kind of kmem_cache for the gzip buffers, so that these buffers could be reused between gzip compression calls. [1] allocations per zio_write_compress() / gzip_compress() call: 1 6642 kobj_alloc:entry sz 5936, fl 1001 1 6642 kobj_alloc:entry sz 65544, fl 1001 1 6642 kobj_alloc:entry sz 65544, fl 1001 1 6642 kobj_alloc:entry sz 65544, fl 1001 1 6642 kobj_alloc:entry sz 65544, fl 1001 1 5769 kobj_free:entry fffffffeeb307000: sz 65544 1 5769 kobj_free:entry fffffffeeb2f5000: sz 65544 1 5769 kobj_free:entry fffffffeeb2e3000: sz 65544 1 5769 kobj_free:entry fffffffeeb2d1000: sz 65544 1 5769 kobj_free:entry fffffffed1c42000: sz 5936 This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
