Re: svn commit: r251894 - in head: lib/libmemstat sys/vm
On 18.06.2013 06:50, Jeff Roberson wrote: Author: jeff Date: Tue Jun 18 04:50:20 2013 New Revision: 251894 URL: http://svnweb.freebsd.org/changeset/base/251894 Log: Refine UMA bucket allocation to reduce space consumption and improve performance. - Always free to the alloc bucket if there is space. This gives LIFO allocation order to improve hot-cache performance. This also allows for zones with a single bucket per-cpu rather than a pair if the entire working set fits in one bucket. - Enable per-cpu caches of buckets. To prevent recursive bucket allocation one bucket zone still has per-cpu caches disabled. - Pick the initial bucket size based on a table driven maximum size per-bucket rather than the number of items per-page. This gives more sane initial sizes. - Only grow the bucket size when we face contention on the zone lock, this causes bucket sizes to grow more slowly. - Adjust the number of items per-bucket to account for the header space. This packs the buckets more efficiently per-page while making them not quite powers of two. - Eliminate the per-zone free bucket list. Always return buckets back to the bucket zone. This ensures that as zones grow into larger bucket sizes they eventually discard the smaller sizes. It persists fewer buckets in the system. The locking is slightly trickier. - Only switch buckets in zalloc, not zfree, this eliminates pathological cases where we ping-pong between two buckets. - Ensure that the thread that fills a new bucket gets to allocate from it to give a better upper bound on allocation time. There used to be a problem with per CPU caches accumulating large amounts of items without freeing back to the global (or socket) pool. Do these updates to UMA change this situation and/or do you have further improvements coming up? -- Andre ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r251894 - in head: lib/libmemstat sys/vm
On Tue, Jun 18, 2013 at 10:25:08AM +0200, Andre Oppermann wrote: A There used to be a problem with per CPU caches accumulating large amounts A of items without freeing back to the global (or socket) pool. A A Do these updates to UMA change this situation and/or do you have further A improvements coming up? This is especially a problem with ZFS, which utilizes UMA extensively. IMHO, we need a flag for uma_zcreate() that would disable per CPU caches, so that certain zones (ZFS at least) would have them off. It might be a good idea to force this flag on every zone that has allocation = then the page size. -- Totus tuus, Glebius. ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r251894 - in head: lib/libmemstat sys/vm
On 6/18/13 4:37 AM, Gleb Smirnoff wrote: On Tue, Jun 18, 2013 at 10:25:08AM +0200, Andre Oppermann wrote: A There used to be a problem with per CPU caches accumulating large amounts A of items without freeing back to the global (or socket) pool. A A Do these updates to UMA change this situation and/or do you have further A improvements coming up? This is especially a problem with ZFS, which utilizes UMA extensively. IMHO, we need a flag for uma_zcreate() that would disable per CPU caches, so that certain zones (ZFS at least) would have them off. It might be a good idea to force this flag on every zone that has allocation = then the page size. What about people running with 256GB+ ram? Do they also want the per cpu caches off? -- Alfred Perlstein VP Software Engineering, iXsystems ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r251894 - in head: lib/libmemstat sys/vm
On Tue, 18 Jun 2013, Alfred Perlstein wrote: On 6/18/13 4:37 AM, Gleb Smirnoff wrote: On Tue, Jun 18, 2013 at 10:25:08AM +0200, Andre Oppermann wrote: A There used to be a problem with per CPU caches accumulating large amounts A of items without freeing back to the global (or socket) pool. A A Do these updates to UMA change this situation and/or do you have further A improvements coming up? This is especially a problem with ZFS, which utilizes UMA extensively. IMHO, we need a flag for uma_zcreate() that would disable per CPU caches, so that certain zones (ZFS at least) would have them off. It might be a good idea to force this flag on every zone that has allocation = then the page size. What about people running with 256GB+ ram? Do they also want the per cpu caches off? If you look at the new system there is a static threshold for the initial item size required for different sized per-cpu buckets. What might make sense is to tune this size based on available memory. For what it's worth I looked at solaris settings and they cache roughly 4x as much on a per-cpu basis. The new system should tend to cache less of large and infrequent allocations vs the old system. I can't say yet whether it is still a problem. I have an implementation of vmem to replace using vm_maps for kmem_map, buffer_map, etc. which may resolve the zfs allocation problems. I hope to get this in over the next few weeks. Thanks, Jeff -- Alfred Perlstein VP Software Engineering, iXsystems ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
Re: svn commit: r251894 - in head: lib/libmemstat sys/vm
On 6/18/13 5:21 PM, Jeff Roberson wrote: On Tue, 18 Jun 2013, Alfred Perlstein wrote: On 6/18/13 4:37 AM, Gleb Smirnoff wrote: On Tue, Jun 18, 2013 at 10:25:08AM +0200, Andre Oppermann wrote: A There used to be a problem with per CPU caches accumulating large amounts A of items without freeing back to the global (or socket) pool. A A Do these updates to UMA change this situation and/or do you have further A improvements coming up? This is especially a problem with ZFS, which utilizes UMA extensively. IMHO, we need a flag for uma_zcreate() that would disable per CPU caches, so that certain zones (ZFS at least) would have them off. It might be a good idea to force this flag on every zone that has allocation = then the page size. What about people running with 256GB+ ram? Do they also want the per cpu caches off? If you look at the new system there is a static threshold for the initial item size required for different sized per-cpu buckets. What might make sense is to tune this size based on available memory. For what it's worth I looked at solaris settings and they cache roughly 4x as much on a per-cpu basis. The new system should tend to cache less of large and infrequent allocations vs the old system. I can't say yet whether it is still a problem. I have an implementation of vmem to replace using vm_maps for kmem_map, buffer_map, etc. which may resolve the zfs allocation problems. I hope to get this in over the next few weeks. That looks really exciting Jeff. Thank you. I'm hoping we can give back some testing numbers when it goes in. -Alfred ___ svn-src-head@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-head To unsubscribe, send any mail to svn-src-head-unsubscr...@freebsd.org
svn commit: r251894 - in head: lib/libmemstat sys/vm
Author: jeff
Date: Tue Jun 18 04:50:20 2013
New Revision: 251894
URL: http://svnweb.freebsd.org/changeset/base/251894
Log:
Refine UMA bucket allocation to reduce space consumption and improve
performance.
- Always free to the alloc bucket if there is space. This gives LIFO
allocation order to improve hot-cache performance. This also allows
for zones with a single bucket per-cpu rather than a pair if the entire
working set fits in one bucket.
- Enable per-cpu caches of buckets. To prevent recursive bucket
allocation one bucket zone still has per-cpu caches disabled.
- Pick the initial bucket size based on a table driven maximum size
per-bucket rather than the number of items per-page. This gives
more sane initial sizes.
- Only grow the bucket size when we face contention on the zone lock, this
causes bucket sizes to grow more slowly.
- Adjust the number of items per-bucket to account for the header space.
This packs the buckets more efficiently per-page while making them
not quite powers of two.
- Eliminate the per-zone free bucket list. Always return buckets back
to the bucket zone. This ensures that as zones grow into larger
bucket sizes they eventually discard the smaller sizes. It persists
fewer buckets in the system. The locking is slightly trickier.
- Only switch buckets in zalloc, not zfree, this eliminates pathological
cases where we ping-pong between two buckets.
- Ensure that the thread that fills a new bucket gets to allocate from
it to give a better upper bound on allocation time.
Sponsored by: EMC / Isilon Storage Division
Modified:
head/lib/libmemstat/memstat_uma.c
head/sys/vm/uma_core.c
head/sys/vm/uma_int.h
Modified: head/lib/libmemstat/memstat_uma.c
==
--- head/lib/libmemstat/memstat_uma.c Tue Jun 18 04:11:16 2013
(r251893)
+++ head/lib/libmemstat/memstat_uma.c Tue Jun 18 04:50:20 2013
(r251894)
@@ -446,7 +446,7 @@ skip_percpu:
kz.uk_ipers;
mtp-mt_byteslimit = mtp-mt_countlimit * mtp-mt_size;
mtp-mt_count = mtp-mt_numallocs - mtp-mt_numfrees;
- for (ubp = LIST_FIRST(uz.uz_full_bucket); ubp !=
+ for (ubp = LIST_FIRST(uz.uz_buckets); ubp !=
NULL; ubp = LIST_NEXT(ub, ub_link)) {
ret = kread(kvm, ubp, ub, sizeof(ub), 0);
mtp-mt_zonefree += ub.ub_cnt;
Modified: head/sys/vm/uma_core.c
==
--- head/sys/vm/uma_core.c Tue Jun 18 04:11:16 2013(r251893)
+++ head/sys/vm/uma_core.c Tue Jun 18 04:50:20 2013(r251894)
@@ -192,27 +192,26 @@ struct uma_kctor_args {
struct uma_bucket_zone {
uma_zone_t ubz_zone;
char*ubz_name;
- int ubz_entries;
+ int ubz_entries;/* Number of items it can hold. */
+ int ubz_maxsize;/* Maximum allocation size per-item. */
};
-#defineBUCKET_MAX 128
+/*
+ * Compute the actual number of bucket entries to pack them in power
+ * of two sizes for more efficient space utilization.
+ */
+#defineBUCKET_SIZE(n) \
+(((sizeof(void *) * (n)) - sizeof(struct uma_bucket)) / sizeof(void *))
+
+#defineBUCKET_MAX BUCKET_SIZE(128)
struct uma_bucket_zone bucket_zones[] = {
- { NULL, 16 Bucket, 16 },
- { NULL, 32 Bucket, 32 },
- { NULL, 64 Bucket, 64 },
- { NULL, 128 Bucket, 128 },
+ { NULL, 32 Bucket, BUCKET_SIZE(32), 512 },
+ { NULL, 64 Bucket, BUCKET_SIZE(64), 256 },
+ { NULL, 128 Bucket, BUCKET_SIZE(128), 128 },
{ NULL, NULL, 0}
};
-
-#defineBUCKET_SHIFT4
-#defineBUCKET_ZONES((BUCKET_MAX BUCKET_SHIFT) + 1)
-
-/*
- * bucket_size[] maps requested bucket sizes to zones that allocate a bucket
- * of approximately the right size.
- */
-static uint8_t bucket_size[BUCKET_ZONES];
+static uma_zone_t largebucket;
/*
* Flags and enumerations to be passed to internal functions.
@@ -250,7 +249,7 @@ static void bucket_init(void);
static uma_bucket_t bucket_alloc(int, int);
static void bucket_free(uma_bucket_t);
static void bucket_zone_drain(void);
-static int zone_alloc_bucket(uma_zone_t zone, int flags);
+static uma_bucket_t zone_alloc_bucket(uma_zone_t zone, int flags);
static uma_slab_t zone_fetch_slab(uma_zone_t zone, uma_keg_t last, int flags);
static uma_slab_t zone_fetch_slab_multi(uma_zone_t zone, uma_keg_t last, int
flags);
static void *slab_alloc_item(uma_keg_t keg, uma_slab_t slab);
@@ -283,7 +282,6 @@ SYSCTL_INT(_vm, OID_AUTO, zone_warnings,
/*
* This routine checks to see whether or not it's safe
