Re: Residency profiles
This recently came up again. It seems that `+RTS -h -i0` will just turn every minor collection into a major one: https://gitlab.haskell.org/ghc/ghc/issues/17387#note_248705 `-i0` seems significantly different from `-i0.001`, say, in that it just turns minor GCs into major ones and doesn't introduce non-determinism otherwise. Sampling rate can be controlled with `-A`, much like `-F1` (but it's still faster for some reason). Am Mo., 10. Dez. 2018 um 09:11 Uhr schrieb Simon Marlow : > https://phabricator.haskell.org/D5428 > > > On Sun, 9 Dec 2018 at 10:12, Sebastian Graf wrote: > >> Ah, I was only looking at `+RTS --help`, not the users guide. Silly me. >> >> Am Do., 6. Dez. 2018 um 20:53 Uhr schrieb Simon Marlow < >> marlo...@gmail.com>: >> >>> It is documented! >>> https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--F%20%E2%9F%A8factor%E2%9F%A9 >>> >>> On Thu, 6 Dec 2018 at 16:21, Sebastian Graf wrote: >>> Hey, thanks, all! Measuring with `-A1M -F1` delivers much more reliable residency numbers. `-F` doesn't seem to be documented. From reading `rts/RtsFlags.c` and `rts/sm/GC.c` I gather that it's the factor by which to multiply the number of live bytes by to get the new old gen size? So effectively, the old gen will 'overflow' on every minor GC, neat! Greetings Sebastian Am Do., 6. Dez. 2018 um 12:52 Uhr schrieb Simon Peyton Jones via ghc-devs : > | Right. A parameter for fixing the nursery size would be easy to > implement, > | I think. Just a new flag, then in GC.c:resize_nursery() use the > flag as the > | nursery size. > > Super! That would be v useful. > > | "Max. residency" is really hard to measure (need to do very > frequent GCs), > | perhaps a better question to ask is "residency when the program is > in state > | S". > > Actually, Sebastian simply wants to see an accurate, reproducible > residency profile, and doing frequent GCs might well be an acceptable > cost. > > Simon > ___ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Residency profiles
https://phabricator.haskell.org/D5428 On Sun, 9 Dec 2018 at 10:12, Sebastian Graf wrote: > Ah, I was only looking at `+RTS --help`, not the users guide. Silly me. > > Am Do., 6. Dez. 2018 um 20:53 Uhr schrieb Simon Marlow >: > >> It is documented! >> https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--F%20%E2%9F%A8factor%E2%9F%A9 >> >> On Thu, 6 Dec 2018 at 16:21, Sebastian Graf wrote: >> >>> Hey, >>> >>> thanks, all! Measuring with `-A1M -F1` delivers much more reliable >>> residency numbers. >>> `-F` doesn't seem to be documented. From reading `rts/RtsFlags.c` and >>> `rts/sm/GC.c` I gather that it's the factor by which to multiply the number >>> of live bytes by to get the new old gen size? >>> So effectively, the old gen will 'overflow' on every minor GC, neat! >>> >>> Greetings >>> Sebastian >>> >>> Am Do., 6. Dez. 2018 um 12:52 Uhr schrieb Simon Peyton Jones via >>> ghc-devs : >>> | Right. A parameter for fixing the nursery size would be easy to implement, | I think. Just a new flag, then in GC.c:resize_nursery() use the flag as the | nursery size. Super! That would be v useful. | "Max. residency" is really hard to measure (need to do very frequent GCs), | perhaps a better question to ask is "residency when the program is in state | S". Actually, Sebastian simply wants to see an accurate, reproducible residency profile, and doing frequent GCs might well be an acceptable cost. Simon ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Residency profiles
Ah, I was only looking at `+RTS --help`, not the users guide. Silly me. Am Do., 6. Dez. 2018 um 20:53 Uhr schrieb Simon Marlow : > It is documented! > https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--F%20%E2%9F%A8factor%E2%9F%A9 > > On Thu, 6 Dec 2018 at 16:21, Sebastian Graf wrote: > >> Hey, >> >> thanks, all! Measuring with `-A1M -F1` delivers much more reliable >> residency numbers. >> `-F` doesn't seem to be documented. From reading `rts/RtsFlags.c` and >> `rts/sm/GC.c` I gather that it's the factor by which to multiply the number >> of live bytes by to get the new old gen size? >> So effectively, the old gen will 'overflow' on every minor GC, neat! >> >> Greetings >> Sebastian >> >> Am Do., 6. Dez. 2018 um 12:52 Uhr schrieb Simon Peyton Jones via ghc-devs >> : >> >>> | Right. A parameter for fixing the nursery size would be easy to >>> implement, >>> | I think. Just a new flag, then in GC.c:resize_nursery() use the flag >>> as the >>> | nursery size. >>> >>> Super! That would be v useful. >>> >>> | "Max. residency" is really hard to measure (need to do very frequent >>> GCs), >>> | perhaps a better question to ask is "residency when the program is in >>> state >>> | S". >>> >>> Actually, Sebastian simply wants to see an accurate, reproducible >>> residency profile, and doing frequent GCs might well be an acceptable >>> cost. >>> >>> Simon >>> ___ >>> ghc-devs mailing list >>> ghc-devs@haskell.org >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >>> >> ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Residency profiles
It is documented! https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/runtime_control.html#rts-flag--F%20%E2%9F%A8factor%E2%9F%A9 On Thu, 6 Dec 2018 at 16:21, Sebastian Graf wrote: > Hey, > > thanks, all! Measuring with `-A1M -F1` delivers much more reliable > residency numbers. > `-F` doesn't seem to be documented. From reading `rts/RtsFlags.c` and > `rts/sm/GC.c` I gather that it's the factor by which to multiply the number > of live bytes by to get the new old gen size? > So effectively, the old gen will 'overflow' on every minor GC, neat! > > Greetings > Sebastian > > Am Do., 6. Dez. 2018 um 12:52 Uhr schrieb Simon Peyton Jones via ghc-devs < > ghc-devs@haskell.org>: > >> | Right. A parameter for fixing the nursery size would be easy to >> implement, >> | I think. Just a new flag, then in GC.c:resize_nursery() use the flag >> as the >> | nursery size. >> >> Super! That would be v useful. >> >> | "Max. residency" is really hard to measure (need to do very frequent >> GCs), >> | perhaps a better question to ask is "residency when the program is in >> state >> | S". >> >> Actually, Sebastian simply wants to see an accurate, reproducible >> residency profile, and doing frequent GCs might well be an acceptable >> cost. >> >> Simon >> ___ >> ghc-devs mailing list >> ghc-devs@haskell.org >> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs >> > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Residency profiles
Hey, thanks, all! Measuring with `-A1M -F1` delivers much more reliable residency numbers. `-F` doesn't seem to be documented. From reading `rts/RtsFlags.c` and `rts/sm/GC.c` I gather that it's the factor by which to multiply the number of live bytes by to get the new old gen size? So effectively, the old gen will 'overflow' on every minor GC, neat! Greetings Sebastian Am Do., 6. Dez. 2018 um 12:52 Uhr schrieb Simon Peyton Jones via ghc-devs < ghc-devs@haskell.org>: > | Right. A parameter for fixing the nursery size would be easy to > implement, > | I think. Just a new flag, then in GC.c:resize_nursery() use the flag as > the > | nursery size. > > Super! That would be v useful. > > | "Max. residency" is really hard to measure (need to do very frequent > GCs), > | perhaps a better question to ask is "residency when the program is in > state > | S". > > Actually, Sebastian simply wants to see an accurate, reproducible > residency profile, and doing frequent GCs might well be an acceptable > cost. > > Simon > ___ > ghc-devs mailing list > ghc-devs@haskell.org > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs > ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
RE: Residency profiles
| Right. A parameter for fixing the nursery size would be easy to implement, | I think. Just a new flag, then in GC.c:resize_nursery() use the flag as the | nursery size. Super! That would be v useful. | "Max. residency" is really hard to measure (need to do very frequent GCs), | perhaps a better question to ask is "residency when the program is in state | S". Actually, Sebastian simply wants to see an accurate, reproducible residency profile, and doing frequent GCs might well be an acceptable cost. Simon ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Re: Residency profiles
Hi, > I think what we want is a way to trigger GC at very regular intervals, after > (say) each 10kbytes or 100kbytes or 1Mbyte of allocation. That might be > expensive, but we’d get reproducible results. If we could fix the nursery size to 10kb that'd trigger a GC in every 10kb of allocation (you could still allocate large objects as those are not allocated in the nursery, but perhaps that's not a problem in your benchmarks). Then by setting -G1 you could turn all GCs to major GCs (because first generation is always collected). Note that because each capability has its own nursery you may want to set the nursery size to alloc_per_gc / num_of_caps if you need more than one capability. > I don’t think that is possible right now – see the ticket – but it would be > easy enough to do wouldn’t it? Just give only 10k or 100k or 1M to the > allocator when setting it running again. Right. A parameter for fixing the nursery size would be easy to implement, I think. Just a new flag, then in GC.c:resize_nursery() use the flag as the nursery size. "Max. residency" is really hard to measure (need to do very frequent GCs), perhaps a better question to ask is "residency when the program is in state S". This is also hard to measure if your program is threaded or have other non-determinism, but this lets you decide when to measure residency. Currently we can't tell the GC to print residency stats, but perhaps we could implement a variant of `performGC` that prints residency after the GC. So in your program you could add `performGCPrintStats` after every iteration or step etc. Not sure how useful this would be, but just an idea.. On 6.12.2018 13:09, Simon Peyton Jones wrote: Simon, Ben, Omer As you’ll see in comments 55-72 of https://ghc.haskell.org/trac/ghc/ticket/9476, Sebastian has been a bit flummoxed by the task of measure residency profiles; that is, how much data is truly live during execution. A major GC measures that, but we are vulnerable to exactly when it happens (even with -G1) and that can lead to irreproducible results. I think what we want is a way to trigger GC at very regular intervals, after (say) each 10kbytes or 100kbytes or 1Mbyte of allocation. That might be expensive, but we’d get reproducible results. I don’t think that is possible right now – see the ticket – but it would be easy enough to do wouldn’t it? Just give only 10k or 100k or 1M to the allocator when setting it running again. Would you consider this? Or are we just missing something obvious? Needless to say, we want to do all this with full optimisation on, no cost-centre profiling. Thanks Simon -- Ömer Sinan Ağacan, Haskell Consultant Well-Typed LLP, http://www.well-typed.com Registered in England & Wales, OC335890 118 Wymering Mansions, Wymering Road, London W9 2NF, England ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Residency profiles
Simon, Ben, Omer As you'll see in comments 55-72 of https://ghc.haskell.org/trac/ghc/ticket/9476, Sebastian has been a bit flummoxed by the task of measure residency profiles; that is, how much data is truly live during execution. A major GC measures that, but we are vulnerable to exactly when it happens (even with -G1) and that can lead to irreproducible results. I think what we want is a way to trigger GC at very regular intervals, after (say) each 10kbytes or 100kbytes or 1Mbyte of allocation. That might be expensive, but we'd get reproducible results. I don't think that is possible right now - see the ticket - but it would be easy enough to do wouldn't it? Just give only 10k or 100k or 1M to the allocator when setting it running again. Would you consider this? Or are we just missing something obvious? Needless to say, we want to do all this with full optimisation on, no cost-centre profiling. Thanks Simon ___ ghc-devs mailing list ghc-devs@haskell.org http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
