Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

commit a19aed147bf1605b21cbe7b1909ff6cbf519fb64 Author: Han-Wen Nienhuys Date: Sat Feb 8 21:02:12 2020 +0100 GUILE2: Scale GC heap with the number of smobs https://codereview.appspot.com/561390043/

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/09 10:08:20, hanwenn wrote: > A way around that is to change all instances of vectors holding SCM values to a > new > gc_vector type that has a custom allocator. That will be significant work, but > probably > desirable in the long term. I'd recommend a two-pronged approach for now,

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Feb 9, 2020, at 04:25, jonas.hahnf...@gmail.com wrote: > > Anyway I think adding a comment here and in smobs.cc how this is > initialized (refer to static lifetime in C/C++) would likely clarify the > intent. Initialization is an area where the legacy of the language is complicated (more

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

https://codereview.appspot.com/561390043/diff/567180043/lily/include/smobs.hh File lily/include/smobs.hh (right): https://codereview.appspot.com/561390043/diff/567180043/lily/include/smobs.hh#newcode312 lily/include/smobs.hh:312: static size_t count; On 2020/02/09 03:29:26, Dan Eble wrote: > It

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

LGTM, thanks for this clearly documented change! https://codereview.appspot.com/561390043/

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

https://codereview.appspot.com/561390043/diff/567180043/lily/include/smobs.hh File lily/include/smobs.hh (right): https://codereview.appspot.com/561390043/diff/567180043/lily/include/smobs.hh#newcode312 lily/include/smobs.hh:312: static size_t count; On 2020/02/09 03:29:26, Dan Eble wrote: > It

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/09 09:55:12, hahnjo wrote: > On 2020/02/09 09:36:40, hanwenn wrote: > > On Sun, Feb 9, 2020 at 10:25 AM wrote: > > > FWIW I tried to research on the internals of GC. I found the following > > > statement that decides whether to collect or just expand

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/09 09:36:40, hanwenn wrote: > On Sun, Feb 9, 2020 at 10:25 AM wrote: > > FWIW I tried to research on the internals of GC. I found the following > > statement that decides whether to collect or just expand the heap: > >

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Sun, Feb 9, 2020 at 10:25 AM wrote: > The approach sounds good to me. I can confirm that this works on my > system and the runtime is pretty good. Please readd the autoconf logic > to detect bdw-gc. > will do. > FWIW I tried to research on the internals of GC. I found the following >

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

The approach sounds good to me. I can confirm that this works on my system and the runtime is pretty good. Please readd the autoconf logic to detect bdw-gc. FWIW I tried to research on the internals of GC. I found the following statement that decides whether to collect or just expand the heap:

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

https://codereview.appspot.com/561390043/diff/567180043/lily/include/smobs.hh File lily/include/smobs.hh (right): https://codereview.appspot.com/561390043/diff/567180043/lily/include/smobs.hh#newcode312 lily/include/smobs.hh:312: static size_t count; It seems that this is initialized to zero

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/08 20:05:53, hanwenn wrote: > Use smob count as memory proxy This looks good to me now; softcoding the 2000 is left as an exercise to the reader in another commit https://codereview.appspot.com/561390043/

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Fri, Feb 7, 2020 at 7:27 PM David Kastrup wrote: > Han-Wen Nienhuys writes: > > > Single threaded, the numbers make more sense: > > > > > > MAX=INIT=2G > > gc time taken: 1.843968805 > > User time (seconds): 49.36 > > > > MAX=INIT=1G > > gc time taken: 3.291264925 > > User time (seconds):

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Han-Wen Nienhuys writes: > Single threaded, the numbers make more sense: > > > MAX=INIT=2G > gc time taken: 1.843968805 > User time (seconds): 49.36 > > MAX=INIT=1G > gc time taken: 3.291264925 > User time (seconds): 51.74 > > MAX=INIT=800M > gc time taken: 5.760042906 > User time (seconds):

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Fri, Feb 7, 2020 at 5:53 PM Han-Wen Nienhuys wrote: > Thanks, I ran the carver score successfully now. > > I took some pauses between runs so thermal throttling wasn't an issue. > > My standard guile2.2 branch (with 40M initial heap), takes about 2m wall > time, 1m of GC time. > > The stats

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Thanks, I ran the carver score successfully now. I took some pauses between runs so thermal throttling wasn't an issue. My standard guile2.2 branch (with 40M initial heap), takes about 2m wall time, 1m of GC time. The stats printing shows that we have a very full heap: In-use heap: 85% (370824

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Am Donnerstag, den 06.02.2020, 00:27 +0100 schrieb David Kastrup: > Han-Wen Nienhuys < > hanw...@gmail.com > > writes: > > > On Wed, Feb 5, 2020 at 12:33 PM < > > jonas.hahnf...@gmail.com > > > wrote: > > > > when you see things like: > > > > World-stopped marking took 187 msecs (77 in average)

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Han-Wen Nienhuys writes: > On Wed, Feb 5, 2020 at 12:33 PM wrote: > > when you see things like: > > World-stopped marking took 187 msecs (77 in average) > In-use heap: 51% (40071 KiB pointers + 6357 KiB other) > > it means it took 187ms for GC, and then reclaimed 49% of the memory. > > If you

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Thu, Feb 6, 2020 at 12:19 AM Han-Wen Nienhuys wrote: > Um, that outputs a lot. Anything in particular? >> > > when you see things like: > > World-stopped marking took 187 msecs (77 in average) > In-use heap: 51% (40071 KiB pointers + 6357 KiB other) > > it means it took 187ms for GC, and then

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Wed, Feb 5, 2020 at 12:33 PM wrote: > > Can you do some timings with different values for GC_MAXIMUM_HEAP_SIZE > , eg. > > > > GC_MAXIMUM_HEAP_SIZE=100M > > Not sure if you actually mean GC_MAXIMUM_HEAP_SIZE or rather > GC_INITIAL_HEAP_SIZE? > > I mean maximum, it sets the total amount of

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Dan Eble writes: > On Feb 5, 2020, at 07:59, jonas.hahnf...@gmail.com wrote: >> >> and it seems stable, but I have yet to build a theory why OS thread >> migration actually improves performance… > > Educated guess: The core gets too hot. The OS is not allowed to move > the thread to a cooler

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Feb 5, 2020, at 07:59, jonas.hahnf...@gmail.com wrote: > > and it seems stable, but I have yet to build a theory why OS thread > migration actually improves performance… Educated guess: The core gets too hot. The OS is not allowed to move the thread to a cooler core, so it throttles the

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Some more numbers: I took guile22-experiment and removed the following: * GC_set_free_space_divisor, * GC_INITIAL_HEAP_SIZE=40M * heap growing in Score_engraver This gives me: ~2m30s (although I saw one run in 1m55s?!?) GC_INITIAL_HEAP_SIZE=40M: ~2m10s (one run in 1m40s)

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/04 22:23:45, hanwenn wrote: > On Tue, Feb 4, 2020 at 7:50 PM wrote: > > > On 2020/02/02 22:33:50, hanwenn wrote: > > > For testing, use > > > > > > https://github.com/hanwen/lilypond/tree/guile22-experiment > > > > So I ran this with the large example

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Tue, Feb 4, 2020 at 7:50 PM wrote: > On 2020/02/02 22:33:50, hanwenn wrote: > > For testing, use > > > > https://github.com/hanwen/lilypond/tree/guile22-experiment > > So I ran this with the large example and I see > GC Warning: Failed to expand heap by 30635458560 bytes > a few times (that's

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/02 22:33:50, hanwenn wrote: > For testing, use > > https://github.com/hanwen/lilypond/tree/guile22-experiment So I ran this with the large example and I see GC Warning: Failed to expand heap by 30635458560 bytes a few times (that's 30 GB, my laptop only has 8 GB!!) and finally

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Am Montag, den 03.02.2020, 09:58 -0500 schrieb Dan Eble: > On Feb 3, 2020, at 04:30, Han-Wen Nienhuys < > hanw...@gmail.com > > wrote: > > Instead of waiting for complaints, a change is > > pushed once it passes tests and someone LGTM'd it. > > I've worked in places where commits were handled

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Dan Eble writes: > On Feb 3, 2020, at 04:30, Han-Wen Nienhuys wrote: >> >> Instead of waiting for complaints, a change is >> pushed once it passes tests and someone LGTM'd it. > > I've worked in places where commits were handled with self-discipline > and mutual accountability, and I've worked

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Feb 3, 2020, at 04:30, Han-Wen Nienhuys wrote: > > Instead of waiting for complaints, a change is > pushed once it passes tests and someone LGTM'd it. I've worked in places where commits were handled with self-discipline and mutual accountability, and I've worked in places where a UI

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Han-Wen Nienhuys writes: > On Mon, Feb 3, 2020 at 9:35 AM Han-Wen Nienhuys wrote: >> > > For me, juggling 15 different outstanding code reviews at the same >> > > time is the bane of the current development process. >> > >> > what do you suggest? >> >> I think we should move to a git-based code

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

Han-Wen Nienhuys writes: > On Sun, Feb 2, 2020 at 11:47 PM wrote: >> >> On 02/02/2020 22:33, Han-Wen Nienhuys wrote: >> > For me, juggling 15 different outstanding code reviews at the same >> > time is the bane of the current development process. >> >> what do you suggest? > > I think we should

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Mon, Feb 3, 2020 at 9:35 AM Han-Wen Nienhuys wrote: > > On 02/02/2020 22:33, Han-Wen Nienhuys wrote: > > > For me, juggling 15 different outstanding code reviews at the same > > > time is the bane of the current development process. > > > > what do you suggest? > > I think we should move to a

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Mon, Feb 3, 2020 at 9:35 AM Han-Wen Nienhuys wrote: > > > For me, juggling 15 different outstanding code reviews at the same > > > time is the bane of the current development process. > > > > what do you suggest? > > I think we should move to a git-based code review tool; Github, Gitlab > or

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Mon, Feb 3, 2020 at 9:35 AM Han-Wen Nienhuys wrote: > > On Sun, Feb 2, 2020 at 11:47 PM wrote: > > > > On 02/02/2020 22:33, Han-Wen Nienhuys wrote: > > > For me, juggling 15 different outstanding code reviews at the same > > > time is the bane of the current development process. > > > > what

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On Sun, Feb 2, 2020 at 11:47 PM wrote: > > On 02/02/2020 22:33, Han-Wen Nienhuys wrote: > > For me, juggling 15 different outstanding code reviews at the same > > time is the bane of the current development process. > > what do you suggest? I think we should move to a git-based code review tool;

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 02/02/2020 22:33, Han-Wen Nienhuys wrote: For me, juggling 15 different outstanding code reviews at the same time is the bane of the current development process. what do you suggest? James

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

For testing, use https://github.com/hanwen/lilypond/tree/guile22-experiment in particular, you need https://github.com/hanwen/lilypond/commit/b696550379831ecec1519be6d59cd8a3003e5329 for the UTF-8 parsing. I fixed this a week ago, but due to the delays in getting the preceding fix in

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

jonas.hahnf...@gmail.com writes: > I just tried to reproduce the timings for commits already in master and > this patch. To be honest I don't see a clear picture yet. > > Yes, this change seems to improve the time spent for garbage collection, > but the real time reported by "time" only decreases

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

I just tried to reproduce the timings for commits already in master and this patch. To be honest I don't see a clear picture yet. Yes, this change seems to improve the time spent for garbage collection, but the real time reported by "time" only decreases by a fraction (less than 50% of the saved

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/02 13:45:34, hahnjo wrote: > On 2020/02/02 13:43:25, hanwenn wrote: > > jonas' comments > > The uploaded diff has the wrong base, it's reverting quite some changes from > master. I hate Rietveld. Fixed. https://codereview.appspot.com/561390043/

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/02 13:43:25, hanwenn wrote: > jonas' comments The uploaded diff has the wrong base, it's reverting quite some changes from master. https://codereview.appspot.com/561390043/

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

https://codereview.appspot.com/561390043/diff/565600046/configure.ac File configure.ac (right): https://codereview.appspot.com/561390043/diff/565600046/configure.ac#newcode270 configure.ac:270: PKG_CHECK_MODULES(BDWGC, bdw-gc) On 2020/02/02 13:21:09, hahnjo wrote: > This should fail if not

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

https://codereview.appspot.com/561390043/diff/565600046/configure.ac File configure.ac (right): https://codereview.appspot.com/561390043/diff/565600046/configure.ac#newcode270 configure.ac:270: PKG_CHECK_MODULES(BDWGC, bdw-gc) This should fail if not found, you're using it unconditionally

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/01 11:00:41, hahnjo wrote: > https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh > File lily/include/score-engraver.hh (right): > > https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh#newcode26 >

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/01 21:59:14, dak wrote: > On 2020/02/01 21:23:51, hanwenn wrote: > > On 2020/02/01 20:59:06, dak wrote: > > > On 2020/02/01 20:49:09, hanwenn wrote: > > > > On 2020/02/01 11:00:41, hahnjo wrote: > > > > > > > > > > > > > > >

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/02 09:49:50, hahnjo wrote: > On 2020/02/01 20:49:09, hanwenn wrote: > > On 2020/02/01 11:00:41, hahnjo wrote: > > > > > > https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh > > > File lily/include/score-engraver.hh (right): > > > > > > > > >

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/01 20:49:09, hanwenn wrote: > On 2020/02/01 11:00:41, hahnjo wrote: > > > https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh > > File lily/include/score-engraver.hh (right): > > > > >

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/01 21:23:51, hanwenn wrote: > On 2020/02/01 20:59:06, dak wrote: > > On 2020/02/01 20:49:09, hanwenn wrote: > > > On 2020/02/01 11:00:41, hahnjo wrote: > > > > > > > > > > https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh > > > > File

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/01 20:59:06, dak wrote: > On 2020/02/01 20:49:09, hanwenn wrote: > > On 2020/02/01 11:00:41, hahnjo wrote: > > > > > > https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh > > > File lily/include/score-engraver.hh (right): > > > > > > > > >

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/01 20:49:09, hanwenn wrote: > On 2020/02/01 11:00:41, hahnjo wrote: > > > https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh > > File lily/include/score-engraver.hh (right): > > > > >

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh File lily/include/score-engraver.hh (right): https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh#newcode33 lily/include/score-engraver.hh:33: GC_word last_gc_count_; On

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

On 2020/02/01 11:00:41, hahnjo wrote: > https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh > File lily/include/score-engraver.hh (right): > > https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh#newcode26 >

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh File lily/include/score-engraver.hh (right): https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh#newcode26 lily/include/score-engraver.hh:26: #include On 2020/02/01

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh File lily/include/score-engraver.hh (right): https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh#newcode26 lily/include/score-engraver.hh:26: #include This effectively adds

Re: Grow heap aggressively during music interpretation (issue 561390043 by hanw...@gmail.com)

https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh File lily/include/score-engraver.hh (right): https://codereview.appspot.com/561390043/diff/557260051/lily/include/score-engraver.hh#newcode33 lily/include/score-engraver.hh:33: GC_word last_gc_count_; FYI: