Re: want to know precise GC benchmarks

2019-10-02 Thread a11e99z via Digitalmars-d-learn
On Wednesday, 2 October 2019 at 06:41:28 UTC, Rainer Schuetze 
wrote:




thanks for the detailed answer



Re: want to know precise GC benchmarks

2019-10-02 Thread Rainer Schuetze via Digitalmars-d-learn



On 01/10/2019 18:24, a11e99z wrote:
> On Tuesday, 1 October 2019 at 16:12:18 UTC, a11e99z wrote:
>> does anybody some kind of benchmark to test conservative and precise GC?
>> precise GC is better or not? is STW improving?

Without false pointers the precise GC is usually a bit slower (by a few
%) due to additional work being done during allocations. But it can be a
lot faster if there are false pointers that pin large amounts of memory
still needed to be scanned during collections. False pointers are more
likely for 32-bit processes, but can also happen with 64-bit processes
(also depending on addresses used by OS allocations: OSX worse than
Windows worse than Linux).

> 
> and another question about GC and app parameters:
>> program.exe “–DRT-gcopt=gc:precise parallel:4”
>> “–DRT-scanDataSeg=precise” output.data
> are this 2 -DRT params combined or overwriting each other?
> link to doc for DRT+GC https://dlang.org/spec/garbage.html#gc_config

These options are independent and can be used in arbitrary order. The
last option wins if you actually overwrite an option, e.g.
'“–DRT-gcopt=gc:precise parallel:4” “–DRT-gcopt=parallel:7”' will still
use the precise GC, but 7 mark threads.

Please note that “–DRT-scanDataSeg=precise” is only supported on Windows.

> 
> I know about rt_options[] but asking about program args
> 
> why I want to know such info?
> CodinGame sometimes use time-limit for bot move for example 100ms, and
> bot will be disqualified in case no answer

There is no actual upper limit for the collection time, it mostly
depends on how much life memory has to be scanned.


Re: want to know precise GC benchmarks

2019-10-01 Thread Mike Parker via Digitalmars-d-learn

On Tuesday, 1 October 2019 at 16:24:49 UTC, a11e99z wrote:



why I want to know such info?
CodinGame sometimes use time-limit for bot move for example 
100ms, and bot will be disqualified in case no answer


Simple solution: don't allocate every frame. The GC only runs 
when it needs to and it only needs to if an allocation request 
takes it over a certain threshold. If you need to allocate at 
all, do it up front. Or better, do it statically. Then the GC 
won't run at all.


What sort of CodinGame bot would you need to allocate from the GC 
for anyway?


Re: want to know precise GC benchmarks

2019-10-01 Thread a11e99z via Digitalmars-d-learn

On Tuesday, 1 October 2019 at 16:12:18 UTC, a11e99z wrote:
does anybody some kind of benchmark to test conservative and 
precise GC?

precise GC is better or not? is STW improving?


and another question about GC and app parameters:
program.exe “–DRT-gcopt=gc:precise parallel:4” 
“–DRT-scanDataSeg=precise” output.data

are this 2 -DRT params combined or overwriting each other?
link to doc for DRT+GC 
https://dlang.org/spec/garbage.html#gc_config


I know about rt_options[] but asking about program args

why I want to know such info?
CodinGame sometimes use time-limit for bot move for example 
100ms, and bot will be disqualified in case no answer


want to know precise GC benchmarks

2019-10-01 Thread a11e99z via Digitalmars-d-learn
does anybody some kind of benchmark to test conservative and 
precise GC?

precise GC is better or not? is STW improving?


Re: Using D's precise GC when running an app with DUB

2019-05-24 Thread Per Nordlöw via Digitalmars-d-learn

On Thursday, 23 May 2019 at 15:25:31 UTC, Per Nordlöw wrote:

You mean wise versa, right?


Nevermind that comment. No "wise versa". You're answer is 
correct, rikki cattermole.


Thanks


Re: Using D's precise GC when running an app with DUB

2019-05-23 Thread Eugene Wissner via Digitalmars-d-learn

On Thursday, 23 May 2019 at 14:50:12 UTC, Per Nordlöw wrote:

How do I specify a druntime flag such as

--DRT-gcopt=gc:precise

when running with dub as

dub run --compiler=dmd --build=unittest

?

The precise GC flag was introduced in verison 2.085.0

See:
- https://dlang.org/changelog/2.085.0.html#gc_precise
- https://dlang.org/spec/garbage.html#precise_gc


You can put into the source:

extern(C) __gshared string[] rt_options = [
"gcopt=gc:precise"
];

you can wrap it into some "version ()" and set the version in the 
dub configuration.


Re: Using D's precise GC when running an app with DUB

2019-05-23 Thread Per Nordlöw via Digitalmars-d-learn

On Thursday, 23 May 2019 at 15:05:15 UTC, rikki cattermole wrote:

Should be as easy as

     dflags "--DRT-gcopt=gc:precise"

right?


That would be passed to dmd, not to the build executable upon 
running.


You mean wise versa, right?

Now I understand,

--DRT-gcopt=gc:precise

is passed to the resulting binary...


Re: Using D's precise GC when running an app with DUB

2019-05-23 Thread rikki cattermole via Digitalmars-d-learn

On 24/05/2019 3:03 AM, Per Nordlöw wrote:

On Thursday, 23 May 2019 at 15:02:12 UTC, rikki cattermole wrote:

And if I want to set this in a dub.sdl?


No can do. There is meant to be a way to set it in D however.
But I have heard mixed results (not that I've tried it).


Should be as easy as

     dflags "--DRT-gcopt=gc:precise"

right?


That would be passed to dmd, not to the build executable upon running.


Re: Using D's precise GC when running an app with DUB

2019-05-23 Thread rikki cattermole via Digitalmars-d-learn

On 24/05/2019 2:58 AM, Per Nordlöw wrote:

On Thursday, 23 May 2019 at 14:51:41 UTC, rikki cattermole wrote:

dub run --compiler=dmd --build=unittest -- --DRT-gcopt=gc:precise


Thanks!

And if I want to set this in a dub.sdl?


No can do. There is meant to be a way to set it in D however.
But I have heard mixed results (not that I've tried it).


Re: Using D's precise GC when running an app with DUB

2019-05-23 Thread Per Nordlöw via Digitalmars-d-learn

On Thursday, 23 May 2019 at 15:02:12 UTC, rikki cattermole wrote:

And if I want to set this in a dub.sdl?


No can do. There is meant to be a way to set it in D however.
But I have heard mixed results (not that I've tried it).


Should be as easy as

dflags "--DRT-gcopt=gc:precise"

right?


Re: Using D's precise GC when running an app with DUB

2019-05-23 Thread rikki cattermole via Digitalmars-d-learn

On 24/05/2019 3:01 AM, Per Nordlöw wrote:

On Thursday, 23 May 2019 at 14:51:41 UTC, rikki cattermole wrote:

dub run --compiler=dmd --build=unittest -- --DRT-gcopt=gc:precise


Hmm, the flag doesn't propagate to dmd when compiling in verbose mode 
via -v as


     dub run -v --compiler=dmd --build=unittest -- --DRT-gcopt=gc:precise

Shouldn't it?


No. You told dub to -v.

DFLAGS="-v" dub run --compiler=dmd --build=unittest -- 
--DRT-gcopt=gc:precise




Re: Using D's precise GC when running an app with DUB

2019-05-23 Thread Per Nordlöw via Digitalmars-d-learn

On Thursday, 23 May 2019 at 14:51:41 UTC, rikki cattermole wrote:
dub run --compiler=dmd --build=unittest -- 
--DRT-gcopt=gc:precise


Hmm, the flag doesn't propagate to dmd when compiling in verbose 
mode via -v as


dub run -v --compiler=dmd --build=unittest -- 
--DRT-gcopt=gc:precise


Shouldn't it?



Re: Using D's precise GC when running an app with DUB

2019-05-23 Thread Per Nordlöw via Digitalmars-d-learn

On Thursday, 23 May 2019 at 14:51:41 UTC, rikki cattermole wrote:
dub run --compiler=dmd --build=unittest -- 
--DRT-gcopt=gc:precise


Thanks!

And if I want to set this in a dub.sdl?


Using D's precise GC when running an app with DUB

2019-05-23 Thread Per Nordlöw via Digitalmars-d-learn

How do I specify a druntime flag such as

--DRT-gcopt=gc:precise

when running with dub as

dub run --compiler=dmd --build=unittest

?

The precise GC flag was introduced in verison 2.085.0

See:
- https://dlang.org/changelog/2.085.0.html#gc_precise
- https://dlang.org/spec/garbage.html#precise_gc


Re: Using D's precise GC when running an app with DUB

2019-05-23 Thread rikki cattermole via Digitalmars-d-learn

On 24/05/2019 2:50 AM, Per Nordlöw wrote:

How do I specify a druntime flag such as

     --DRT-gcopt=gc:precise

when running with dub as

     dub run --compiler=dmd --build=unittest


dub run --compiler=dmd --build=unittest -- --DRT-gcopt=gc:precise



Re: precise GC

2019-03-05 Thread Rainer Schuetze via Digitalmars-d-learn



On 05/03/2019 22:30, H. S. Teoh wrote:
> On Tue, Mar 05, 2019 at 09:50:34PM +0100, Rainer Schuetze via 
> Digitalmars-d-learn wrote:
>> On 04/03/2019 12:12, KnightMare wrote:
> [...]
>>> 3) closures: do the closures have any internal types that helps to
>>> GC or are they (full closure memory block) scanned as in the
>>> conservative mode?
>>
>> No type information is generated for closures by the compiler, so
>> these are always scanned conservatively.
> [...]
> 
> Just out of curiosity, how hard would it be for the compiler to emit
> type information for closures?  Given the prevalence of the range-based
> idiom in D, I'd think this is a worthwhile area for GC improvements.

I tried that first when I added debug information for closures on
Windows recently, but it didn't easily work out. I suspect it cannot be
generated early in the front-end as closures might also change due to
inlining and optimizations.

Maybe even worse than the conservative scanning: if structs are moved
into the closure, their destructors are never called, even if the
closure is collected.


Re: precise GC

2019-03-05 Thread H. S. Teoh via Digitalmars-d-learn
On Tue, Mar 05, 2019 at 09:50:34PM +0100, Rainer Schuetze via 
Digitalmars-d-learn wrote:
> On 04/03/2019 12:12, KnightMare wrote:
[...]
> > 3) closures: do the closures have any internal types that helps to
> > GC or are they (full closure memory block) scanned as in the
> > conservative mode?
> 
> No type information is generated for closures by the compiler, so
> these are always scanned conservatively.
[...]

Just out of curiosity, how hard would it be for the compiler to emit
type information for closures?  Given the prevalence of the range-based
idiom in D, I'd think this is a worthwhile area for GC improvements.


T

-- 
One Word to write them all, One Access to find them, One Excel to count them 
all, And thus to Windows bind them. -- Mike Champion


Re: precise GC

2019-03-05 Thread Rainer Schuetze via Digitalmars-d-learn



On 04/03/2019 12:12, KnightMare wrote:
> For example, we have some rooted memory block as
> auto rooted = new long[1_000_000];
> 1) conservative-GC will scan it for false pointers every GC-cycle. is it
> true?
> 2) precise-GC will NOT scan it at all. is it true?

As Adam pointed out, this memory block is neither scanned by the default
GC nor the precise GC because they don't contain any references.

> 
> 3) closures: do the closures have any internal types that helps to GC or
> are they (full closure memory block) scanned as in the conservative mode?

No type information is generated for closures by the compiler, so these
are always scanned conservatively.

> 
> 4) associative arrays:
> SomeTypeWithRefsToClasses[string]
> any pair will be allocated at some memory block [hash, key, value] as I
> imagine.
> Will be precise-GC scan at every pair block only some fields of
> SomeTypeWithRefsToClasses or full [pair-block]?
> will be scanned string-key memory block: span-struct and\or chars data?

associative arrays use allocations containing both key and value. These
are scanned as if they are combined to a struct, so only actual pointers
with the precise GC, semi-precisely (as Adam put it) with the default GC.


Re: precise GC

2019-03-04 Thread Adam D. Ruppe via Digitalmars-d-learn

On Monday, 4 March 2019 at 10:38:29 UTC, KnightMare wrote:

For example, we have some rooted memory block as
auto rooted = new long[1_000_000];
1) conservative-GC will scan it for false pointers every 
GC-cycle. is it true?


Well, conservative GC in general might, but D's GC would NOT.

D's old GC is arguably semi-precise. It would scan void[] 
conservatively, anything that looks like a pointer is considered 
maybe a pointer. But it would NOT scan blocks allocated as long[] 
or ubyte[]. It would allocate those as NO_SCAN blocks.


The difference is in combination things, like the stack or 
structs with static blocks.


struct {
   int a;
   void* b;
}

The old GC would treat that whole struct as potentially pointers, 
both a and b. The new precise GC would know only b needs to be 
scanned inside that struct.


The even bigger deal with precise is it also knows only b would 
need to be changed if the pointer were to move - that's the big 
gain precise is setting the groundwork for, to enable moving GC 
optimizations.


Re: precise GC

2019-03-04 Thread KnightMare via Digitalmars-d-learn
IMO need more explanations about precise-GC and cases where 
behavior of precise and conservative same and differs


Re: precise GC

2019-03-04 Thread KnightMare via Digitalmars-d-learn
/* English is not my native, and I tried to use Google translate. 
I hope u will understand subtleties of questions */


For precise-GC:

3) closures: do the closures have any internal types that helps 
to GC or are they (full closure memory block) scanned as in the 
conservative mode?


4) associative arrays:
SomeTypeWithRefsToClasses[string]
any pair will be allocated at some memory block [hash, key, 
value] as I imagine.
Will be precise-GC scan at every pair block only some fields of 
SomeTypeWithRefsToClasses or full [pair-block]?
will be scanned string-key memory block: span-struct and\or chars 
data?


precise GC

2019-03-04 Thread KnightMare via Digitalmars-d-learn
As I understood conservative-GC scans all allocated memory blocks 
for false pointers. In other hand precise-GC scans only explicit 
memory blocks that contains (objects of types that contains) 
pointers/refs or "muddy" types (void, void[]...).


For example, we have some rooted memory block as
auto rooted = new long[1_000_000];
1) conservative-GC will scan it for false pointers every 
GC-cycle. is it true?

2) precise-GC will NOT scan it at all. is it true?


Re: Regarding the more precise GC

2012-04-22 Thread Dmitry Olshansky

On 22.04.2012 4:28, bearophile wrote:

In the main D newsgroup I have seen the two recent threads regarding a
more precise GC in D. I have two questions about that, that seem more
fit for D.learn.

1) I have not fully understood the performance and memory implications
of the more precise GC. In the thread I think I've seen a memory
overhead in every allocated struct. Is it possible to disable this
bookkeeping/overhead for smaller programs that enjoy/need a GC but
probably don't need a precise GC because they run only for no more than
30 seconds? Many of my small D programs are command-line utilities with
a short run-time, but they often need a GC.

AFAIKT there is overhead per user defined type. In typical cases or no 
indirections or structs/classes of length  32*size_t.sizeof (64 on 
64bit) fit within one word per struct/class.


The above assumes it will use simple bitmask where applicable, there are 
better and more compact ways still.


Most likely + another pointer per entity type for a hook in TypeInfo 
inserted by compiler.


So, on average it's about 2 * num_of_abstractions* pointer.sizeof + 
small_extra




2) If I have a tagged union, like:


static bool isPointer;

union Foo {
size_t count;
int* ptr;
}

In every point of my program I know that inside a Foo there is a pointer
or a size_t according to the value isPointer (a similar case is if I use
a single bit tagging inside a size_t to denote if it's a pointer or an
integral value. The D docs say that in the current conservative GC
design such pointer tagging with a single bit is not allowed if the
pointer is to CG-managed memory, but the union is allowed. Maybe with
the more precise GC even the pointer tagging gets possible).

Is it possible to tell to the precise GC every time it performs a
collection if a Foo contains a pointer to follow, or if it instead
contains just an integral value to ignore? I think it needs to be a
function pointer or some kind of callback.


It can be the case that the cost of calling per instance callbacks just 
too high so being conservative with them is preferable.


But otherwise  I think that currently proposed scheme allows it (as long 
as the collector supports this - the meta info has to be in sync with 
GC). Namely a type provides a callback that returns a bitmask when 
called with its memory block as parameter by GC.


Yet I believe GC per type callbacks were presently assumed to be 
agnostic to the actual content within the memory block. It can be 
extended from here if deemed useful.




Bye and thank you,
bearophile



--
Dmitry Olshansky