Re: Garbage collector collects live objects
On Thursday, 11 December 2014 at 18:36:59 UTC, Steven Schveighoffer wrote: My analysis so far: 2. In the array append code, the block attributes are obtained via GC.query, which has this code for getting the attributes: https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L1792 Quoting from that function: // reset the offset to the base pointer, otherwise the bits // are the bits for the pointer, which may be garbage offset = cast(size_t)(info.base - pool.baseAddr); info.attr = getBits(pool, cast(size_t)(offset pool.shiftBy)); Which should get the correct bits. I suspected there was an issue with getting the wrong bits, but this code looks correct. 3. The runtime caches the block info for thread local data for append speed. A potential issue is that the attributes are cached from a previous use for that block, but the GC (and the runtime itself) SHOULD clear that cache entry when that block is freed, avoiding this issue. A potential way to check this is to assert in a debug build of druntime that the cached block info always equals the actual block info. Are you able to build a debug version of druntime to test this? I can give you the changes you should make. This would explain the great difficulty in reproducing the issue. I will try to build debug version of dmd compiler and check the issue. 4. If your code is multi-threaded, but using __gshared, it can make the cache incorrect. Are you doing this? the app is multi-threaded via std.concurrency. there is only one known to me place where __gshared is used: logging library (checked by searching through whole source tree). make stub for this lib and try, so identify whether cache invalidated by _gshared or not. But the cache is really the only possible place I can see where the bits are set incorrectly, given that you just verified the bits are correct before the append. Can you just list the version of the compiler you are using? I want to make sure this isn't an issue that has already been fixed. the last. first of all i updated whole toolchain (dmd, dub). $ dmd DMD64 D Compiler v2.066.1 -Steve I started looking druntime and dmd source code myself before i checked the thread (thsnks for your help and feedback) and i have some questions. could you explain to me something? i_m looking here https://github.com/D-Programming-Language/druntime/blob/v2.066.1/src/rt/lifetime.d#L591 --- line #603 auto size = ti.next.tsize; why `next`? it can be even null if this is last TypeInfo in the linked list. - btw, i used suggested trackallocs.d and GC defenetely receives NO_SCAN before tag: 1 len: 2 ptr: 103A78058 root: 103A77000:8192 attr: APPENDABLE gc_qalloc(41, NO_SCAN APPENDABLE ) cc: 29106 asz: 10152603, ti: null ret: BlkInfo_(104423800, 64, 10) after tag: 1 len: 3 ptr: 104423810 root: 104423800:64 attr: NO_SCAN APPENDABLE
Re: Garbage collector collects live objects
On Friday, 12 December 2014 at 12:53:00 UTC, Ruslan Mullakhmetov wrote: On Thursday, 11 December 2014 at 18:36:59 UTC, Steven Schveighoffer wrote: My analysis so far: 4. If your code is multi-threaded, but using __gshared, it can make the cache incorrect. Are you doing this? the app is multi-threaded via std.concurrency. there is only one known to me place where __gshared is used: logging library (checked by searching through whole source tree). make stub for this lib and try, so identify whether cache invalidated by _gshared or not. removing __gshared seems does not helped.
Re: Garbage collector collects live objects
On Friday, 12 December 2014 at 15:50:26 UTC, Steven Schveighoffer wrote: Can I email you at this address? If not, email me at the address from my post to let me know your contact, no reason to work through building issues on the public forum :) -Steve reach me at theambient [] me__com
Re: Garbage collector collects live objects
On Wednesday, 10 December 2014 at 02:43:19 UTC, ketmar via Digitalmars-d-learn wrote: On Tue, 09 Dec 2014 17:18:44 + Ruslan Mullakhmetov via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: but i still have no clue how to overcome GC =( why do you want to fight with GC? most of the time GC is your friend. see the topic: i got corruption when dereferencing object.
Re: Garbage collector collects live objects
On Tuesday, 9 December 2014 at 21:38:57 UTC, Steven Schveighoffer wrote: On 12/9/14 2:56 PM, Steven Schveighoffer wrote: On 12/9/14 12:40 PM, Ruslan Mullakhmetov wrote: array holds 11 64bit pointers but it's block size is only 128 bytes 11 * 64 = 704 bytes. what's wrong with this arithmetics? Hah, just realized what's wrong. It's not 64 *bytes* per pointer, it's 64 *bits*. So 8 bytes. 11 * 8 == 88. Starting to sound more and more normal... -Steve yes. that was the mistake. also after fixing bug in Blk Attributes printing i got more reasonable attrs for object blk: FINALIZE for array of objects blk: NO_SCAN APPENDABLE this is sound good except for NO_SCAN. I did simple test file in which allocate array of Foo objects (http://dpaste.dzfl.pl/89ab00a897f6) there i see blk attrs only APPENDABLE without NO_SCAN. as far as i understand GC will not scan this array for references and those if the only reference to object is stored in this array will not see it, those assume this object as **not** referenced and collects it, am i right? the other question why this happens... try to debug more.
Re: Garbage collector collects live objects
On Wednesday, 10 December 2014 at 08:46:12 UTC, Ruslan Mullakhmetov wrote: yes. that was the mistake. also after fixing bug in Blk Attributes printing i got more reasonable attrs for object blk: FINALIZE for array of objects blk: NO_SCAN APPENDABLE this is sound good except for NO_SCAN. ... the other question why this happens... try to debug more. I've done more dubugging. what i've found: initially array blk has only attrs APPENDABLE, but after some time this blk is shrinked and reallocated (moved) and then NO_SCAN attr appears. here the output of my extended logs: before tag: 1 len: 2 ptr: 103DD9058 root: 103DD8000:8192 attr: APPENDABLE after tag: 1 len: 3 ptr: 103A21DD0 root: 103A21DC0:64 attr: NO_SCAN APPENDABLE this is produced by the following code http://dpaste.dzfl.pl/0c6dc16270a1 so in a nutshell after appending to array via ~= operator blk attrs changed from APPENDABLE to NO_SCAN APPENDABLE which cause the problem. why and how this happens? can anybody explain it to me?
Re: Garbage collector collects live objects
On Wednesday, 10 December 2014 at 12:52:24 UTC, Ruslan Mullakhmetov wrote: why and how this happens? can anybody explain it to me? I tried to extract this and saw NO NO_SCAN attrs after moving blk: the following piece of output produced by http://dpaste.dzfl.pl/6f773e17de92 len: 6 ptr: 109DF0010 root: 109DF:1048576 attr: APPENDABLE len: 7 ptr: 109DF0010 root: 109DF:1048576 attr: APPENDABLE len: 8 ptr: 109DF0010 root: 109DF:1048576 attr: APPENDABLE len: 9 ptr: 109DF0010 root: 109DF:1048576 attr: APPENDABLE len: 10 ptr: 109DF0010 root: 109DF:1048576 attr: APPENDABLE --- shrinked -- len: 1 ptr: 109EB3508 root: 109DF:1048576 attr: APPENDABLE len: 2 ptr: 109EB3508 root: 109DF:1048576 attr: APPENDABLE len: 3 ptr: 109EB3508 root: 109DF:1048576 attr: APPENDABLE len: 4 ptr: 109EB3508 root: 109DF:1048576 attr: APPENDABLE len: 5 ptr: 109EB3508 root: 109DF:1048576 attr: APPENDABLE len: 6 ptr: 109F60640 root: 109F60640:64 attr: APPENDABLE len: 7 ptr: 109F60640 root: 109F60640:64 attr: APPENDABLE
Re: Garbage collector collects live objects
On Wednesday, 10 December 2014 at 13:00:45 UTC, ketmar via Digitalmars-d-learn wrote: can you give us a minified code that causes this behavior? see previous post. the problem vanish if i try to extract it.
Garbage collector collects live objects
Hi, I experience very strange problem: GC somehow collects live objects. I found it because i got segfaults. After debugging and tracing i found this is because of accessing not allocated memory. I did the following checks: - added to some class invariant check for access to suspicious members with assertion assert(GC.addrOf(cast(void*)x) !is null); where it fails DETERMINISTICALLY at some point - printing address of allocated classes where i observe the following pattern - ctor check check check - dtor check (fails) could anybody advice me with something? I got really frustrated by this strange behaviour which i can not fix right now. key observations: - it is deterministically behaviour (what gets me even more confused cause GC collections as far as i know runs from time to time) - i do not play with pointers optimisation like hiding its in ints or floats. - i operate with large uniformly distributed (video) data in memory where pointer like patterns may occur. but this is not the case cause (1) it brings at worst long living objects (2) input sequence constant but allocated pointers each run different.
Re: Garbage collector collects live objects
On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven Schveighoffer
wrote:
On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:
Hi,
I experience very strange problem: GC somehow collects live
objects.
I found it because i got segfaults. After debugging and
tracing i found
this is because of accessing not allocated memory.
I did the following checks:
- added to some class invariant check for access to suspicious
members
with assertion
assert(GC.addrOf(cast(void*)x) !is null);
where it fails DETERMINISTICALLY at some point
- printing address of allocated classes where i observe the
following
pattern
- ctor
check
check
check
- dtor
check (fails)
could anybody advice me with something? I got really
frustrated by this
strange behaviour which i can not fix right now.
key observations:
- it is deterministically behaviour (what gets me even more
confused
cause GC collections as far as i know runs from time to time)
- i do not play with pointers optimisation like hiding its in
ints or
floats.
- i operate with large uniformly distributed (video) data in
memory
where pointer like patterns may occur. but this is not the
case cause
(1) it brings at worst long living objects (2) input sequence
constant
but allocated pointers each run different.
A random guess, since you haven't posted any code, are you
accessing GC resources inside a destructor? If so, that is not
guaranteed to work. A class destructor, or a destructor of a
struct that is contained inside a class, can only be used to
destroy NON-GC resources.
If you want more help, you need to post some code. Something
that minimally causes the issue would be good.
-Steve
No, there is no accessing GC resources in dtors.
the only usage of dtor in one class is
~this()
{
_file.close();
}
where _file is of type std.file.File
i'll try to extract problem to any observable source code but all
my previous attempts lead to problem being diminish.
Re: Garbage collector collects live objects
On Tuesday, 9 December 2014 at 16:53:02 UTC, Steven Schveighoffer
wrote:
On 12/9/14 11:17 AM, ketmar via Digitalmars-d-learn wrote:
On Tue, 09 Dec 2014 14:52:53 +
Ruslan Mullakhmetov via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:
On Tuesday, 9 December 2014 at 14:23:06 UTC, Steven
Schveighoffer
wrote:
On 12/9/14 8:54 AM, Ruslan Mullakhmetov wrote:
Hi,
I experience very strange problem: GC somehow collects live
objects.
I found it because i got segfaults. After debugging and
tracing i found
this is because of accessing not allocated memory.
I did the following checks:
- added to some class invariant check for access to
suspicious
members
with assertion
assert(GC.addrOf(cast(void*)x) !is null);
where it fails DETERMINISTICALLY at some point
- printing address of allocated classes where i observe the
following
pattern
- ctor
check
check
check
- dtor
check (fails)
could anybody advice me with something? I got really
frustrated by this
strange behaviour which i can not fix right now.
key observations:
- it is deterministically behaviour (what gets me even more
confused
cause GC collections as far as i know runs from time to
time)
- i do not play with pointers optimisation like hiding its
in
ints or
floats.
- i operate with large uniformly distributed (video) data in
memory
where pointer like patterns may occur. but this is not the
case cause
(1) it brings at worst long living objects (2) input
sequence
constant
but allocated pointers each run different.
A random guess, since you haven't posted any code, are you
accessing GC resources inside a destructor? If so, that is
not
guaranteed to work. A class destructor, or a destructor of a
struct that is contained inside a class, can only be used to
destroy NON-GC resources.
If you want more help, you need to post some code. Something
that minimally causes the issue would be good.
-Steve
No, there is no accessing GC resources in dtors.
the only usage of dtor in one class is
~this()
{
_file.close();
}
where _file is of type std.file.File
i'll try to extract problem to any observable source code but
all
my previous attempts lead to problem being diminish.
that file can be already finalized. please remember that
`~this()` is
more a finalizer than destructor, and it's called on *dead*
object.
here this means that any other object in your object (including
structs) can be already finalized at the time GC decides to
call your
finalizer.
File is specially designed (although it's not perfect) to be
able to close in the GC. Its ref-counted payload is placed on
the C heap to allow access during finalization.
That being said, you actually don't need to write the above in
the class finalizer, _file's destructor will automatically be
called.
just avoid destructors unless you *really* need that. in
your case
simply let GC finalize your File, don't try to help GC. this
is not C++
(or any other language without GC) and destructors aren't
destructing
anything at all. destructors must clean up the things that
GC cannot
(malloc()'ed memory, for example), and nothing else.
Good advice ;)
I would say other than library writers, nobody should ever
write a class dtor.
-Steve
thanks, I got it: either C++ or D dtors are minefield =)
but i still have no clue how to overcome GC =(
Re: Garbage collector collects live objects
On Tuesday, 9 December 2014 at 16:13:25 UTC, Dicebot wrote: It may happen if only reference to an object is stored in memory block marked as data-only (using ubyte[] for a buffer is probably most common reason I have encountered) Thanks for interesting hypothesis, but that's not the issue. innocent though collected objects are living in D array MyClass[] which are living in assoc array as value. i checked attributes for GC block holding this array: ``` FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR ``` I really doubting about NO_INTERIOR. can anybody confirm me that is's working with array slicing which i heavily use? also i found that block size is quite small pre array: [100A2FD00, 100A2F700, 100A33B80, 100A33500, 100A3FE80, 100A3F980, 100A3F400, 100A72600, 100A7DF80, 100A7DA80, 100A7D500] array ptr: 100A72580 root: 100A72580:128 attr: FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR [100985A00] keys: [1] as: 1 au: 100A2FD00 [100985A00] keys: [1] as: 1 au: 100A2F700 [100985A00] keys: [1] as: 1 au: 100A33B80 /pre array holds 11 64bit pointers but it's block size is only 128 bytes 11 * 64 = 704 bytes. what's wrong with this arithmetics?
Re: Garbage collector collects live objects
On Tuesday, 9 December 2014 at 19:56:30 UTC, Steven Schveighoffer wrote: On 12/9/14 12:40 PM, Ruslan Mullakhmetov wrote: On Tuesday, 9 December 2014 at 16:13:25 UTC, Dicebot wrote: i checked attributes for GC block holding this array: FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR That does not sound right at all. No block should ever have both FINALIZE (reserved for objects only) and APPENDABLE (reserved for arrays only). also i found that block size is quite small pre array: [100A2FD00, 100A2F700, 100A33B80, 100A33500, 100A3FE80, 100A3F980, 100A3F400, 100A72600, 100A7DF80, 100A7DA80, 100A7D500] array ptr: 100A72580 root: 100A72580:128 attr: FINALIZE NO_SCAN NO_MOVE APPENDABLE NO_INTERIOR [100985A00] keys: [1] as: 1 au: 100A2FD00 [100985A00] keys: [1] as: 1 au: 100A2F700 [100985A00] keys: [1] as: 1 au: 100A33B80 /pre array holds 11 64bit pointers but it's block size is only 128 bytes 11 * 64 = 704 bytes. what's wrong with this arithmetics? I think there is something you are missing, or something is very corrupt. Can you show the code that prints this? -Steve here the piece of code i used to output this value http://pastebin.com/cQf9Nghp StreamIndex is ubyte AccessUnit is some class
