Re: Invalid memory operation during allocation with `new`

[Per Nordlöw via Digitalmars-d-learn]

On Monday, 13 January 2020 at 23:06:01 UTC, Per Nordlöw wrote:
Unfortunately that didn't tell me the cause of the problem but 
only the stack trace of the failing call to `new` which is not 
called inside a destructor.


I found the problem. I was greedly using the `__monitor__` slot 
of some classes in order to get one extra word of class storage. 
Setting it to `null` in those classes' destructors made the 
memory error go away. Thanks anyway for valuable input.

Re: Invalid memory operation during allocation with `new`

[Steven Schveighoffer via Digitalmars-d-learn]

On 1/13/20 6:06 PM, Per Nordlöw wrote:

On Monday, 13 January 2020 at 20:30:33 UTC, Adam D. Ruppe wrote:

Reading symbols from pain...
(gdb) break onInvalidMemoryOperationError
Breakpoint 1 at 0x4614f8
(gdb) r
Starting program: /home/me/test/pain
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".


Thanks a lot.

Unfortunately that didn't tell me the cause of the problem but only the 
stack trace of the failing call to `new` which is not called inside a 
destructor.


can you post the stack trace?

-Steve

Re: Invalid memory operation during allocation with `new`

[Per Nordlöw via Digitalmars-d-learn]

On Monday, 13 January 2020 at 20:30:33 UTC, Adam D. Ruppe wrote:

Reading symbols from pain...
(gdb) break onInvalidMemoryOperationError
Breakpoint 1 at 0x4614f8
(gdb) r
Starting program: /home/me/test/pain
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".


Thanks a lot.

Unfortunately that didn't tell me the cause of the problem but 
only the stack trace of the failing call to `new` which is not 
called inside a destructor.

Re: Invalid memory operation during allocation with `new`

[Adam D. Ruppe via Digitalmars-d-learn]

On Monday, 13 January 2020 at 09:34:42 UTC, Per Nordlöw wrote:
Without finding any potential problems. Any other suggestions 
on how to trace this problem?


Run the program in a debugger with the `--DRT-trapExceptions=0` 
argument to the program with certain versions of druntime, or set 
a breakpoint at that function.


On Linux for example, I made a program "pain" that deliberately 
crashes:


-

$ gdb pain
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-slackware-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from pain...
(gdb) break onInvalidMemoryOperationError
Breakpoint 1 at 0x4614f8
(gdb) r
Starting program: /home/me/test/pain
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, 0x004614f8 in onInvalidMemoryOperationError 
()

(gdb) where
#0  0x004614f8 in onInvalidMemoryOperationError ()
#1  0x0045da4b in 
_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs12mallocNoSyncMFNbmkKmxC8TypeInfoZPvS_DQEgQEgQEeQEp10mallocTimelS_DQFiQFiQFgQFr10numMallocslTmTkTmTxQCzZQFcMFNbKmKkKmKxQDsZQDl ()
#2  0x00455a6e in 
_D2gc4impl12conservativeQw14ConservativeGC6mallocMFNbmkxC8TypeInfoZPv ()

#3  0x0043546b in gc_malloc ()
#4  0x0043160b in _d_newclass ()
#5  0x0042f65c in pain.Foo.~this() (this=0x77af1000) 
at pain.d:3

#6  0x0043671c in rt_finalize2 ()
#7  0x00460647 in rt_finalizeFromGC ()
#8  0x00459790 in 
_D2gc4impl12conservativeQw3Gcx5sweepMFNbZm ()
#9  0x0045a202 in 
_D2gc4impl12conservativeQw3Gcx11fullcollectMFNbbZm ()
#10 0x0045e408 in 
_D2gc4impl12conservativeQw14ConservativeGC__T9runLockedS_DQCeQCeQCcQCnQBs18fullCollectNoStackMFNbZ2goFNbPSQEaQEaQDyQEj3GcxZmTQvZQDfMFNbKQBgZm ()
#11 0x00456f38 in 
_D2gc4impl12conservativeQw14ConservativeGC18fullCollectNoStackMFNbZv ()
#12 0x00456ed6 in 
_D2gc4impl12conservativeQw14ConservativeGC14collectNoStackMFNbZv 
()

#13 0x00446e21 in gc_term ()
#14 0x00435bc3 in rt_term ()
#15 0x00430882 in 
_D2rt6dmain212_d_run_main2UAAamPUQgZiZ6runAllMFZv ()
#16 0x00430769 in 
_D2rt6dmain212_d_run_main2UAAamPUQgZiZ7tryExecMFMDFZvZv ()

#17 0x004306d2 in _d_run_main2 ()
#18 0x0043048e in _d_run_main ()
#19 0x0042f6a2 in main ()
#20 0x77c19e5b in __libc_start_main () from 
/lib64/libc.so.6

(gdb)

-


So, first make a debug build of the program (dmd -g pain.d). 
Then, run the program in the debugger:


gdb ./pain

And then ask the debugger to break on this invalid operation:

break onInvalidMemoryOperationError

NOTE: the onInvalidMemoryOperationError is actually a function 
time inside druntime.


Then run the program with gdb's `r` command.

It breaks when that function is called, right before it actually 
would throw


Breakpoint 1, 0x004614f8 in onInvalidMemoryOperationError 
()


So then I asked the debugger where this occurred:

(gdb) where

And visually scanned for a function I recognized (and my gdb also 
conveniently highlighted the filename in green)


#5  0x0042f65c in pain.Foo.~this() (this=0x77af1000) 
at pain.d:3


That looks like my code! pain.d line 3 caused the problem.

class Foo {
~this() {
auto a = new Object(); // this
}
}

void main() {
auto foo = new Foo();
}


It might not be this simple in a real bug situation - this 
program was specifically written to cause it - but the same 
approach should narrow it down pretty quickly for you.

Re: Invalid memory operation during allocation with `new`

[Steven Schveighoffer via Digitalmars-d-learn]

On 1/13/20 4:34 AM, Per Nordlöw wrote:

On Sunday, 12 January 2020 at 17:36:23 UTC, Adam D. Ruppe wrote:
Of course that doesn't help the case when you don't know the rule and 
don't even think to call it...


Thanks. I checked and @nogc-qualified all the class destructors defined 
in my project. Without finding any potential problems. Any other 
suggestions on how to trace this problem?


Finding where this is happening is a really difficult problem. Not only 
because there is no stack trace (I believe this is because the invalid 
memory operation error cannot allocate any GC memory, and there are very 
small parts of the stack trace printing code that allocate), but because 
it is invariably happening inside a GC collection -- so the error is far 
away from where you allocated (sometimes the conditions that trigger the 
dtor to allocate are hard to figure out from looking at the object when 
it's destroyed).


I actually had a problem in my vibe.d code where the error didn't 
display until the system was shutting down. Figuring out where it was 
involved defining the extern(C) function onInvalidMemoryOperationError 
(I think if you define this in your code, it overrides the runtime 
library's version), so I could do some diagnostics, and set a breakpoint.


Try starting there and see if you can find why the IMO is happening.

https://github.com/dlang/druntime/blob/c85dea29cb721ec76af6793076aa93e2f62a23da/src/core/exception.d#L535

-Steve

Re: Invalid memory operation during allocation with `new`

[Per Nordlöw via Digitalmars-d-learn]

On Sunday, 12 January 2020 at 17:36:23 UTC, Adam D. Ruppe wrote:
Of course that doesn't help the case when you don't know the 
rule and don't even think to call it...


Thanks. I checked and @nogc-qualified all the class destructors 
defined in my project. Without finding any potential problems. 
Any other suggestions on how to trace this problem?

Re: Invalid memory operation during allocation with `new`

[Adam D. Ruppe via Digitalmars-d-learn]

On Sunday, 12 January 2020 at 15:21:05 UTC, Per Nordlöw wrote:

- in the short run,
  Qualifying all user-defined class destructors with `@nogc`?

- in the long run,
  Making DMD forbid GC-allocations transitively inside class 
destructors?


The class doesn't necessarily know what its destructor is going 
to be used for, nor does as struct. So such static checks would 
end problematic; limiting to some legit cases and not catching 
some problem cases (because destructors are called with child 
classes to).


The most recent release added a runtime function you can test 
though:


https://dlang.org/phobos/core_memory.html#.GC.inFinalizer

so if that is true, avoid doing the call.

Of course that doesn't help the case when you don't know the rule 
and don't even think to call it...

Re: Invalid memory operation during allocation with `new`

[Per Nordlöw via Digitalmars-d-learn]

On Sunday, 12 January 2020 at 15:21:05 UTC, Per Nordlöw wrote:

Thanks. So what about,

- in the short run,
  Qualifying all user-defined class destructors with `@nogc`?

- in the long run,
  Making DMD forbid GC-allocations transitively inside class 
destructors?


AFAICT, these rules should in most cases hold also for struct 
destructors as those might be called inside class destructors, 
transitively.

Re: Invalid memory operation during allocation with `new`

[Per Nordlöw via Digitalmars-d-learn]

On Sunday, 12 January 2020 at 13:58:25 UTC, Adam D. Ruppe wrote:
Check all the classes created by those unittests. If any of 
them have destructors that allocate memory in any way - 
including calling like `writeln(this)` cuz that can call 
toString, change that.


Thanks. So what about,

- in the short run,
  Qualifying all user-defined class destructors with `@nogc`?

- in the long run,
  Making DMD forbid GC-allocations transitively inside class 
destructors?

Re: Invalid memory operation during allocation with `new`

[Adam D. Ruppe via Digitalmars-d-learn]

On Sunday, 12 January 2020 at 13:39:42 UTC, Per Nordlöw wrote:

core.exception.InvalidMemoryOperationError@src/core/exception.d(647): Invalid 
memory operation


That means a destructor tried to perform a GC operation while the 
GC was running.


The crash happens only if the `unittests` are run prior to 
`main`.


Check all the classes created by those unittests. If any of them 
have destructors that allocate memory in any way - including 
calling like `writeln(this)` cuz that can call toString, change 
that.

Invalid memory operation during allocation with `new`

[Per Nordlöw via Digitalmars-d-learn]

My application has suddendly started crashing as

core.exception.InvalidMemoryOperationError@src/core/exception.d(647): Invalid 
memory operation

during a plain allocation with `new` during execution of `main`.

The crash happens only if the `unittests` are run prior to `main`.

The crash goes away when I disable at least on the `unittest`s 
run before `main`.


I'm compiling with DMD version 2.090.

How do I check the amount of memory currently being used in a 
process?


If the amount of memory used is not the problem, how can I find 
the reason for this?