Re: Memory allocation failed. Why?
On Tuesday, 22 November 2016 at 15:53:39 UTC, Steven Schveighoffer wrote: On 11/21/16 11:53 AM, ag0aep6g wrote: Thank you very much for explaining such a difficult and slippery situation.
Re: Memory allocation failed. Why?
On 11/21/16 11:53 AM, ag0aep6g wrote: On Monday, 21 November 2016 at 16:37:32 UTC, Kagamin wrote: Anything in .data and .bss sections and stack. See https://issues.dlang.org/show_bug.cgi?id=15723 Ok, not an actual reference then, but a false pointer. Yes. 100 million bytes is 1/40 of all addressable space on 32-bits. There only needs to be one 4-byte segment somewhere on the stack that points at this, and it won't be collected. Assuming you have a quite large segment that can't be extended or collected (due to false pointer), this means you have to allocate another large one to satisfy the next allocation (which then could be pinned). And it gets worse from there. -Steve
Re: Memory allocation failed. Why?
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote: core.exception.OutOfMemoryError@src\core\exception.d(693): Memory allocation failed Simple program and error. Why? Windows 7 (32) dmd 2.072.0 Making a 100 million bytes array by appending one byte at a time creates a lot of intermediate-size arrays. Ideally these should be garbage collected away but GC in D is not only slow but also leaky. In 32 bits if you have 1000 random int values on the stack or data segment, with uniform distribution, this is 1000 random locations in memory pinned and seen by GC as live, i.e. one per 4 MB of address space. Which means if your array is 4 MB or larger it's almost doomed to be never collected by GC in this scenario. Your program creates a lot of large arrays and they don't get collected because of false pointers and not precise enough GC. Moral of the story: in 32 bits don't allocate anything big (1 MB or more) in GC heap, otherwise there are good chances it will create a memory leak. Use std.container.array or something similar.
Re: Memory allocation failed. Why?
On Monday, 21 November 2016 at 16:37:32 UTC, Kagamin wrote: Anything in .data and .bss sections and stack. See https://issues.dlang.org/show_bug.cgi?id=15723 Ok, not an actual reference then, but a false pointer.
Re: Memory allocation failed. Why?
On Monday, 21 November 2016 at 11:22:40 UTC, ag0aep6g wrote: Who could "someone" be? It's a self-contained example, and buf doesn't leave the test function. Anything in .data and .bss sections and stack. See https://issues.dlang.org/show_bug.cgi?id=15723 As for GC compaction: https://issues.dlang.org/show_bug.cgi?id=3284
Re: Memory allocation failed. Why?
On 11/21/2016 08:27 AM, Stefan Koch wrote: Someone could still be hanging on to an old Reference of buf. Who could "someone" be? It's a self-contained example, and buf doesn't leave the test function.
Re: Memory allocation failed. Why?
On Monday, 21 November 2016 at 06:45:04 UTC, ag0aep6g wrote: On 11/21/2016 07:36 AM, Stefan Koch wrote: On Monday, 21 November 2016 at 03:58:00 UTC, MGW wrote: On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote: On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote: [...] [...] You are appending 100_000_000 Times. Note that this will take up much more then 100 MB. Because the allocator cannot always grow the array. All previously allocated blocks are still in scope and could be reached. Therefore the memory is not released. How could they be reached? The only reference is `buf', and when appending triggers relocation, `buf` should point to the new location, no? Someone could still be hanging on to an old Reference of buf.
Re: Memory allocation failed. Why?
On 11/21/2016 07:36 AM, Stefan Koch wrote:
On Monday, 21 November 2016 at 03:58:00 UTC, MGW wrote:
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
[...]
You are appending 100_000_000 Times.
Note that this will take up much more then 100 MB.
Because the allocator cannot always grow the array.
All previously allocated blocks are still in scope and could be reached.
Therefore the memory is not released.
How could they be reached? The only reference is `buf', and when
appending triggers relocation, `buf` should point to the new location, no?
Re: Memory allocation failed. Why?
On Monday, 21 November 2016 at 03:58:00 UTC, MGW wrote:
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
core.exception.OutOfMemoryError@src\core\exception.d(693):
Memory allocation failed
This short program doesn't work at Windows 32. Why GC doesn't
release memory in case of its shortage in Windows! This is bug?
You are appending 100_000_000 Times.
Note that this will take up much more then 100 MB.
Because the allocator cannot always grow the array.
All previously allocated blocks are still in scope and could be
reached.
Therefore the memory is not released.
At least this is what I think happens.
it would help more if you could post the specs of your machine
and what else is running on it.
Re: Memory allocation failed. Why?
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
core.exception.OutOfMemoryError@src\core\exception.d(693):
Memory allocation failed
This short program doesn't work at Windows 32. Why GC doesn't
release memory in case of its shortage in Windows! This is bug?
Re: Memory allocation failed. Why?
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote: On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote: [...] For me there's no exception. Maybe the GC is poluted. Try to add this after each iteration in the first test loop: import core.memory: GC; GC.collect(); Also note that the text you pas in the message box should be null terminated, eg: MessageBoxA(null, "--The end--\0".ptr, "".ptr, 0); string literals are implicitly null terminated explicitly for the interoperation with C. there is no need.
Re: Memory allocation failed. Why?
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
core.exception.OutOfMemoryError@src\core\exception.d(693):
Memory allocation failed
Simple program and error. Why? Windows 7 (32) dmd 2.072.0
For me there's no exception. Maybe the GC is poluted. Try to add
this after each iteration in the first test loop:
import core.memory: GC;
GC.collect();
Also note that the text you pas in the message box should be null
terminated, eg:
MessageBoxA(null, "--The end--\0".ptr, "".ptr, 0);
Memory allocation failed. Why?
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
core.exception.OutOfMemoryError@src\core\exception.d(693): Memory
allocation failed
Simple program and error. Why? Windows 7 (32) dmd 2.072.0
