Re: "Memory allocation failed" on big array
Big thank. Switch to /bin64 and now works until full memory using.
Re: "Memory allocation failed" on big array
My guess is you are compiling to 32bit and the GC tries to reserve >4gb wich it can't, therefore out of memory Compiling to 64bit with: `dmd -m64 -run test.d` works no problem
"Memory allocation failed" on big array
Hello.
having about 6 Gb free RAM, trying to execute:
```d
string[] dic;
for (int i = 0; i < 100_000_000; i++)
{
dic ~= "just a word number "~to!string(i);
if (i%1_000_000 == 0)
{
writef("processed line %s: %s\n", format("%,3d", i),
format("%,3d", dic.length));
}
}
```
got an error:
"processed line 32,000,000: 32,000,001
core.exception.OutOfMemoryError@src\core\lifetime.d(126): Memory
allocation failed"
app used is about 1.6 Gb, more than 4 stays free.
1. What is happened, why dynamic array have mem limit? For
example, golang uses all free memory that it can.
2. How to fix this?
3. (optional) How to display memory usage? Cannot find function
for this.
Thanks.
Re: 2D arrays, slices and manual memory allocation
On Friday, 25 February 2022 at 06:13:35 UTC, 9il wrote: On Friday, 25 February 2022 at 06:03:34 UTC, Nonobvious wrote: From [Go Your Own Way (Part Two: The Heap)](https://dlang.org/blog/2017/09/25/go-your-own-way-part-two-the-heap/): [...] http://mir-algorithm.libmir.org/mir_ndslice_allocation.html#.stdcUninitSlice Thank you.
Re: 2D arrays, slices and manual memory allocation
On Friday, 25 February 2022 at 06:03:34 UTC, Nonobvious wrote: From [Go Your Own Way (Part Two: The Heap)](https://dlang.org/blog/2017/09/25/go-your-own-way-part-two-the-heap/): [...] http://mir-algorithm.libmir.org/mir_ndslice_allocation.html#.stdcUninitSlice
2D arrays, slices and manual memory allocation
From [Go Your Own Way (Part Two: The
Heap)](https://dlang.org/blog/2017/09/25/go-your-own-way-part-two-the-heap/):
`import core.stdc.stdlib;`
`// Allocate a block of untyped bytes that can be managed`
`// as a slice.`
`void[] allocate(size_t size)`
`{`
`// malloc(0) is implementation defined (might return
null `
`// or an address), but is almost certainly not what we
want.`
`assert(size != 0);`
`void* ptr = malloc(size);`
`if(!ptr) assert(0, "Out of memory!");`
`// Return a slice of the pointer so that the address is
coupled`
`// with the size of the memory block.`
`return ptr[0 .. size];`
}
`T[] allocArray(T)(size_t count) `
`{ `
`// Make sure to account for the size of the`
`// array element type!`
`return cast(T[])allocate(T.sizeof * count); `
`}`
What is the equivalent for higher dimensional arrays?
Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 13:43:36 UTC, Steven Schveighoffer wrote: On 10/26/21 2:32 AM, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! You guys aren't getting it: ``` ubyte[] expand(ArchiveMember de); Decompress the contents of a member. Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[]. ``` Where is it supposed to store that `ubyte[]`? -Steve It's not supposed, but a new implementation can utilize something like a stream.
Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 20:33:17 UTC, Steven Schveighoffer wrote: On 10/26/21 1:38 PM, Imperatorn wrote: That's the current implementation. No, that's the API. You cannot fix the implementation with that API and not end up allocating an array to hold the entire unzipped contents. You can't even decompress to a file, and then mmap those contents -- the address space isn't there. I don't know about *nix, but my Windows machine can easily extract a file bigger than my RAM. zlib does not require decompressing in entirety. This is just the way that std.zip decided to expose the API. e.g. iopipe uses an expandable buffer for decompression and compression, which does not have to contain the entire file. -Steve Yes, that's how it's written. That's the problem.
Re: std.zip expand: memory allocation failed
On 10/26/21 1:38 PM, Imperatorn wrote: That's the current implementation. No, that's the API. You cannot fix the implementation with that API and not end up allocating an array to hold the entire unzipped contents. You can't even decompress to a file, and then mmap those contents -- the address space isn't there. I don't know about *nix, but my Windows machine can easily extract a file bigger than my RAM. zlib does not require decompressing in entirety. This is just the way that std.zip decided to expose the API. e.g. iopipe uses an expandable buffer for decompression and compression, which does not have to contain the entire file. -Steve
Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 17:38:22 UTC, Imperatorn wrote: On Tuesday, 26 October 2021 at 13:43:36 UTC, Steven Schveighoffer wrote: On 10/26/21 2:32 AM, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: [...] Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! You guys aren't getting it: ``` ubyte[] expand(ArchiveMember de); Decompress the contents of a member. Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[]. ``` Where is it supposed to store that `ubyte[]`? -Steve That's the current implementation. I don't know about *nix, but my Windows machine can easily extract a file bigger than my RAM. It ofc also depends on the dictionary. The biggest file I've ever decompressed on my own hardware was about 200 GB. Needless to say, it wasn't using the algorithm in std.zip
Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 13:43:36 UTC, Steven Schveighoffer wrote: On 10/26/21 2:32 AM, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: [...] Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! You guys aren't getting it: ``` ubyte[] expand(ArchiveMember de); Decompress the contents of a member. Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[]. ``` Where is it supposed to store that `ubyte[]`? -Steve That's the current implementation. I don't know about *nix, but my Windows machine can easily extract a file bigger than my RAM. It ofc also depends on the dictionary.
Re: std.zip expand: memory allocation failed
On 10/26/21 2:32 AM, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! You guys aren't getting it: ``` ubyte[] expand(ArchiveMember de); Decompress the contents of a member. Fills in properties extractVersion, flags, compressionMethod, time, crc32, compressedSize, expandedSize, expandedData[], name[], extra[]. ``` Where is it supposed to store that `ubyte[]`? -Steve
Re: std.zip expand: memory allocation failed
On Tuesday, 26 October 2021 at 06:32:21 UTC, bauss wrote: On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: [...] Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks! Exactly my thoughts
Re: std.zip expand: memory allocation failed
On Monday, 25 October 2021 at 22:38:38 UTC, Imperatorn wrote: On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory? Definitely not. It shouldn't use a lot of memory when unzipping as it should be done in chunks!
Re: std.zip expand: memory allocation failed
On Monday, 25 October 2021 at 20:50:40 UTC, Steven Schveighoffer wrote: On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve Good catch, but still, should it use so much memory?
Re: std.zip expand: memory allocation failed
On 10/24/21 8:00 AM, Selim Ozel wrote: It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. Wait, x86 is 32-bit. Max address space is 4GB. So maybe it was just trying to use 4GB and running out of memory? -Steve
Re: std.zip expand: memory allocation failed
On Sunday, 24 October 2021 at 14:14:08 UTC, Imperatorn wrote: Create an issue and we can solve it Thanks. I opened an issue. https://issues.dlang.org/show_bug.cgi?id=22436
Re: std.zip expand: memory allocation failed
On Sunday, 24 October 2021 at 12:00:39 UTC, Selim Ozel wrote: On Friday, 15 October 2021 at 20:41:36 UTC, Selim Ozel wrote: [...] It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. My work around was to call 7z from D Lang and do the compression over there. That worked like a charm. It seems that zip.d [1] calls uncompress routine from zlib.d [2]. Would calling zlib uncompress by chunks solve this memory issue? Any ideas? S [1] https://github.com/dlang/phobos/blob/master/std/zip.d [2] https://github.com/dlang/phobos/blob/master/std/zlib.d Create an issue and we can solve it
Re: std.zip expand: memory allocation failed
On Friday, 15 October 2021 at 20:41:36 UTC, Selim Ozel wrote: I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB. The code is very similar to what's explained in the documentation [1] and it works for smaller files. Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure. Selim [1] https://dlang.org/phobos/std_zip.html [2] https://forum.dlang.org/thread/mfnleztnwrbgivjvz...@forum.dlang.org It turns out my computer was literally running out of memory as the file was getting unzipped. For some reason to uncompress a 1-gig file with uncompressed size of 4-gig, Zip Archive of D-Lang tries to use more than 16 gig of RAM. I don't know why. Maybe I missed something. I use a Windows 10, DMD v2.091 with x86_mscoff. My work around was to call 7z from D Lang and do the compression over there. That worked like a charm. It seems that zip.d [1] calls uncompress routine from zlib.d [2]. Would calling zlib uncompress by chunks solve this memory issue? Any ideas? S [1] https://github.com/dlang/phobos/blob/master/std/zip.d [2] https://github.com/dlang/phobos/blob/master/std/zlib.d
Re: std.zip expand: memory allocation failed
Did you try the MmFile workaround? I did. I also pinpointed the problem, I use x86_mscoff to run dub and it's specific to that architecture selection. It's related to MapViewOfFileEx [1]. I still haven't found a way around it though. [1] https://stackoverflow.com/questions/12121843/mapviewoffileex-valid-lpbaseaddress
Re: std.zip expand: memory allocation failed
On Friday, 15 October 2021 at 20:41:36 UTC, Selim Ozel wrote: I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB. The code is very similar to what's explained in the documentation [1] and it works for smaller files. Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure. Selim [1] https://dlang.org/phobos/std_zip.html [2] https://forum.dlang.org/thread/mfnleztnwrbgivjvz...@forum.dlang.org Did you try the MmFile workaround?
std.zip expand: memory allocation failed
I am simply trying to unzip a compressed zip file slightly over 1GB. The de-compressed size is about 4 GB. The code is very similar to what's explained in the documentation [1] and it works for smaller files. Anyone has a solution? Memory mapping [2] previously solved some part of my issue but expand is still throwing memory allocation failure. Selim [1] https://dlang.org/phobos/std_zip.html [2] https://forum.dlang.org/thread/mfnleztnwrbgivjvz...@forum.dlang.org
Re: Missing stacktrace on memory allocation failure
On Wednesday, 2 June 2021 at 17:52:12 UTC, mw wrote: On Thursday, 12 November 2020 at 13:16:21 UTC, Steven Schveighoffer wrote: On 11/12/20 4:22 AM, Per Nordlöw wrote: Why don't I get a stack trace on Memory allocation exceptions? In my case I only get: src/core/exception.d(647): [unittest] Memory allocation failed core.exception.OutOfMemoryError@src/core/exception.d(647): Memory allocation failed Certain errors are flagged as not able to allocate memory. So you don't get a stack trace. I think it's a huge mistake, and we don't need memory allocation to do stack trace printing. But that's the way it is. Hit this one today, since it's a mistake, what does it take to fix it? or you are saying it's unfix-able in the current D runtime? It is fixable, but it'll take a lot of work. The main blocker is that the interface that is exposed to the user relies on memory allocation. Namely, the `TraceInfo` interface is: https://github.com/dlang/druntime/blob/2d8b28da39e8bc3bc3172c69bb96c35d77f40d2a/src/object.d#L2257-L2261 Note that: 1) `toString` returns a `string` (allocation); 2) The `opApply`'s delegate use a `char[]` as parameter; What we *should* have is an `opApply` on a struct that contains all necessary info, which itself writes to a sink / OutputRange. Such struct exists (https://github.com/dlang/druntime/blob/a17bb23b418405e1ce8e4a317651039758013f39/src/core/internal/backtrace/dwarf.d#L77-L160) but need to be exposed to user code. Then *printing* the stack trace will no longer allocate. However, `defaultTraceHandler` still returns an object, so acquiring the handler will also need to be addressed. There have been talks of deprecating chained Exceptions, and that would make such a task much simpler.
Re: Missing stacktrace on memory allocation failure
On Thursday, 12 November 2020 at 13:16:21 UTC, Steven Schveighoffer wrote: On 11/12/20 4:22 AM, Per Nordlöw wrote: Why don't I get a stack trace on Memory allocation exceptions? In my case I only get: src/core/exception.d(647): [unittest] Memory allocation failed core.exception.OutOfMemoryError@src/core/exception.d(647): Memory allocation failed Certain errors are flagged as not able to allocate memory. So you don't get a stack trace. I think it's a huge mistake, and we don't need memory allocation to do stack trace printing. But that's the way it is. Hit this one today, since it's a mistake, what does it take to fix it? or you are saying it's unfix-able in the current D runtime?
Re: Memory allocation
On Wednesday, 24 February 2021 at 06:14:58 UTC, Imperatorn wrote: On Tuesday, 23 February 2021 at 19:44:39 UTC, David wrote: Not sure if `learn` is the right topic or not to post this.. I've been going through Bob Nystrom's "Crafting Interpreters" for a bit of fun and over the weekend put together a toy allocator in D - free and gc not yet done. It's single threaded and unsurprisingly faster than malloc for small objects up to 512 bytes, and about the same for larger objects (simulated with a linear distribution of size requests). [...] Have you looked at the "experimental" allocator(s) in D? std.experimental.allocator Hi, yes, good question, yes I did - my first cut using the mmap_allocator was slower - I presume because I should really allocate a good chunk of virtual memory and manage the pages myself. So some questions I'm asking myself - What people's heuristics for switching methods say from, some sort of pool to another approach - e.g. linked list of pages? At the page level does a bitmap allocator make more sense? Knuth (it seems) was under the impression that a third of memory is wasted by fragmentation, aka the 50% rule - that doesn't seem likely, does anyone measure these things? How does one go about choosing a set of pool sizes? Given that virtual pages are not necessarily physically contiguous are some of the considerations that inspired the buddy system no longer relevant. I should also say I'm aiming for a single thread longish running process (one + working days) that I anticipate could use a good chunk say 100GB or so on occasion but I'd like to be more efficient than java in terms of memory reuse - past experience was I had to reboot my workstation a few times a day as it did something really horrible (I don't know what). Any there any GC frameworks that someone has shared in D?
Re: Memory allocation
On Tuesday, 23 February 2021 at 19:44:39 UTC, David wrote: Not sure if `learn` is the right topic or not to post this.. I've been going through Bob Nystrom's "Crafting Interpreters" for a bit of fun and over the weekend put together a toy allocator in D - free and gc not yet done. It's single threaded and unsurprisingly faster than malloc for small objects up to 512 bytes, and about the same for larger objects (simulated with a linear distribution of size requests). [...] Have you looked at the "experimental" allocator(s) in D? std.experimental.allocator
Memory allocation
Not sure if `learn` is the right topic or not to post this.. I've been going through Bob Nystrom's "Crafting Interpreters" for a bit of fun and over the weekend put together a toy allocator in D - free and gc not yet done. It's single threaded and unsurprisingly faster than malloc for small objects up to 512 bytes, and about the same for larger objects (simulated with a linear distribution of size requests). But I'd like to be able to benchmark it to see how I'm doing. A colleague suggested I ask here for resources and pointers - I was trying to find stats on malloc performance (internal and external fragmentation, long running reuse, distributions of size requests for testing and comparison, how to reason about trade-offs of cache missing, branch misprediction, meta data, etc). So academic papers, books, blogs, experiences are all welcome, especially with regard to optimising for immutability / COW - would be welcome or even just some good ol' conversation.
Re: Missing stacktrace on memory allocation failure
On 11/12/20 4:22 AM, Per Nordlöw wrote: Why don't I get a stack trace on Memory allocation exceptions? In my case I only get: src/core/exception.d(647): [unittest] Memory allocation failed core.exception.OutOfMemoryError@src/core/exception.d(647): Memory allocation failed Certain errors are flagged as not able to allocate memory. So you don't get a stack trace. I think it's a huge mistake, and we don't need memory allocation to do stack trace printing. But that's the way it is. -Steve
Re: Missing stacktrace on memory allocation failure
On Thursday, 12 November 2020 at 09:22:10 UTC, Per Nordlöw wrote: Why don't I get a stack trace on Memory allocation exceptions? Is it because the stack may be corrupted?
Missing stacktrace on memory allocation failure
Why don't I get a stack trace on Memory allocation exceptions? In my case I only get: src/core/exception.d(647): [unittest] Memory allocation failed core.exception.OutOfMemoryError@src/core/exception.d(647): Memory allocation failed
Re: Memory allocation failed in CT
On Saturday, 13 July 2019 at 06:25:37 UTC, Andrey wrote: On Tuesday, 9 July 2019 at 19:04:53 UTC, Max Haughton wrote: Is this a 64 or 32 bit compiler? Also could you post the source code if possible? You could try "--DRT-gcopt=profile:1" druntime flag to see if the compiler is running out of memory for real Thanks for help. I solved my issue by rewritting code. can u describe how u rewrote the code? so it can be solution for all of us: when u generate long string in CT for mixins and u get an exception u can do next ...
Re: Memory allocation failed in CT
On Tuesday, 9 July 2019 at 19:04:53 UTC, Max Haughton wrote: Is this a 64 or 32 bit compiler? Also could you post the source code if possible? You could try "--DRT-gcopt=profile:1" druntime flag to see if the compiler is running out of memory for real Thanks for help. I solved my issue by rewritting code.
Re: Memory allocation failed in CT
Is this a 64 or 32 bit compiler? Also could you post the source code if possible? You could try "--DRT-gcopt=profile:1" druntime flag to see if the compiler is running out of memory for real
Re: Memory allocation failed in CT
On Tuesday, 9 July 2019 at 17:59:24 UTC, Max Haughton wrote: On Tuesday, 9 July 2019 at 17:48:52 UTC, Andrey wrote: Hello, I have got a problem with compile-time calulations. I have some code generator that should create some long string of code during CT and after generation I mixin it. If I run it normally - in run time - then there is no error and I get expected output - string with size ~ 3.5 MB. If I run it in CT then I recieve an error: [...] I don't understand why... The only operation in my generator is string concatination: _result ~= "some code...". Are you using the -lowmem flag? This enables the GC during compilation i.e. you might be running out of memory (CTFE is not efficient with memory during evaluation) I tried to turn on this flag but no success. The same error...
Re: Memory allocation failed in CT
On Tuesday, 9 July 2019 at 17:59:24 UTC, Max Haughton wrote: On Tuesday, 9 July 2019 at 17:48:52 UTC, Andrey wrote: I in addition wrote "buffer.reserve(10 * 1014 * 1024);" and it also doesn't help.
Re: Memory allocation failed in CT
On Tuesday, 9 July 2019 at 17:48:52 UTC, Andrey wrote: Hello, I have got a problem with compile-time calulations. I have some code generator that should create some long string of code during CT and after generation I mixin it. If I run it normally - in run time - then there is no error and I get expected output - string with size ~ 3.5 MB. If I run it in CT then I recieve an error: [...] I don't understand why... The only operation in my generator is string concatination: _result ~= "some code...". Are you using the -lowmem flag? This enables the GC during compilation i.e. you might be running out of memory (CTFE is not efficient with memory during evaluation)
Memory allocation failed in CT
Hello, I have got a problem with compile-time calulations. I have some code generator that should create some long string of code during CT and after generation I mixin it. If I run it normally - in run time - then there is no error and I get expected output - string with size ~ 3.5 MB. If I run it in CT then I recieve an error: core.exception.OutOfMemoryError@core\exception.d(702): Memory allocation failed 0x7FF65A80BE73 0x7FF65A80BE73 0x7FF65A80BB46 0x7FF65A7FBA5C 0x7FF65A8035EC 0x7FF65A804311 0x7FF658BA733E 0x7FF658CC561F 0x7FF658CC0CD8 0x7FF658CB3728 0x7FF658CB3826 0x7FF658CB4762 0x7FF658CB3826 0x7FF658CB3CD0 E:\Programs\LDC2\bin\ldc2.exe failed with exit code 1. I don't understand why... The only operation in my generator is string concatination: _result ~= "some code...".
Re: Closures and memory allocation
On Saturday, 22 June 2019 at 16:52:07 UTC, Anonymouse wrote:
When entering the following function, does it allocate:
1. 0 times, because while there are closures defined, none is
ever called?
2. 2 times, because there are closures over two variables?
3. 20 times, because there are 20 unique closures?
4. One time, allocating a single closure for all variables
referenced in nested functions. Easy to check by inspecting the
generated asm (https://run.dlang.io/is/qO3JxO):
void receive(void delegate() a, void delegate() b) {}
void foo()
{
int i;
long l;
void foo1() { i += 1; }
void foo2() { l += 1; }
receive(&foo1, &foo2);
}
When adding `scope` to the `receive()` params, the closure is
allocated on the stack.
Re: Closures and memory allocation
On Saturday, 22 June 2019 at 19:26:13 UTC, Cym13 wrote: On Saturday, 22 June 2019 at 16:52:07 UTC, Anonymouse wrote: [...] Clearly this is a good time for you to learn about the tools D offers to profile allocations. There is the --profile=gc DMD argument that you can use but here there's something better: DMD's GC has a few hooks that are directly inside druntime and therefore available to any D program. [...] Ooops, sorry I went a bit fast there, --DRT-gcopt gives you the number of collections, not allocations.
Re: Closures and memory allocation
On Saturday, 22 June 2019 at 16:52:07 UTC, Anonymouse wrote: I'm looking into why my thing does so many memory allocations. Profiling with kcachegrind shows _d_allocmemory being called upon entering a certain function, lots and lots of times. It's a function that receives concurrency messages, so it contains nested functions that close over local variables. Think receiveTimeout(0.seconds, &nested1, &nested2, &nested3, ...) with 13 pointers to nested functions passed. When entering the following function, does it allocate: 1. 0 times, because while there are closures defined, none is ever called? 2. 2 times, because there are closures over two variables? 3. 20 times, because there are 20 unique closures? Clearly this is a good time for you to learn about the tools D offers to profile allocations. There is the --profile=gc DMD argument that you can use but here there's something better: DMD's GC has a few hooks that are directly inside druntime and therefore available to any D program. Putting your above code in test.d you can then do: $ dmd test.d $ ./test --DRT-gcopt=profile:1 Number of collections: 2 Total GC prep time: 0 milliseconds Total mark time: 0 milliseconds Total sweep time: 0 milliseconds Max Pause Time: 0 milliseconds Grand total GC time: 0 milliseconds GC summary:1 MB,2 GC0 ms, Pauses0 ms <0 ms And here is your answer: two allocations. More information about --DRT-gcopt there: https://dlang.org/spec/garbage.html
Closures and memory allocation
I'm looking into why my thing does so many memory allocations.
Profiling with kcachegrind shows _d_allocmemory being called upon
entering a certain function, lots and lots of times.
It's a function that receives concurrency messages, so it
contains nested functions that close over local variables. Think
receiveTimeout(0.seconds, &nested1, &nested2, &nested3, ...) with
13 pointers to nested functions passed.
When entering the following function, does it allocate:
1. 0 times, because while there are closures defined, none is
ever called?
2. 2 times, because there are closures over two variables?
3. 20 times, because there are 20 unique closures?
void foo()
{
int i;
long l;
void foo01(bool b_) { i += 1; }
void foo02(string s) { i += 2; }
void foo03(int n) { i += 3; }
void foo04(float f) { i += 4; }
void foo05(double d) { i += 5; }
void foo06() { i += 6; }
void foo07() { i += 7; }
void foo08() { i += 8; }
void foo09() { i += 9; }
void foo10() { i += 10; }
void foo11() { l += 11; }
void foo12() { l += 12; }
void foo13() { l += 13; }
void foo14() { l += 14; }
void foo15() { l += 15; }
void foo16() { l += 16; }
void foo17() { l += 17; }
void foo18() { l += 18; }
void foo19() { l += 19; }
void foo20() { l += 20; }
auto f01 = &foo01;
auto f02 = &foo02;
auto f03 = &foo03;
auto f04 = &foo04;
auto f05 = &foo05;
auto f06 = &foo06;
auto f07 = &foo07;
auto f08 = &foo08;
auto f09 = &foo09;
auto f10 = &foo10;
auto f11 = &foo11;
auto f12 = &foo12;
auto f13 = &foo13;
auto f14 = &foo14;
auto f15 = &foo15;
auto f16 = &foo16;
auto f17 = &foo17;
auto f18 = &foo18;
auto f19 = &foo19;
auto f20 = &foo20;
}
Re: Does to!(string)(char[]) do any memory allocation on conversion?
On Monday, December 25, 2017 15:00:19 aliak via Digitalmars-d-learn wrote: > On Monday, 25 December 2017 at 14:12:32 UTC, Marc wrote: > > Does to!(string)(char[]) do any memory allocation on conversion > > or is this similar to a cast or what else? > > As said it calls idup, which calls _trustedDup which seems to > call _dup which does memory allocation -> > https://github.com/dlang/druntime/blob/v2.077.1/src/object.d#L3863 > > I think you can use assumeUnique to avoid allocs though. See code > gen here: > > https://godbolt.org/g/44pLpL assumeUnique just casts to immutable and should be used with extreme care as casting from mutable or const to immutable runs a serious risk of subtle bugs if done incorrectly. If you do it, the reference being cast really needs to be the only reference to that object. Almost always, the better approach is to construct an object with a pure function that is able to implicitly convert its return value to immutable, because the compiler is able to prove that the mutable object was not passed into the function and therefore that it is unique. In the case of strings, to!string is good to use, but it will result in a new string being allocated if it's not given an immutable string. std.conv.to generally tries to ensure that conversions are done in a safe manner, and casting mutability is risky business and best to be avoided in general. - Jonathan M Davis
Re: Does to!(string)(char[]) do any memory allocation on conversion?
On Monday, 25 December 2017 at 14:12:32 UTC, Marc wrote: Does to!(string)(char[]) do any memory allocation on conversion or is this similar to a cast or what else? As said it calls idup, which calls _trustedDup which seems to call _dup which does memory allocation -> https://github.com/dlang/druntime/blob/v2.077.1/src/object.d#L3863 I think you can use assumeUnique to avoid allocs though. See code gen here: https://godbolt.org/g/44pLpL
Re: Does to!(string)(char[]) do any memory allocation on conversion?
On Monday, 25 December 2017 at 14:37:01 UTC, Mengu wrote: On Monday, 25 December 2017 at 14:12:32 UTC, Marc wrote: Does to!(string)(char[]) do any memory allocation on conversion or is this similar to a cast or what else? yes, it is allocating memory. you can test such cases with @nogc [0]. you can get a char[] via .dup of a string and then you can cast(string) if you don't want to allocate any memory. [0] https://dlang.org/spec/attribute.html#nogc dup allocates memory too
Re: Does to!(string)(char[]) do any memory allocation on conversion?
On Monday, 25 December 2017 at 14:12:32 UTC, Marc wrote: Does to!(string)(char[]) do any memory allocation on conversion or is this similar to a cast or what else? yes, it is allocating memory. you can test such cases with @nogc [0]. you can get a char[] via .dup of a string and then you can cast(string) if you don't want to allocate any memory. [0] https://dlang.org/spec/attribute.html#nogc
Re: Does to!(string)(char[]) do any memory allocation on conversion?
On Monday, 25 December 2017 at 14:37:01 UTC, Mengu wrote: yes, it is allocating memory. you can test such cases with @nogc [0]. nogc is really conservative and thus gives a lot of false positives. I'd just compare instr.ptr is outstr.ptr here and see if it changes (it will tho)
Re: Does to!(string)(char[]) do any memory allocation on conversion?
On Monday, 25 December 2017 at 14:12:32 UTC, Marc wrote: Does to!(string)(char[]) do any memory allocation on conversion or is this similar to a cast or what else? It is translated to idup. So yes, it allocates memory.
Does to!(string)(char[]) do any memory allocation on conversion?
Does to!(string)(char[]) do any memory allocation on conversion or is this similar to a cast or what else?
Re: DMD [-O flag] vs. [memory allocation in a synchronized class]
On Thursday, 8 June 2017 at 17:39:41 UTC, Ivan Kazmenko wrote: Reported: https://issues.dlang.org/show_bug.cgi?id=17481 Thank You!
Re: DMD [-O flag] vs. [memory allocation in a synchronized class]
On Thursday, 8 June 2017 at 15:35:06 UTC, Ivan Kazmenko wrote: Perhaps a regression should be filed, or searched for, at issues.dlang.org. I can do it, but not right now, and would be glad if someone beats me to it. Reported: https://issues.dlang.org/show_bug.cgi?id=17481
Re: DMD [-O flag] vs. [memory allocation in a synchronized class]
On Thursday, 8 June 2017 at 11:41:40 UTC, realhet wrote:
I've managed to narrow the problem even more:
//win32 dmd -O
class Obj{
synchronized void trigger(){ new ubyte[1]; }
}
void main(){
auto k = new shared Obj;
k.trigger;
}
This time I got a more sophisticated error message:
object.Error@(0): Access Violation
0x7272456D in SymInitialize
0x00402667
0x00402A97
0x00402998
0x004022A0
0x76F13744 in BaseThreadInitThunk
0x773B9E54 in RtlSetCurrentTransaction
0x773B9E1F in RtlSetCurrentTransaction
I can reproduce this under win32, and it breaks somewhere between
2.068.2 and 2.069.0.
The move instructive message with "dmd -O -g":
object.Error@(0): Access Violation
0x0065
0x00402C33 in _d_newarrayU
0x004022EB in _d_newarrayT
0x00402A7F in scope void rt.dmain2._d_run_main(int, char**,
extern (C) int function(char[][])*).runAll()
0x00402980 in _d_run_main
0x00402288 in main at C:\a.d(8)
0x757F336A in BaseThreadInitThunk
0x77409902 in RtlInitializeExceptionChain
0x774098D5 in RtlInitializeExceptionChain
Perhaps a regression should be filed, or searched for, at
issues.dlang.org. I can do it, but not right now, and would be
glad if someone beats me to it.
Ivan Kazmenko.
Re: DMD [-O flag] vs. [memory allocation in a synchronized class]
I've managed to narrow the problem even more:
//win32 dmd -O
class Obj{
synchronized void trigger(){ new ubyte[1]; }
}
void main(){
auto k = new shared Obj;
k.trigger;
}
This time I got a more sophisticated error message:
object.Error@(0): Access Violation
0x7272456D in SymInitialize
0x00402667
0x00402A97
0x00402998
0x004022A0
0x76F13744 in BaseThreadInitThunk
0x773B9E54 in RtlSetCurrentTransaction
0x773B9E1F in RtlSetCurrentTransaction
Re: DMD [-O flag] vs. [memory allocation in a synchronized class]
realhet wrote: On Thursday, 8 June 2017 at 10:48:41 UTC, ketmar wrote: worksforme with -O, and with -O -inline. I forgot to mention, that I'm generating win32 output. DMD32 D Compiler v2.074.0 mine: GNU/Linux, 32 bit, dmd git HEAD.
Re: DMD [-O flag] vs. [memory allocation in a synchronized class]
On Thursday, 8 June 2017 at 10:48:41 UTC, ketmar wrote: worksforme with -O, and with -O -inline. I forgot to mention, that I'm generating win32 output. DMD32 D Compiler v2.074.0
Re: DMD [-O flag] vs. [memory allocation in a synchronized class]
worksforme with -O, and with -O -inline.
DMD [-O flag] vs. [memory allocation in a synchronized class]
Hi,
This code works well with the unoptimized compilation with DMD.
import std.array;
synchronized class Obj{
private int[] arr;
void trigger(){ arr.length += 1; }
}
void main(){
auto k = new shared Obj;
k.trigger;
}
And when I use the -O option, it shows the following error in the
trigger() method:
core.exception.InvalidMemoryOperationError@src\core\exception.d(696): Invalid
memory operation
Why? o.O
Re: Memory Allocation
On Thursday, 30 March 2017 at 11:22:05 UTC, Gary Willoughby wrote: On Wednesday, 29 March 2017 at 19:19:48 UTC, Enigma wrote: I have a memory buffer allocated using different methods. It is simply a pointer and a size. I would like to be able to manage this buffer by treating it as a memory pool or heap. I think I can use allocators to do this but not sure how. You can re-task the GC to use an arena if that's suitable: https://bitbucket.org/infognition/dstuff/src/1dca752af1021ed5998bb041004465674695113f/gcarena.d?fileviewer=file-view-default cool. thanks.
Re: Memory Allocation
On Wednesday, 29 March 2017 at 23:26:04 UTC, H. S. Teoh wrote: On Wed, Mar 29, 2017 at 11:01:12PM +, Enigma via Digitalmars-d-learn wrote: On Wednesday, 29 March 2017 at 21:36:14 UTC, Petar Kirov [ZombineDev] wrote: > [...] But these seem to require passing a mallocator. I simply want to pass an already allocated region/block/buffer and have the allocators use it. I will allocate and free on my own. Huh? Where does it say that a mallocator is required? As far as I can tell, you could simply do this: void[] myBuffer = ...; auto allocator = Region!()(myBuffer); auto p = allocator.allocate(...); The default parent allocator is NullAllocator, which does nothing, so that leaves the management of myBuffer entirely up to you. T Thanks. All the examples show using a size rather than an array so I assumed that was the case.
Re: Memory Allocation
On Wednesday, 29 March 2017 at 19:19:48 UTC, Enigma wrote: I have a memory buffer allocated using different methods. It is simply a pointer and a size. I would like to be able to manage this buffer by treating it as a memory pool or heap. I think I can use allocators to do this but not sure how. You can re-task the GC to use an arena if that's suitable: https://bitbucket.org/infognition/dstuff/src/1dca752af1021ed5998bb041004465674695113f/gcarena.d?fileviewer=file-view-default
Re: Memory Allocation
On Wed, Mar 29, 2017 at 11:01:12PM +, Enigma via Digitalmars-d-learn wrote: > On Wednesday, 29 March 2017 at 21:36:14 UTC, Petar Kirov [ZombineDev] wrote: > > On Wednesday, 29 March 2017 at 19:19:48 UTC, Enigma wrote: > > > [...] > > > > It looks like you are looking for this: > > http://dlang.org/phobos-prerelease/std_experimental_allocator_building_blocks_region.html. > > But these seem to require passing a mallocator. I simply want to pass > an already allocated region/block/buffer and have the allocators use > it. I will allocate and free on my own. Huh? Where does it say that a mallocator is required? As far as I can tell, you could simply do this: void[] myBuffer = ...; auto allocator = Region!()(myBuffer); auto p = allocator.allocate(...); The default parent allocator is NullAllocator, which does nothing, so that leaves the management of myBuffer entirely up to you. T -- Life would be easier if I had the source code. -- YHL
Re: Memory Allocation
On Wednesday, 29 March 2017 at 21:36:14 UTC, Petar Kirov [ZombineDev] wrote: On Wednesday, 29 March 2017 at 19:19:48 UTC, Enigma wrote: [...] It looks like you are looking for this: http://dlang.org/phobos-prerelease/std_experimental_allocator_building_blocks_region.html. But these seem to require passing a mallocator. I simply want to pass an already allocated region/block/buffer and have the allocators use it. I will allocate and free on my own.
Re: Memory Allocation
On Wednesday, 29 March 2017 at 19:19:48 UTC, Enigma wrote: I have a memory buffer allocated using different methods. It is simply a pointer and a size. I would like to be able to manage this buffer by treating it as a memory pool or heap. I think I can use allocators to do this but not sure how. Effectively I want something like new or malloc but it pulls from the memory buffer rather than the program heap. // Allocated once at program start void* FancyBuffer = FancyAlloc(1000); Then when I want to use the buffer I'll do stuff like auto myptr = FancyMalloc(10); FancyFree(myptr); or whatever. The main thing is, I don't want to have to write my own allocator to manage this buffer as it seems that D's allocators would do a better job. I imagine that most of the time the buffer will not have more than one piece of code using it(no overlapping uses) but since I won't be 100% sure, I need allow for the cases where there might be overlapping usage. (else I wouldn't ever have to worry about "allocating or releasing" from it. Thanks. It looks like you are looking for this: http://dlang.org/phobos-prerelease/std_experimental_allocator_building_blocks_region.html.
Re: Memory Allocation
On 2017-03-29 23:30, Faux Amis wrote: On 2017-03-29 21:19, Enigma wrote: I have a memory buffer allocated using different methods. It is simply a pointer and a size. Can you maybe just tread it like an array and slice it for allocation? *treat*
Re: Memory Allocation
On 2017-03-29 21:19, Enigma wrote: I have a memory buffer allocated using different methods. It is simply a pointer and a size. Can you maybe just tread it like an array and slice it for allocation?
Memory Allocation
I have a memory buffer allocated using different methods. It is simply a pointer and a size. I would like to be able to manage this buffer by treating it as a memory pool or heap. I think I can use allocators to do this but not sure how. Effectively I want something like new or malloc but it pulls from the memory buffer rather than the program heap. // Allocated once at program start void* FancyBuffer = FancyAlloc(1000); Then when I want to use the buffer I'll do stuff like auto myptr = FancyMalloc(10); FancyFree(myptr); or whatever. The main thing is, I don't want to have to write my own allocator to manage this buffer as it seems that D's allocators would do a better job. I imagine that most of the time the buffer will not have more than one piece of code using it(no overlapping uses) but since I won't be 100% sure, I need allow for the cases where there might be overlapping usage. (else I wouldn't ever have to worry about "allocating or releasing" from it. Thanks.
Re: Memory allocation failed. Why?
On Tuesday, 22 November 2016 at 15:53:39 UTC, Steven Schveighoffer wrote: On 11/21/16 11:53 AM, ag0aep6g wrote: Thank you very much for explaining such a difficult and slippery situation.
Re: Memory allocation failed. Why?
On 11/21/16 11:53 AM, ag0aep6g wrote: On Monday, 21 November 2016 at 16:37:32 UTC, Kagamin wrote: Anything in .data and .bss sections and stack. See https://issues.dlang.org/show_bug.cgi?id=15723 Ok, not an actual reference then, but a false pointer. Yes. 100 million bytes is 1/40 of all addressable space on 32-bits. There only needs to be one 4-byte segment somewhere on the stack that points at this, and it won't be collected. Assuming you have a quite large segment that can't be extended or collected (due to false pointer), this means you have to allocate another large one to satisfy the next allocation (which then could be pinned). And it gets worse from there. -Steve
Re: Memory allocation failed. Why?
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote: core.exception.OutOfMemoryError@src\core\exception.d(693): Memory allocation failed Simple program and error. Why? Windows 7 (32) dmd 2.072.0 Making a 100 million bytes array by appending one byte at a time creates a lot of intermediate-size arrays. Ideally these should be garbage collected away but GC in D is not only slow but also leaky. In 32 bits if you have 1000 random int values on the stack or data segment, with uniform distribution, this is 1000 random locations in memory pinned and seen by GC as live, i.e. one per 4 MB of address space. Which means if your array is 4 MB or larger it's almost doomed to be never collected by GC in this scenario. Your program creates a lot of large arrays and they don't get collected because of false pointers and not precise enough GC. Moral of the story: in 32 bits don't allocate anything big (1 MB or more) in GC heap, otherwise there are good chances it will create a memory leak. Use std.container.array or something similar.
Re: Memory allocation failed. Why?
On Monday, 21 November 2016 at 16:37:32 UTC, Kagamin wrote: Anything in .data and .bss sections and stack. See https://issues.dlang.org/show_bug.cgi?id=15723 Ok, not an actual reference then, but a false pointer.
Re: Memory allocation failed. Why?
On Monday, 21 November 2016 at 11:22:40 UTC, ag0aep6g wrote: Who could "someone" be? It's a self-contained example, and buf doesn't leave the test function. Anything in .data and .bss sections and stack. See https://issues.dlang.org/show_bug.cgi?id=15723 As for GC compaction: https://issues.dlang.org/show_bug.cgi?id=3284
Re: Memory allocation failed. Why?
On 11/21/2016 08:27 AM, Stefan Koch wrote: Someone could still be hanging on to an old Reference of buf. Who could "someone" be? It's a self-contained example, and buf doesn't leave the test function.
Re: Memory allocation failed. Why?
On Monday, 21 November 2016 at 06:45:04 UTC, ag0aep6g wrote: On 11/21/2016 07:36 AM, Stefan Koch wrote: On Monday, 21 November 2016 at 03:58:00 UTC, MGW wrote: On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote: On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote: [...] [...] You are appending 100_000_000 Times. Note that this will take up much more then 100 MB. Because the allocator cannot always grow the array. All previously allocated blocks are still in scope and could be reached. Therefore the memory is not released. How could they be reached? The only reference is `buf', and when appending triggers relocation, `buf` should point to the new location, no? Someone could still be hanging on to an old Reference of buf.
Re: Memory allocation failed. Why?
On 11/21/2016 07:36 AM, Stefan Koch wrote:
On Monday, 21 November 2016 at 03:58:00 UTC, MGW wrote:
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
[...]
You are appending 100_000_000 Times.
Note that this will take up much more then 100 MB.
Because the allocator cannot always grow the array.
All previously allocated blocks are still in scope and could be reached.
Therefore the memory is not released.
How could they be reached? The only reference is `buf', and when
appending triggers relocation, `buf` should point to the new location, no?
Re: Memory allocation failed. Why?
On Monday, 21 November 2016 at 03:58:00 UTC, MGW wrote:
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
core.exception.OutOfMemoryError@src\core\exception.d(693):
Memory allocation failed
This short program doesn't work at Windows 32. Why GC doesn't
release memory in case of its shortage in Windows! This is bug?
You are appending 100_000_000 Times.
Note that this will take up much more then 100 MB.
Because the allocator cannot always grow the array.
All previously allocated blocks are still in scope and could be
reached.
Therefore the memory is not released.
At least this is what I think happens.
it would help more if you could post the specs of your machine
and what else is running on it.
Re: Memory allocation failed. Why?
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
core.exception.OutOfMemoryError@src\core\exception.d(693):
Memory allocation failed
This short program doesn't work at Windows 32. Why GC doesn't
release memory in case of its shortage in Windows! This is bug?
Re: Memory allocation failed. Why?
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote: On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote: [...] For me there's no exception. Maybe the GC is poluted. Try to add this after each iteration in the first test loop: import core.memory: GC; GC.collect(); Also note that the text you pas in the message box should be null terminated, eg: MessageBoxA(null, "--The end--\0".ptr, "".ptr, 0); string literals are implicitly null terminated explicitly for the interoperation with C. there is no need.
Re: Memory allocation failed. Why?
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
core.exception.OutOfMemoryError@src\core\exception.d(693):
Memory allocation failed
Simple program and error. Why? Windows 7 (32) dmd 2.072.0
For me there's no exception. Maybe the GC is poluted. Try to add
this after each iteration in the first test loop:
import core.memory: GC;
GC.collect();
Also note that the text you pas in the message box should be null
terminated, eg:
MessageBoxA(null, "--The end--\0".ptr, "".ptr, 0);
Memory allocation failed. Why?
import core.sys.windows.windows: MessageBoxA;
void test() {
for(int i; i != 10; i++) {
ubyte[] buf;
for(int j; j != 1; j++) buf ~= 65;
MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
// delete buf; // if ON - then memory delete in step
}
MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}
void main() {
test();
MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}
core.exception.OutOfMemoryError@src\core\exception.d(693): Memory
allocation failed
Simple program and error. Why? Windows 7 (32) dmd 2.072.0
Re: What happens if memory allocation fails?
On Sat, 20 Feb 2016 16:58:02 +, Adam D. Ruppe wrote: > On Saturday, 20 February 2016 at 14:21:28 UTC, tcak wrote: >> What happens if memory allocation fails with "new" keyword? > > Be aware that memory allocation might never actually fail. It really > depends on the operating system. > > But if it did fail, it would throw OutOfMemoryError > > http://dpldocs.info/experimental-docs/ core.exception.OutOfMemoryError.html That should work for address space exhaustion on most any OS and configuration. Physical memory plus swap exhaustion, not so much, thanks to memory overcommitment.
Re: What happens if memory allocation fails?
On Saturday, 20 February 2016 at 14:21:28 UTC, tcak wrote: What happens if memory allocation fails with "new" keyword? Be aware that memory allocation might never actually fail. It really depends on the operating system. But if it did fail, it would throw OutOfMemoryError http://dpldocs.info/experimental-docs/core.exception.OutOfMemoryError.html
Re: What happens if memory allocation fails?
Currently it crashes: https://issues.dlang.org/show_bug.cgi?id=1180
What happens if memory allocation fails?
This is not easy to try. So I need ask, maybe someone has experienced. What happens if memory allocation fails with "new" keyword? Does it throw an exception? throwable? All I want is to be able to catch OutOfMemory event, and take other steps based on that.
Re: C string to D without memory allocation?
On Monday, 21 December 2015 at 09:46:58 UTC, Shriramana Sharma wrote: Jonathan M Davis via Digitalmars-d-learn wrote: If it isn't, all that means is that the array's capacity will be 0, so it's going to have to reallocate So it's safe to return a string produced by fromStringz without having to worry that the user would append to it? Yes. Then why is it marked @system? Only because one cannot be sure that the input point refers to a valid null-terminated string? Exactly.
Re: C string to D without memory allocation?
Jonathan M Davis via Digitalmars-d-learn wrote: > If it isn't, all that means is that the > array's capacity will be 0, so it's going to have to reallocate So it's safe to return a string produced by fromStringz without having to worry that the user would append to it? Then why is it marked @system? Only because one cannot be sure that the input point refers to a valid null-terminated string? -- Shriramana Sharma, Penguin #395953
Re: C string to D without memory allocation?
On Monday, 21 December 2015 at 08:35:22 UTC, Jonathan M Davis wrote: There's also fromStringz that Jakob suggests using elsewhere in this thread, but that really just boils down to return cString ? cString[0 .. strlen(cString)] : null; So, using that over simply slicing is primarily for documentation purposes, though it does make it so that you don't have to call strlen directly or check for null before calling it. To add to this, the main motivation behind `fromStringz` is that `cString` is often a non-trivial expression, such as a function call. With `fromStringz`, this expression can always be put in the argument list and it will only be evaluated once. Otherwise a variable has to be added: auto cString = foo(); return cString[0 .. strlen(cString)];
Re: C string to D without memory allocation?
On Monday, December 21, 2015 18:39:32 Rikki Cattermole via Digitalmars-d-learn wrote: > size_t strLen = ...; > char* ptr = ...; > > string myCString = cast(string)ptr[0 .. strLen]; > > I can't remember if it will include the null terminator or not, but if > it does just decrease strLen by 1. Casting to string is almost always wrong. Casting anything to immutable is almost always wrong. If you want to use C strings in D without allocating, they're going to have to be const(char)[], not immutable(char)[], which is what string is. Otherwise, you risk violating the type system. Slicing explicitly like this can work just fine, and it will result in either char[] or const(char)[] depending on whether the pointer is char* or const char*, but the cast should _not_ be done. There's also fromStringz that Jakob suggests using elsewhere in this thread, but that really just boils down to return cString ? cString[0 .. strlen(cString)] : null; So, using that over simply slicing is primarily for documentation purposes, though it does make it so that you don't have to call strlen directly or check for null before calling it. Regardless, do _not_ cast something to immutable if it's not already immutable. It's just begging for trouble. - Jonathan M Davis
Re: C string to D without memory allocation?
On Monday, December 21, 2015 05:43:59 Jakob Ovrum via Digitalmars-d-learn wrote: > On Monday, 21 December 2015 at 05:41:31 UTC, Shriramana Sharma > wrote: > > Rikki Cattermole wrote: > > > >> string myCString = cast(string)ptr[0 .. strLen]; > > > > Thanks but does this require that one doesn't attempt to append > > to the returned string using ~= or such? In which case it is > > not safe, right? > > Growing operations like ~= will copy the array to a GC-allocated, > druntime-managed array if it isn't one already. Exactly. As long as the GC has not been disabled, that there is sufficient memory to allocate, and that appending elements does not result in an exception being thrown (which it wouldn't with arrays of char) ~= should always work. When ~= is used, the runtime looks at the capacity of the dynamic array to see whether it has enough room to grow to fit the new elements. If it does, then the array is grown into that space. If it doesn't, then a block of GC memory is allocated, the elements are copied into that memory, and the dynamic array is set to point to that block of memory. Whether the array is pointing to a GC-allocated block of memory or not when ~= is called is irrelevant. If it isn't, all that means is that the array's capacity will be 0, so it's going to have to reallocate, whereas if it were GC-allocated, it might have enough capacity to not need to reallocate. In either case, the operation will work. - Jonathan M Davis
Re: C string to D without memory allocation?
On Monday, 21 December 2015 at 06:00:45 UTC, Shriramana Sharma
wrote:
I suppose what you mean is, the onus of guaranteeing that
const(char)* refers to a null-terminated string is upon the
person calling the to! function? Yes I understand, and Phobos
documentation does say that using a pointer for input makes
this "@system". Wouldn't it be better to just reject pointer as
input and force people to use fromStringz?
It could also simply have done the same as format("%p", p). The
problem isn't that to!string is too accepting, but that it makes
an unsafe assumption with an inconspicuous, generic interface.
I see that "%s".format(str) where str is a const(char)* just
prints the pointer value in hex. So perhaps one should say that
std.format just treats it like any other pointer (and not
specifically that it treats it as a pointer to a single char)?
Indeed.
Re: C string to D without memory allocation?
Jakob Ovrum wrote: > Use std.string.fromStringz. to!string assumes that pointers to > characters are null-terminated strings which is not safe or > general I suppose what you mean is, the onus of guaranteeing that const(char)* refers to a null-terminated string is upon the person calling the to! function? Yes I understand, and Phobos documentation does say that using a pointer for input makes this "@system". Wouldn't it be better to just reject pointer as input and force people to use fromStringz? > > (unlike std.format, which safely assumes they are > pointers to single characters); I see that "%s".format(str) where str is a const(char)* just prints the pointer value in hex. So perhaps one should say that std.format just treats it like any other pointer (and not specifically that it treats it as a pointer to a single char)? > > it is a poor design. fromStringz > is explicit about this assumption. OK thank you. -- Shriramana Sharma, Penguin #395953
Re: C string to D without memory allocation?
On Monday, 21 December 2015 at 05:39:32 UTC, Rikki Cattermole wrote: size_t strLen = ...; char* ptr = ...; string myCString = cast(string)ptr[0 .. strLen]; I can't remember if it will include the null terminator or not, but if it does just decrease strLen by 1. Strings from C libraries shouldn't be casted to immutable. If the characters of the C string are truly immutable, mark it as immutable(char)* in the binding (and needless to say, use fromStringz instead of explicit counting when the length isn't known).
Re: C string to D without memory allocation?
On Monday, 21 December 2015 at 05:43:04 UTC, Rikki Cattermole wrote: On 21/12/15 6:41 PM, Shriramana Sharma wrote: Rikki Cattermole wrote: string myCString = cast(string)ptr[0 .. strLen]; Thanks but does this require that one doesn't attempt to append to the returned string using ~= or such? In which case it is not safe, right? Correct, ~= should only be used by GC controlled memory. Which this is not. This is just plain wrong. No idea where you got this from.
Re: C string to D without memory allocation?
Rikki Cattermole wrote: > string myCString = cast(string)ptr[0 .. strLen]; Thanks but does this require that one doesn't attempt to append to the returned string using ~= or such? In which case it is not safe, right? -- Shriramana Sharma, Penguin #395953
Re: C string to D without memory allocation?
On 21/12/15 6:41 PM, Shriramana Sharma wrote: Rikki Cattermole wrote: string myCString = cast(string)ptr[0 .. strLen]; Thanks but does this require that one doesn't attempt to append to the returned string using ~= or such? In which case it is not safe, right? Correct, ~= should only be used by GC controlled memory. Which this is not.
Re: C string to D without memory allocation?
On Monday, 21 December 2015 at 05:41:31 UTC, Shriramana Sharma wrote: Rikki Cattermole wrote: string myCString = cast(string)ptr[0 .. strLen]; Thanks but does this require that one doesn't attempt to append to the returned string using ~= or such? In which case it is not safe, right? Growing operations like ~= will copy the array to a GC-allocated, druntime-managed array if it isn't one already.
Re: C string to D without memory allocation?
On Monday, 21 December 2015 at 05:34:07 UTC, Shriramana Sharma
wrote:
Hello. I have the following code:
import std.stdio, std.conv;
extern(C) const(char) * textAttrN(const (char) * specString,
size_t n);
string textAttr(const(char)[] specString)
{
const(char) * ptr = textAttrN(specString.ptr,
specString.length);
writeln(ptr);
return to!string(ptr);
}
void main()
{
auto s = textAttr("w /g");
writeln(s.ptr);
}
Now I'm getting different pointer values printed, like:
7F532A85A440
7F532A954000
Is it possible to get D to create a D string from a C string
but not allocate memory?
I thought perhaps the allocation is because C does not
guarantee immutability but a D string has to. So I tried
changing the return type of textAttr to const(char)[] but I
find it is still allocating for the return value. Is this
because a slice can potentially be appended to but it may
overflow a C buffer?
Finally, I just want to return a safe D type encapsulating a C
string but avoid allocation – is it possible or not?
Thanks!
Use std.string.fromStringz. to!string assumes that pointers to
characters are null-terminated strings which is not safe or
general (unlike std.format, which safely assumes they are
pointers to single characters); it is a poor design. fromStringz
is explicit about this assumption.
That said, to!string shouldn't allocate when given
immutable(char)*.
Re: C string to D without memory allocation?
On 21/12/15 6:34 PM, Shriramana Sharma wrote:
Hello. I have the following code:
import std.stdio, std.conv;
extern(C) const(char) * textAttrN(const (char) * specString, size_t n);
string textAttr(const(char)[] specString)
{
const(char) * ptr = textAttrN(specString.ptr, specString.length);
writeln(ptr);
return to!string(ptr);
}
void main()
{
auto s = textAttr("w /g");
writeln(s.ptr);
}
Now I'm getting different pointer values printed, like:
7F532A85A440
7F532A954000
Is it possible to get D to create a D string from a C string but not
allocate memory?
I thought perhaps the allocation is because C does not guarantee
immutability but a D string has to. So I tried changing the return type of
textAttr to const(char)[] but I find it is still allocating for the return
value. Is this because a slice can potentially be appended to but it may
overflow a C buffer?
Finally, I just want to return a safe D type encapsulating a C string but
avoid allocation – is it possible or not?
Thanks!
size_t strLen = ...;
char* ptr = ...;
string myCString = cast(string)ptr[0 .. strLen];
I can't remember if it will include the null terminator or not, but if
it does just decrease strLen by 1.
C string to D without memory allocation?
Hello. I have the following code:
import std.stdio, std.conv;
extern(C) const(char) * textAttrN(const (char) * specString, size_t n);
string textAttr(const(char)[] specString)
{
const(char) * ptr = textAttrN(specString.ptr, specString.length);
writeln(ptr);
return to!string(ptr);
}
void main()
{
auto s = textAttr("w /g");
writeln(s.ptr);
}
Now I'm getting different pointer values printed, like:
7F532A85A440
7F532A954000
Is it possible to get D to create a D string from a C string but not
allocate memory?
I thought perhaps the allocation is because C does not guarantee
immutability but a D string has to. So I tried changing the return type of
textAttr to const(char)[] but I find it is still allocating for the return
value. Is this because a slice can potentially be appended to but it may
overflow a C buffer?
Finally, I just want to return a safe D type encapsulating a C string but
avoid allocation – is it possible or not?
Thanks!
--
Shriramana Sharma, Penguin #395953
some memory allocation/GC benchmarks here - fwiw
(not translated into D yet) http://blog.mgm-tp.com/2013/12/benchmarking-g1-and-other-java-7-garbage-collectors/ http://www.mm-net.org.uk/resources/benchmarks.html http://www.ccs.neu.edu/home/will/GC/sourcecode.html http://yoda.arachsys.com/csharp/benchmark.html it's possible we already have better ones in phobos test suite, but maybe interesting also to be able to compare to other languages/platforms.
Re: How to get nogc to work with manual memory allocation
On Friday, 5 September 2014 at 06:43:56 UTC, monarch_dodra wrote: On Sunday, 24 August 2014 at 09:29:53 UTC, Jacob Carlborg wrote: On 2014-08-24 10:03, Bienlein wrote: I have omitted the code for the TestClass class to save space. Problem is that the compiler outputs this: Error: @nogc function 'main.nogcNew!(TestClass, ).nogcNew' cannot call non-@nogc function 'core.exception.onOutOfMemoryError' Error: @nogc function 'main.nogcNew!(TestClass, ).nogcNew' cannot call non-@nogc function 'std.conv.emplace!(TestClass, ).emplace' Error: @nogc function 'main.nogcDel!(TestClass).nogcDel' cannot call non-@nogc function 'object.destroy!(TestClass).destroy' Is there a way to get around this? @nogc is a very new attribute. The runtime and standard library have not been properly annotated with this attribute yet. As a workaround you could try copy implementation of these functions to you're own code and annotate them as appropriate. The real issue here is actually the *language*. Exceptions and Errors are GC allocated, so if you try to "check" your no-GC allocation, you'll use the GC... A possible workaround is to preemptively static allocate the Error. However, this can cause issues if you chain Errors, or re-enter your function while throwing. It was also discussed to exempt `new Error` (and assert) from the @nogc restriction.
Re: How to get nogc to work with manual memory allocation
On Sunday, 24 August 2014 at 09:29:53 UTC, Jacob Carlborg wrote: On 2014-08-24 10:03, Bienlein wrote: I have omitted the code for the TestClass class to save space. Problem is that the compiler outputs this: Error: @nogc function 'main.nogcNew!(TestClass, ).nogcNew' cannot call non-@nogc function 'core.exception.onOutOfMemoryError' Error: @nogc function 'main.nogcNew!(TestClass, ).nogcNew' cannot call non-@nogc function 'std.conv.emplace!(TestClass, ).emplace' Error: @nogc function 'main.nogcDel!(TestClass).nogcDel' cannot call non-@nogc function 'object.destroy!(TestClass).destroy' Is there a way to get around this? @nogc is a very new attribute. The runtime and standard library have not been properly annotated with this attribute yet. As a workaround you could try copy implementation of these functions to you're own code and annotate them as appropriate. The real issue here is actually the *language*. Exceptions and Errors are GC allocated, so if you try to "check" your no-GC allocation, you'll use the GC... A possible workaround is to preemptively static allocate the Error. However, this can cause issues if you chain Errors, or re-enter your function while throwing.
