Re: Why many programmers don't like GC?
On Thu, Jan 14, 2021 at 12:36:12PM +, claptrap via Digitalmars-d-learn wrote: [...] > I think you also have to consider that the GC you get with D is not > state of the art, and if the opinions expressed on the newsgroup are > accurate, it's not likely to get any better. So while you can find > examples of high performance applications, AAA games, or whatever that > use GC, I doubt any of them would be feasible with Ds GC. And given > the design choices D has made as a language, a high performance GC is > not really possible. To be fair, the GC *has* improved over the years. Just not as quickly as people would like, but it *has* improved. > So the GC is actually a poor fit for D as a language. It's like a > convertible car with a roof that is only safe up to 50 mph, go over > that and its likely to be torn off. So if you want to drive fast you > have to put the roof down. How much D code have you actually written and optimized? That analogy is inaccurate. IME, performance issues caused by the GC are generally localized, and easy to fix by replacing that small part of the code with a bit of manual memory management (you *can* rewrite a function not to use the GC; this isn't the Java straitjacket, y'know!), or standard GC optimization techniques like reducing GC load in hot loops. There's also GC.stop and GC.collect for those times when you want more control over exactly when collection pauses happen. I wrote a compute-intensive program once, and after some profiling revealed the GC being a bottleneck, I: (1) Refactored one function called from an inner loop to reuse a buffer instead of allocating a new one each time, thus eliminating a large amoun of garbage from small allocations; (2) Used GC.stop and scheduled my own GC.collect with slightly reduced frequency. The result was about 40-50% reduction in runtime, which is close to about a 2x speedup. Now, you'll argue that had I written this code without a GC in the first place I wouldn't have needed to do all this. However: (a) Because I *had* the GC, I could write this code in about 1/5 of the time it would've taken me to write it in C++; (b) The optimization involved only changing a couple of lines of code in 2-3 functions -- a couple of days' work at most -- as opposed to blindly optimizing *every* single danged line of code, 95% of which wouldn't even have had any noticeable effect because they *are not the bottleneck*; (c) The parts of the code that aren't in the hot path can still freely take advantage of the GC require minimal effort to write, and be free of the time-consuming bugs that often creep into code that manually manages memory. As I said, it's an ROI question. I *could* have spent 5x the amount of time and effort to write the perfect, GC-less, macho-hacker-style code, and get maybe about a 1-2% performance improvement. But why would I? It takes 5x less effort to write GC code, and requires only a couple more days of effort to fix GC-related performance issues, vs. 5x the development effort to write the entire program GC-less, and who knows how much longer after that to debug obscure pointer bugs. Life is too short to be squandered chasing down the 1000th double-free and the 20,000th dangling pointer in my life. A lot of naysayers keep repeating GC performance issues as if it's a black-and-white, all-or-nothing question. It's not. You *can* write high-performance programs even with D's supposedly lousy GC -- just profile the darned thing, and refactor the hotspots to reduce GC load or avoid the GC. *In those parts of the code that actually matter*. You don't have to do this for the *entire* lousy program. The non-hot parts of the code can still GC away like there's no tomorrow, and your customers would hardly notice a difference. This isn't Java where you have no choice but to use the GC everywhere. Another example: one day I had some spare time, and wrote fastcsv (http://github.com/quickfur/fastcsv). It's an order of magnitude faster than std.csv, *and it uses the evil GC*. I just applied the same technique: write it with GC, then profile to find the bottlenecks. The first round of profiling showed that there tend to be a lot of small allocations, which create lots of garbage, which means slow collection cycles. The solution? Use a linear buffer instead of individual allocations for field/row data, and use slices where possible instead of copying the data. By reducing GC load and minimizing copying, I got huge boosts in performance -- without throwing out the GC with the bathwater. (And note: it's *because* I can rely on the GC that I can use slices so freely; if I had to use RC or manage this stuff manually, it'd take 5x longer to write and would involve copying data all over the place, which means it'd probably lose out in overall performance.) But then again, it's futile to argue with people who have already made up their minds about the GC, so meh. Let the bystanders judge for themselves. I'll shut
Re: Directory recursive walking
On Friday, 15 January 2021 at 07:16:21 UTC, Daniel Kozak wrote:
On Fri, Jan 15, 2021 at 8:00 AM dog2002 via Digitalmars-d-learn
< digitalmars-d-learn@puremagic.com> wrote:
On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote:
> On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus
> wrote:
>>
>> You can save a little bit of memory here by allocating
>> tempBuffer on the stack:
>>
>> ubyte[512] tempBuffer;
>> _inputFile.rawRead(tempBuffer[]); // note the explicit
>> []
>
> I made a mistake; this should be:
>
> ubyte[512] tempArray;
> ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]);
>
> ...with the rest the same as your original version.
Thank you so much! It saves a lot of memory!
And one last question: why the application crashes, if I
allocate 1 MB array?
>ubyte[1024000] tempBuffer;
Because of stack overflow
A compiler parameter can be used to increase the maximum stack
size
"dflags": ["-L/STACK:15"]
or recursion can be somehow emulated using heap memory. Here is
my "fake" recursion:
// wins is a range
auto stack = wins.save;
while(!stack.empty){
immutable n = stack.length - 1;
auto window = stack[n];
doSomeThingforEachRecursiveElement(window)
stack.popBack;
if(window.children.length){
foreach (ref child; window.children)
stack.pushBack(child);
}
}
stack.free;
Re: Directory recursive walking
On Fri, Jan 15, 2021 at 8:20 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > On Friday, 15 January 2021 at 06:56:36 UTC, dog2002 wrote: > > On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote: > >> On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote: > >>> > >>> You can save a little bit of memory here by allocating > >>> tempBuffer on the stack: > >>> > >>> ubyte[512] tempBuffer; > >>> _inputFile.rawRead(tempBuffer[]); // note the explicit [] > >> > >> I made a mistake; this should be: > >> > >> ubyte[512] tempArray; > >> ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]); > >> > >> ...with the rest the same as your original version. > > > > Thank you so much! It saves a lot of memory! > > > > And one last question: why the application crashes, if I > > allocate 1 MB array? > > > >>ubyte[1024000] tempBuffer; > > Solved: > > ubyte[] tempBuffer = new ubyte[1024000]; > You can still use ubyte[1024000] tempBuffer; but you have to place it somewhere outside recursion or use a static static ubyte[1024000] tempBuffer;
Re: Directory recursive walking
On Friday, 15 January 2021 at 06:56:36 UTC, dog2002 wrote: On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote: On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote: You can save a little bit of memory here by allocating tempBuffer on the stack: ubyte[512] tempBuffer; _inputFile.rawRead(tempBuffer[]); // note the explicit [] I made a mistake; this should be: ubyte[512] tempArray; ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]); ...with the rest the same as your original version. Thank you so much! It saves a lot of memory! And one last question: why the application crashes, if I allocate 1 MB array? ubyte[1024000] tempBuffer; Solved: ubyte[] tempBuffer = new ubyte[1024000];
Re: Directory recursive walking
On Fri, Jan 15, 2021 at 8:00 AM dog2002 via Digitalmars-d-learn < digitalmars-d-learn@puremagic.com> wrote: > On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote: > > On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote: > >> > >> You can save a little bit of memory here by allocating > >> tempBuffer on the stack: > >> > >> ubyte[512] tempBuffer; > >> _inputFile.rawRead(tempBuffer[]); // note the explicit [] > > > > I made a mistake; this should be: > > > > ubyte[512] tempArray; > > ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]); > > > > ...with the rest the same as your original version. > > Thank you so much! It saves a lot of memory! > > And one last question: why the application crashes, if I allocate > 1 MB array? > > >ubyte[1024000] tempBuffer; > Because of stack overflow
Re: Directory recursive walking
On Friday, 15 January 2021 at 06:33:55 UTC, Paul Backus wrote: On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote: You can save a little bit of memory here by allocating tempBuffer on the stack: ubyte[512] tempBuffer; _inputFile.rawRead(tempBuffer[]); // note the explicit [] I made a mistake; this should be: ubyte[512] tempArray; ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]); ...with the rest the same as your original version. Thank you so much! It saves a lot of memory! And one last question: why the application crashes, if I allocate 1 MB array? ubyte[1024000] tempBuffer;
Re: Directory recursive walking
On Friday, 15 January 2021 at 06:31:18 UTC, Paul Backus wrote: You can save a little bit of memory here by allocating tempBuffer on the stack: ubyte[512] tempBuffer; _inputFile.rawRead(tempBuffer[]); // note the explicit [] I made a mistake; this should be: ubyte[512] tempArray; ubyte[] tempBuffer = _inputFile.rawRead(tempArray[]); ...with the rest the same as your original version.
Re: Directory recursive walking
On Friday, 15 January 2021 at 06:15:06 UTC, dog2002 wrote:
void func(string inputFile, string outFile, uint chunk_size) {
try {
File _inputFile = File(inputFile, "r");
File _outputFile = File(outFile, "w");
ubyte[] tempBuffer = _inputFile.rawRead(new ubyte[](512));
//doing some operations with the tempBuffer
_outputFile.rawWrite(tempBuffer);
_inputFile.seek(tempBuffer.length, SEEK_SET);
foreach(_buffer; _inputFile.byChunk(chunk_size)) {
_outputFile.rawWrite(_buffer);
}
_inputFile.close();
_outputFile.close();
}
catch (Throwable) {}
}
You can save a little bit of memory here by allocating tempBuffer
on the stack:
ubyte[512] tempBuffer;
_inputFile.rawRead(tempBuffer[]); // note the explicit []
// ...
_outputFile.rawWrite(tempBuffer[]);
However, those allocations alone shouldn't be enough to get you
to 4GB+, so the real issue is probably elsewhere.
Re: Directory recursive walking
On Thursday, 14 January 2021 at 22:28:19 UTC, Paul Backus wrote:
On Thursday, 14 January 2021 at 20:23:37 UTC, dog2002 wrote:
About 1000 large files.
I want to replace several first bytes in all the files, so I
just copy the remaining bytes into a new file. Might this be
the reason for high memory consumption? If so, is there a way
not to copy the entire file, just delete first bytes and write
the replaced bytes into the beginning of the file?
I use Windows x64.
What code are you using to copy the bytes? If you're reading
the whole file into memory at once, that will consume a lot of
memory.
void func(string inputFile, string outFile, uint chunk_size) {
try {
File _inputFile = File(inputFile, "r");
File _outputFile = File(outFile, "w");
ubyte[] tempBuffer = _inputFile.rawRead(new ubyte[](512));
//doing some operations with the tempBuffer
_outputFile.rawWrite(tempBuffer);
_inputFile.seek(tempBuffer.length, SEEK_SET);
foreach(_buffer; _inputFile.byChunk(chunk_size)) {
_outputFile.rawWrite(_buffer);
}
_inputFile.close();
_outputFile.close();
}
catch (Throwable) {}
}
Re: Open question: what code pattern you use usually for null safety problem
On Thursday, 14 January 2021 at 20:35:49 UTC, Dennis wrote:
On Thursday, 14 January 2021 at 18:24:44 UTC, ddcovery wrote:
If it's not a bother, I'd like to know how you usually
approach it
Usually I don't deal with null because my functions get
primitive types, slices, or structs. `ref` parameters can be
used to replace pointers that may not be null.
When something is nullable by design, I usually do this:
```
if (!person) {
return; // early return if possible
}
if (auto f0 = person.father) {
if (auto f1 = f0.father) {
if (f1.name == "Peter") {
doSomething();
}
}
}
```
It doesn't matter whether you're working with a class, pointer,
or struct with opCast, this works. When access patterns get
complex the nesting may get very deep.
Only if you can't avoid this I would consider using fancy
helper functions, otherwise just use an if-statement or the &&
operator.
I agree: using null safety is a sign of something wrong in the
design (the need of dealing with nulls)... but if eventually you
need it, simple **if** or **&&** should be enough.
Curiously, languages like Dart (and its flutter framework)
performs extensive use of null safety (null is everywhere!!!) and
it seems that every "modern" language must deal with it.
Any case, I'm learning a lot: thank you Dennis for sharing!!!
Re: Open question: what code pattern you use usually for null safety problem
On Thursday, 14 January 2021 at 20:23:08 UTC, Steven
Schveighoffer wrote:
You could kinda automate it like:
struct NullCheck(T)
{
private T* _val;
auto opDispatch(string mem)() if (__traits(hasMember, T,
mem)) {
alias Ret = typeof(() { return __traits(getMember,
*_val, mem); }());
if(_val is null) return NullCheck!(Ret)(null);
else return NullCheck!(Ret)(__trats(getMember, *_val,
mem));
}
bool opCast(V: bool)() { return _val !is null; }
}
auto nullCheck(T)(T *val) { return AutoNullCheck!T(val);}
// usage
if(nullCheck(person).father.father && person.father.father.name
== "Peter")
Probably doesn't work for many circumstances, and I'm sure I
messed something up.
-Steve
I'm seeing "opDispatch" everywhere last days :-). It's really
powerful!!!
If we define an special T _(){ return _val; } method, then you
can write
if( nullCheck(person).father.father.name._ == "Peter")
And renaming
if( ns(person).father.father.name._ == "Peter" )
And adding some extra check like ** isAssignable!(Ret,
typeof(null) )** we can add special treatment for not nullable
types
if( ns(person).father.father.age._(0) == 92 ){ ... }
assert( ns(person).father.father.father.age._ == int.init );
If for some strange reason I ever need null safety, I think this
is the most beautiful solution or at least a great lesson on
templates.
Thanks a lot for the lesson, Steve
Re: Directory recursive walking
On Thursday, 14 January 2021 at 20:23:37 UTC, dog2002 wrote: About 1000 large files. I want to replace several first bytes in all the files, so I just copy the remaining bytes into a new file. Might this be the reason for high memory consumption? If so, is there a way not to copy the entire file, just delete first bytes and write the replaced bytes into the beginning of the file? I use Windows x64. What code are you using to copy the bytes? If you're reading the whole file into memory at once, that will consume a lot of memory.
Re: Anything in D to avoid check for null everywhere?
On Thursday, 14 January 2021 at 21:49:41 UTC, Christian Köstlin wrote: ... Did you have a look at https://code.dlang.org/packages/optional? Especially https://aliak00.github.io/optional/optional/oc/oc.html might go in the right direction. Kind regards, Christian Thats nice!!! I was commenting the need of a MayBe (some/none) monad compatible with Ranges (as an alternative to Nullable!T ) in https://forum.dlang.org/post/gxapatzfkoigcdhrd...@forum.dlang.org Thanks a lot Christian.
Re: Open question: what code pattern you use usually for null safety problem
On Thursday, 14 January 2021 at 19:24:54 UTC, Adam D. Ruppe wrote:
On Thursday, 14 January 2021 at 18:24:44 UTC, ddcovery wrote:
This is only an open question to know what code patterns you
usually use to solve this situation in D:
I'm almost never in this situation except for reading things
like xml or json data that may be missing.
Yes, this is the usual situation (Personally, I use "DTO"
structured objects... that are serialized/unserialized to JSON)
So I just special cased those. My json lib doesn't return null
per se, it returns var(null) which is allowed to just return
more harmless nulls. Thus you write
`person.father.father.name.get!string` and it will be empty if
anything was null in the chain.
With dom, you can optionSelector("person > father > father >
name").innerText and again if the selector returned null, all
its methods also just return empty strings or whatever.
Selectors are a good option to navigate on dom/json, but on
structured objects too. The good think with "templates" in D is
that this "path/selector" can be compiled internally to a
map/filter combination completly "null" free... I was
experimenting last days with this and I think that a functional
orientation (using a MayBe monad implemented as Range ) is the
best way to begin.
So the library handles these special cases and then I don't
worry about nested nulls anywhere else since I consider it bad
style to even be in that situation in the first place.
I agree: it is a bad style. Personally I allways use MayBe monads
in my "DTO"s (that is the effect of having worked with scala :-).
The only "cons" with Nullable!T (the "standard" D MayBe
equivalent) is that it is not "compatible" with Range libraries
(it is not a Range: you use "apply" instead "map", you have not
"filter", you can't "chain" a range and Nullable, you can't
"join" a range of Nullables like a Range of ranges). This is the
reason I'm "experimenting" with my own "MayBe" InputRange that
can be created in the form of "Some" or "None" (it's inmutable
contrary to what happens with Nullable) and is compatible with
all std.algorithm (and array) library.
Re: Anything in D to avoid check for null everywhere?
On 12.01.21 22:37, Jack wrote: I was looking for a way to avoid null checks everywhere. I was checking the Null object pattern, or use something like enforce pattern, or even if I could make a new operator and implement something like C#'s .? operator, that Java was going to have one but they refused[1] (doesn't behave exactly as C#'s actually), Kotlin also got something in this area[2] What some D ways to avoid those checks? [1]: https://mail.openjdk.java.net/pipermail/coin-dev/2009-March/47.html [2]: https://kotlinlang.org/docs/reference/null-safety.html#safe-calls Did you have a look at https://code.dlang.org/packages/optional? Especially https://aliak00.github.io/optional/optional/oc/oc.html might go in the right direction. Kind regards, Christian
Re: Open question: what code pattern you use usually for null safety problem
On Thursday, 14 January 2021 at 18:24:44 UTC, ddcovery wrote:
If it's not a bother, I'd like to know how you usually approach
it
Usually I don't deal with null because my functions get primitive
types, slices, or structs. `ref` parameters can be used to
replace pointers that may not be null.
When something is nullable by design, I usually do this:
```
if (!person) {
return; // early return if possible
}
if (auto f0 = person.father) {
if (auto f1 = f0.father) {
if (f1.name == "Peter") {
doSomething();
}
}
}
```
It doesn't matter whether you're working with a class, pointer,
or struct with opCast, this works. When access patterns get
complex the nesting may get very deep.
Only if you can't avoid this I would consider using fancy helper
functions, otherwise just use an if-statement or the && operator.
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 15:18:28 UTC, ddcovery wrote: I understand perfectly the D community people that needs to work without GC: **it is not snobbish**: it is a real need. But not only a "need"... sometimes it is basically the way a team wants to work: explicit memory management vs GC. D already supports manual memory management so that escape hatch was always there. My main criticism of D is the inability to freely exchange the GC algorithms as one type of GC might not be the best fit for everyone. The problem is of course that there is no differentiation between raw and fat pointers. With fat pointers, the community would have a better opportunities to experiment with different GC designs which would lead to a larger palette of GC algorithms.
Re: Directory recursive walking
On Thursday, 14 January 2021 at 16:47:45 UTC, drug wrote: On 1/14/21 7:06 PM, dog2002 wrote: On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote: On 1/14/21 6:55 PM, drug wrote: But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory? DirEntry is a struct. First of all I would try this: ```D foreach(ref entry; dirEntries(path, SpanMode.shallow, false)) ``` Does your directory just contain large amount of files? Yes. I forgot to add it in the original post. How much files do you have? DirEntry size is 168 bytes only and dirEntry is lazy range so I'm curious what is the reason of huge memory consumption. Do you use Windows 32 bits between? About 1000 large files. I want to replace several first bytes in all the files, so I just copy the remaining bytes into a new file. Might this be the reason for high memory consumption? If so, is there a way not to copy the entire file, just delete first bytes and write the replaced bytes into the beginning of the file? I use Windows x64.
Re: Open question: what code pattern you use usually for null safety problem
On 1/14/21 1:24 PM, ddcovery wrote:
I know there is other threads about null safety and the "possible" ways
to support this in D and so on.
This is only an open question to know what code patterns you usually use
to solve this situation in D:
if(person.father.father.name == "Peter") doSomething();
if(person.father.age > 80 ) doSomething();
knowing that *person*, or its *father* property can be null
i.e.: the incremental null check solution
if(
person !is null &&
person.father !is null &&
person.father.father !is null &&
person.father.father.name == "Peter"
)
{
doSomething();
}
or the "monad" way
[person].
filter!"a !is null".map!"a.father".
filter!"a !is null".map!"a.father".
filter!"a !is null".map!"a.name".
each!( (name) {
if(name == "Peter") doSomething();
});
or, may be, you have some helper function/structs/templates
if( person.d!"father".d!"father".d!"name".get == "Peter" ){
doSomething()
}
if( person.d!"father".d!"age".get(0) > 80 ){
doSomething()
}
or an "xml path" like template
if( person.get!"father.father.name" == "Peter" )
if( person.get!"father.father.name.length"(0) == 5 )
if( person.get!"father.father.age"(0) > 80 )
If it's not a bother, I'd like to know how you usually approach it
Thanks!!!
You could kinda automate it like:
struct NullCheck(T)
{
private T* _val;
auto opDispatch(string mem)() if (__traits(hasMember, T, mem)) {
alias Ret = typeof(() { return __traits(getMember, *_val, mem);
}());
if(_val is null) return NullCheck!(Ret)(null);
else return NullCheck!(Ret)(__trats(getMember, *_val, mem));
}
bool opCast(V: bool)() { return _val !is null; }
}
auto nullCheck(T)(T *val) { return AutoNullCheck!T(val);}
// usage
if(nullCheck(person).father.father && person.father.father.name == "Peter")
Probably doesn't work for many circumstances, and I'm sure I messed
something up.
-Steve
Re: Open question: what code pattern you use usually for null safety problem
On Thursday, 14 January 2021 at 18:24:44 UTC, ddcovery wrote:
I know there is other threads about null safety and the
"possible" ways to support this in D and so on.
This is only an open question to know what code patterns you
usually use to solve this situation in D:
if(person.father.father.name == "Peter") doSomething();
if(person.father.age > 80 ) doSomething();
knowing that *person*, or its *father* property can be null
i.e.: the incremental null check solution
I just use this most simple one:
if(
person !is null &&
person.father !is null &&
person.father.father !is null &&
person.father.father.name == "Peter"
)
{
doSomething();
}
Reason: easy to read and reason about, esp for non-authors of
this piece of the code.
Re: Open question: what code pattern you use usually for null safety problem
On Thursday, 14 January 2021 at 18:24:44 UTC, ddcovery wrote:
This is only an open question to know what code patterns you
usually use to solve this situation in D:
I'm almost never in this situation except for reading things like
xml or json data that may be missing.
So I just special cased those. My json lib doesn't return null
per se, it returns var(null) which is allowed to just return more
harmless nulls. Thus you write
`person.father.father.name.get!string` and it will be empty if
anything was null in the chain.
With dom, you can optionSelector("person > father > father >
name").innerText and again if the selector returned null, all its
methods also just return empty strings or whatever.
So the library handles these special cases and then I don't worry
about nested nulls anywhere else since I consider it bad style to
even be in that situation in the first place.
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 18:10:43 UTC, mw wrote: Python's `del` isn't guaranteed to free the memory, that's what Fair point, but I was thinking of the C interop interface. You can create your own wrapper (e.g. numpy) and do manual memory management, but it isn't something people want to do! It is mostly pointless to do that within Python because of the existing overhead. That applies to most high level languages; you can, but it is pointless. You only do it for interop... One can follow the same kind of reasoning for D. It makes no sense for people who want to stay high level and do batch programming. Which is why this disconnect exists in the community... I think.
Re: Static constructor
On Tuesday, 12 January 2021 at 11:28:12 UTC, ludo wrote: Ok, I agree that ends up being a kind of strange singleton. But yes it was D v1 code. Do we agree that the following multi-threaded singleton pattern is the proper way as of today. It looks fine to me. The D wiki has the following example which prevents the need for entering the synchronized block more than once per thread: https://wiki.dlang.org/Low-Lock_Singleton_Pattern I should note that the example doesn't specify the value as shared, which technically isn't the correct thing to do, but `shared` itself can be a bit of an annoyance. However, you do need to ensure that your static variable is either shared or __gshared, otherwise it becomes thread-local. If I understand, as of today an AA init depends on a runtime function for historical reasons. A bit weird for an array indeed perfectly known at compile time, but someday some core language contributor will have a look at it, I guess. Maybe in 5 years' time we'll get an uneventful discussion on it where it'd take 2 years to come to a conclusion before another year for it to actually be implemented. If I understand well, this will accomplish the goal quoted with no further comestics needed! Only one keyword over explicitely using object mutex. Am I right? I believe so. I've never used OpenAL so it may have additional restrictions with multithreading, but from a simple "This function is only ever executed on one thread at a time", your above suggestions should work. Apologies for the late reply.
Open question: what code pattern you use usually for null safety problem
I know there is other threads about null safety and the
"possible" ways to support this in D and so on.
This is only an open question to know what code patterns you
usually use to solve this situation in D:
if(person.father.father.name == "Peter") doSomething();
if(person.father.age > 80 ) doSomething();
knowing that *person*, or its *father* property can be null
i.e.: the incremental null check solution
if(
person !is null &&
person.father !is null &&
person.father.father !is null &&
person.father.father.name == "Peter"
)
{
doSomething();
}
or the "monad" way
[person].
filter!"a !is null".map!"a.father".
filter!"a !is null".map!"a.father".
filter!"a !is null".map!"a.name".
each!( (name) {
if(name == "Peter") doSomething();
});
or, may be, you have some helper function/structs/templates
if( person.d!"father".d!"father".d!"name".get == "Peter" ){
doSomething()
}
if( person.d!"father".d!"age".get(0) > 80 ){
doSomething()
}
or an "xml path" like template
if( person.get!"father.father.name" == "Peter" )
if( person.get!"father.father.name.length"(0) == 5 )
if( person.get!"father.father.age"(0) > 80 )
If it's not a bother, I'd like to know how you usually approach it
Thanks!!!
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 09:26:06 UTC, Ola Fosheim Grøstad wrote: On Thursday, 14 January 2021 at 00:37:29 UTC, mw wrote: ok, what I really mean is: ... in other "(more popular) languages (than D, and directly supported by the language & std library only)" ... Well, even Python supports both Python's `del` isn't guaranteed to free the memory, that's what we are discussing here: core.memory.GC.free / core.stdc.stdlib.free https://www.quora.com/Why-doesnt-Python-release-the-memory-when-I-delete-a-large-object In CPython (the default reference distribution), the Garbage collection in Python is not guaranteed to run when you delete the object - all del (or the object going out of scope) does is decrement the reference count on the object. The memory used by the object is not guaranteed to be freed and returned to the processes pool at any time before the process exits. Even if the Garbage collection does run - all it needs is another object referencing the deleted object and the garbage collection won’t free the object at all.
Re: Directory recursive walking
On 1/14/21 7:06 PM, dog2002 wrote: On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote: On 1/14/21 6:55 PM, drug wrote: But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory? DirEntry is a struct. First of all I would try this: ```D foreach(ref entry; dirEntries(path, SpanMode.shallow, false)) ``` Does your directory just contain large amount of files? Yes. I forgot to add it in the original post. How much files do you have? DirEntry size is 168 bytes only and dirEntry is lazy range so I'm curious what is the reason of huge memory consumption. Do you use Windows 32 bits between?
Re: Directory recursive walking
On 1/14/21 7:30 PM, dog2002 wrote:
On Thursday, 14 January 2021 at 16:18:28 UTC, drug wrote:
On 1/14/21 7:06 PM, dog2002 wrote:
On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote:
[...]
Yes. I forgot to add it in the original post.
Does using `ref` changed anything?
Try following:
```
import std;
void DirIteration(ref DirEntry dir) {
try {
foreach(ref entry; dirEntries(dir, SpanMode.shallow, false)) {
//SpanMode.shallow allows skip directories if any error happens
if (entry.isFile && !entry.isSymlink)
writeln(entry); //Or something instead of this
if (entry.isDir)
DirIteration(entry);
}
}
catch (Throwable) {}
}
void main()
{
auto de = DirEntry(".");
DirIteration(de);
}
```
No, it doesn't. Seems like memory can't clear.
It is a recursion. Memory will be freed only after completion. Then I
would try to get rid of recursion.
Re: Directory recursive walking
On Thursday, 14 January 2021 at 16:18:28 UTC, drug wrote:
On 1/14/21 7:06 PM, dog2002 wrote:
On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote:
[...]
Yes. I forgot to add it in the original post.
Does using `ref` changed anything?
Try following:
```
import std;
void DirIteration(ref DirEntry dir) {
try {
foreach(ref entry; dirEntries(dir, SpanMode.shallow,
false)) { //SpanMode.shallow allows skip directories if any
error happens
if (entry.isFile && !entry.isSymlink)
writeln(entry); //Or something instead of this
if (entry.isDir)
DirIteration(entry);
}
}
catch (Throwable) {}
}
void main()
{
auto de = DirEntry(".");
DirIteration(de);
}
```
No, it doesn't. Seems like memory can't clear.
Re: Directory recursive walking
On 1/14/21 7:06 PM, dog2002 wrote:
On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote:
On 1/14/21 6:55 PM, drug wrote:
But this method consumes a huge amount of memory (up to 4 GB and
more). Is there a more appropriate way to walk directories
recursively that does not consume a lot of memory?
DirEntry is a struct. First of all I would try this:
```D
foreach(ref entry; dirEntries(path, SpanMode.shallow, false))
```
Does your directory just contain large amount of files?
Yes. I forgot to add it in the original post.
Does using `ref` changed anything?
Try following:
```
import std;
void DirIteration(ref DirEntry dir) {
try {
foreach(ref entry; dirEntries(dir, SpanMode.shallow, false)) {
//SpanMode.shallow allows skip directories if any error happens
if (entry.isFile && !entry.isSymlink)
writeln(entry); //Or something instead of this
if (entry.isDir)
DirIteration(entry);
}
}
catch (Throwable) {}
}
void main()
{
auto de = DirEntry(".");
DirIteration(de);
}
```
Re: Directory recursive walking
On Thursday, 14 January 2021 at 16:01:43 UTC, drug wrote: On 1/14/21 6:55 PM, drug wrote: But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory? DirEntry is a struct. First of all I would try this: ```D foreach(ref entry; dirEntries(path, SpanMode.shallow, false)) ``` Does your directory just contain large amount of files? Yes. I forgot to add it in the original post.
Re: Directory recursive walking
On 1/14/21 6:55 PM, drug wrote: But this method consumes a huge amount of memory (up to 4 GB and more). Is there a more appropriate way to walk directories recursively that does not consume a lot of memory? DirEntry is a struct. First of all I would try this: ```D foreach(ref entry; dirEntries(path, SpanMode.shallow, false)) ``` Does your directory just contain large amount of files?
Re: Directory recursive walking
On 1/14/21 6:46 PM, dog2002 wrote:
I need to make some operations with all the files in a directory and
subdirectories. Currently, I do it like this:
import std;
void DirIteration(string path) {
try {
foreach(entry; dirEntries(path, SpanMode.shallow, false)) {
//SpanMode.shallow allows skip directories if any error happens
if (entry.isFile && !entry.isSymlink)
writeln(entry); //Or something instead of this
if (entry.isDir)
DirIteration(entry);
}
}
catch (Throwable) {}
}
void main()
{
DirIteration("C:\\Users\\angrypuppy\\MyDir");
}
But this method consumes a huge amount of memory (up to 4 GB and more).
Is there a more appropriate way to walk directories recursively that
does not consume a lot of memory?
DirEntry is a struct. First of all I would try this:
```D
foreach(ref entry; dirEntries(path, SpanMode.shallow, false))
```
Directory recursive walking
I need to make some operations with all the files in a directory
and subdirectories. Currently, I do it like this:
import std;
void DirIteration(string path) {
try {
foreach(entry; dirEntries(path, SpanMode.shallow, false))
{ //SpanMode.shallow allows skip directories if any error happens
if (entry.isFile && !entry.isSymlink)
writeln(entry); //Or something instead of this
if (entry.isDir)
DirIteration(entry);
}
}
catch (Throwable) {}
}
void main()
{
DirIteration("C:\\Users\\angrypuppy\\MyDir");
}
But this method consumes a huge amount of memory (up to 4 GB and
more). Is there a more appropriate way to walk directories
recursively that does not consume a lot of memory?
Re: Why doesn't this work when the function is a static method?
On Thursday, 14 January 2021 at 15:20:54 UTC, Jack wrote: On Thursday, 14 January 2021 at 09:13:27 UTC, evilrat wrote: On Thursday, 14 January 2021 at 05:44:43 UTC, Jack wrote: On Wednesday, 13 January 2021 at 17:21:23 UTC, Paul Backus wrote: Member functions (including static ones) can't be called with UFCS. is this documented somewhere? Is this going to change? It will stay as is. It is somewhat vaguely described in p.7 under UFCS section in functions https://dlang.org/spec/function.html#pseudo-member I see, thanks If it's really important you can make a module-level alias to the static method. https://run.dlang.io/is/4IFsjr
Re: Why doesn't this work when the function is a static method?
On Thursday, 14 January 2021 at 09:13:27 UTC, evilrat wrote: On Thursday, 14 January 2021 at 05:44:43 UTC, Jack wrote: On Wednesday, 13 January 2021 at 17:21:23 UTC, Paul Backus wrote: Member functions (including static ones) can't be called with UFCS. is this documented somewhere? Is this going to change? It will stay as is. It is somewhat vaguely described in p.7 under UFCS section in functions https://dlang.org/spec/function.html#pseudo-member I see, thanks
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 10:28:13 UTC, Basile B. wrote: On Wednesday, 13 January 2021 at 18:58:56 UTC, Marcone wrote: I've always heard programmers complain about Garbage Collector GC. But I never understood why they complain. What's bad about GC? Semi serious answer: In the domain of hoby-ism and small companies programmers that work with statically typed languages all believe that they are super hero in the domain of memory managment. When they see "GC" they think that they are considered as 2nd grade student ^^ It's basically snobbism. Hi Basile, My experience: in 90's I worked with Pascal, C and C++ with rudimentary memory management: basically it was no difference between working with memory or files in terms of life-cycle management: you must alloc/free memory and you must open/close files. The secret for "stability" was a set of conventions to determine who was the responsible of the resource handler or memory pointer: I developed some ERP/CRMs, some multimedia products and some industrial environment applications (real time ones). At the end of 90's I began to work with VB and the COM model (that uses references counter) and I discovered that the best way to manage memory (avoiding death-locks) was treating objects as "external" unmanaged resources: The VB6 "WITH" statement was key to use ARM techniques (similar to future "using" in C#). And then arrived GC with C#, Java and Scala: I have found GC good enough for all applications and services that I have been developing last 20 years because this languages (and it's frameworks+based libraries) have never crossed certain limits: they always separated managed and unmanaged resources: developer is responsible of unmanaged resources, and Memory is managed by GC. Language itself offers you good tooling to ARM (like "using" in c#, "try-with-resources" in java, ...). Finally arrived the last actors to the scene: mainly javascript and derivatives (when working in a browser context), where developer is abstracted of how memory and resources are really managed (I can remember critical bugs in chrome like Image object memory leaks because this "abstraction"). GC has introduced a "productive" way of working removing old memory problems for large scale projects (and finally with other kind of resources in some scenarios) but, as developers/architects, we have de responsibility to recognize the limits to each technique and when it fits to our needs. After all, my opinion is that if I was to develop something like a Real Time app (industrial/medical/aeronautics/...) or a game where a large amount of objects must be mutated ~30 times per second, GC "unpredictable" or "large" time cost will be enough to stop using it. There is other reasons (like "efficient" memory management when we need to manage large amounts of memory or to run in limited memory environments). I understand perfectly the D community people that needs to work without GC: **it is not snobbish**: it is a real need. But not only a "need"... sometimes it is basically the way a team wants to work: explicit memory management vs GC. D toke the way of GC without "cutting" the relationship with C/C++ developers: I really don't have enough knowledge of the language and libraries to know the level of support that D offers to non GC based developments, but I find completely logic trying to maintain this relationship (in the basis that GC must continue been the default way of working) Sorry for my "extended", may be unnecessary, explanation (and my "poor" english :-p).
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 14:28:43 UTC, Виталий Фадеев wrote: On Wednesday, 13 January 2021 at 18:58:56 UTC, Marcone wrote: I've always heard programmers complain about Garbage Collector GC. But I never understood why they complain. What's bad about GC? How write quickly without GC ? In DMD style: never release memory! This is not an option for long-running programs though, nor for anything that otherwise uses significant amounts of memory. Better to just use the GC if unsure.
Re: Why many programmers don't like GC?
On Wednesday, 13 January 2021 at 18:58:56 UTC, Marcone wrote: I've always heard programmers complain about Garbage Collector GC. But I never understood why they complain. What's bad about GC? I like GC. How write quickly without GC ?
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 13:05:31 UTC, sighoya wrote: But this is already the case for C++ and Rust. Remembering the days back developing in C++ there were a huge amount of memory deallocation side effects because opencv's memory management differs from qt's memory management. The problem in C++ is that older frameworks have their own ways of doing things for performance reasons or because the C++ standard they started with didn't provide what they needed... And... most C++ frameworks that are big are old... If you avoid big frameworks then it gets better. Personally, I find it better to prefer encapsulating manual memory management and not to leak them outside. Yes. Most programmers don't need system level programming. So if D defines itself to not be a system level programming language then there would be room to improve a lot, but then it should move towards more high level features and prevent the usage of some low level features like untagged non-discriminating unions of pointers. Rust is more high level than D... I think.
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 13:16:16 UTC, Ola Fosheim Grøstad wrote: 1. Use "shared" to prevent GC allocated memory from entering other threads and switch to thread local GC. Then use ARC for shared. 2. Redefine language semantics/type system for a different GC model. This will break existing code. 3. Keep the existing GC for existing code and introduce ARC across the board for new code. Add a versioning statement that people can add to their libraries to tell the compiler which models they support.
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 13:10:14 UTC, sighoya wrote: On Thursday, 14 January 2021 at 13:08:06 UTC, Ola Fosheim Grøstad wrote: Because Java has a well defined virtual machine with lots of restrictions. So you're insisting this isn't possible in D? It isn't possible in a meaningful way. In system level programming languages you have to manually uphold the invariants needed to not break the GC collection algorithm. So if you change to a significantly different collection model, the needed invariants will change. The possible alternatives are: 1. Use "shared" to prevent GC allocated memory from entering other threads and switch to thread local GC. Then use ARC for shared. 2. Redefine language semantics/type system for a different GC model. This will break existing code.
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 13:08:06 UTC, Ola Fosheim Grøstad wrote: Because Java has a well defined virtual machine with lots of restrictions. So you're insisting this isn't possible in D?
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 11:11:58 UTC, Ola Fosheim Grøstad wrote: I know your response is *tongue in cheek*, but I actually find it easier to use c++11 style memory management across the board than mixing two models. But this is already the case for C++ and Rust. Remembering the days back developing in C++ there were a huge amount of memory deallocation side effects because opencv's memory management differs from qt's memory management. Just to say it was a hell. Personally, I find it better to prefer encapsulating manual memory management and not to leak them outside.
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 13:01:04 UTC, sighoya wrote: Why not offering more than one just as it is the case in Java? The advantage hereby is to adapt the GC algorithm after the program was compiled, so you can reuse the same program with different GC algorithms. Because Java has a well defined virtual machine with lots of restrictions.
Re: Why many programmers don't like GC?
As other people already mentioned, garbage collection incurs some amount of non-determinism and people working in low level areas prefer to handle things deterministically because they think non-deterministic handling of memory makes your code slow. For example rendering in gaming gets paused by the GC prolonging the whole rendering process. The conclusion arrives that code runs slow, but it doesn't. In fact, a tracing GC tries to do the opposite, in order to make the whole process faster, it releases memory in a buffer like mode often yielding faster execution in the long run, but not in the short run where it is mandatory not to introduce any kind of pauses. So, in the end, it isn't a fate about performance rather a fate of determinism vs non-determinism. Non-determinism has the potential to let the code run faster. For instance, no one really uses the proposed threading model in Rust with owned values where one thread can only work mutably on an owned value because other threads wanting to mutate that value have to wait. This works by creating a deterministic order of thread execution where each thread can only work after the other thread releases its work. This model gets often praised in Rust but the potential seems only a theoretical one. As a result you most often see the use of atomic reference counting (ARC) cluttering in codebases in Rust. The other point is the increased memory footprint because you have a runtime memory manager taking responsibility over (de)allocation which is impossible to have in some areas of limited memory systems. However, why just provide a one size fits all solution when there are plenty of GC algorithms for different kinds of problem domains? Why not offering more than one just as it is the case in Java? The advantage hereby is to adapt the GC algorithm after the program was compiled, so you can reuse the same program with different GC algorithms.
Re: Why many programmers don't like GC?
On Wednesday, 13 January 2021 at 20:06:51 UTC, H. S. Teoh wrote: On Wed, Jan 13, 2021 at 06:58:56PM +, Marcone via Digitalmars-d-learn wrote: I've always heard programmers complain about Garbage Collector GC. But I never understood why they complain. What's bad about GC? It's not merely a technical issue, but also a historical and sociological one. The perception of many people, esp. those with C/C++ background, is heavily colored by the GC shipped with early versions of Java, which was stop-the-world, inefficient, and associated with random GUI freezes and jerky animations. This initial bad impression continues to persist today esp. among the C/C++ crowd, despite GC technology having made great advances since those early Java days. Aside from skewed impressions, there's still these potential concerns with the GC: (1) Stop-the-world GC pauses (no longer a problem with modern generational collectors, but still applies to D's GC); (2) Non-deterministic destruction of objects (D's dtors are not even guaranteed to run if it's a GC'd object) -- you cannot predict when an object will be collected; (3) GC generally needs more memory than the equivalent manual memory management system. I think you also have to consider that the GC you get with D is not state of the art, and if the opinions expressed on the newsgroup are accurate, it's not likely to get any better. So while you can find examples of high performance applications, AAA games, or whatever that use GC, I doubt any of them would be feasible with Ds GC. And given the design choices D has made as a language, a high performance GC is not really possible. So the GC is actually a poor fit for D as a language. It's like a convertible car with a roof that is only safe up to 50 mph, go over that and its likely to be torn off. So if you want to drive fast you have to put the roof down.
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 10:28:13 UTC, Basile B. wrote: Semi serious answer: In the domain of hoby-ism and small companies programmers that work with statically typed languages all believe that they are super hero in the domain of memory managment. When they see "GC" they think that they are considered as 2nd grade student ^^ It's basically snobbism. I know your response is *tongue in cheek*, but I actually find it easier to use c++11 style memory management across the board than mixing two models. C style memory management, on the other hand, is pretty horrible and you'll end up spend much of your time debugging "unexplainable" crashes. I don't experience that much in C++ when staying within their standard regime. When you want more performance than standard C++ memory management, things can go wrong e.g. manual emplace strategies and forgetting to call destructors etc, but that is the same in D. And frankly, you seldom need that, maybe 2-3 critical places in your program (e.g. graphics/audio).
Re: Why many programmers don't like GC?
On Wednesday, 13 January 2021 at 18:58:56 UTC, Marcone wrote: I've always heard programmers complain about Garbage Collector GC. But I never understood why they complain. What's bad about GC? Semi serious answer: In the domain of hoby-ism and small companies programmers that work with statically typed languages all believe that they are super hero in the domain of memory managment. When they see "GC" they think that they are considered as 2nd grade student ^^ It's basically snobbism.
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 10:05:51 UTC, Guillaume Piolat wrote: On Wednesday, 13 January 2021 at 18:58:56 UTC, Marcone wrote: I've always heard programmers complain about Garbage Collector GC. But I never understood why they complain. What's bad about GC? Languages where the GC usage is unavoidable (Javascript and Java) have created a lot of situations where there is a GC pause in realtime program and the cause is this dynamically allocated memory. So a lot of people make their opinion of GC while using setup where you couldn't really avoid it. Indeed, but I don't think we should underestimate the perceived value of having a minimal runtime. Like, if D had a better GC solution that involved an even heavier runtime, it would still be a big issue for people interested in low level system programming. Transparency is an issue. System level programming means you want to have a clear picture of what is going on in the system at all levels, all the way down to the hardware. If you cannot understand how the runtime works you also cannot fix issues... so a simple runtime is more valuable than a feature rich complex runtime. That is kinda what defines system level programming: you know exactly what every subsystem is doing so that you can anticipate performance/resource issues. And that is the opposite of high level programming where you make no assumptions about the underlying machinery and only care about the abstract descriptions of language semantics.
Re: Why many programmers don't like GC?
On Wednesday, 13 January 2021 at 18:58:56 UTC, Marcone wrote: I've always heard programmers complain about Garbage Collector GC. But I never understood why they complain. What's bad about GC? Languages where the GC usage is unavoidable (Javascript and Java) have created a lot of situations where there is a GC pause in realtime program and the cause is this dynamically allocated memory. So a lot of people make their opinion of GC while using setup where you couldn't really avoid it. For example in Javascript from 10 years ago just using a closure or an array literals could make your web game stutter.
Re: Why many programmers don't like GC?
On Thursday, 14 January 2021 at 00:37:29 UTC, mw wrote: ok, what I really mean is: ... in other "(more popular) languages (than D, and directly supported by the language & std library only)" ... Well, even Python supports both, if you want to, so... I suppose you mean system level programming languages? The reality is that GC for a system level programming language is not popular to begin with. In that domain it is fairly common to not use the standard library and use custom runtimes as we can see for C and C++. Anyway, what makes the D GC weak is exactly that there is not much support for it in the D language or the compilers, only in the runtime and the bare minimum of RTTI. LLVM support more advanced GC features than D provides. So, the D GC doesn't do much more for programmers than Boehm. And Boehm is not popular either... Oilpan, that Chrome uses has more advanced features than the D GC, and does what most system level programmers want: limits it to designated GC types and supports incremental collection. The downside is that each Oilpan GC type also has to specify which pointers to trace, but then again, not being able to do that in D is a disadvantage... For systems programming, I think D would be better off appropriating the approach taken by Oilpan and mix it with reference counting, but make it a language/compiler feature. That is at least a proven approach for one big interactive application. Basically make "D class" objects GC and everything else RC or manual.
Re: Why doesn't this work when the function is a static method?
On Thursday, 14 January 2021 at 05:44:43 UTC, Jack wrote: On Wednesday, 13 January 2021 at 17:21:23 UTC, Paul Backus wrote: Member functions (including static ones) can't be called with UFCS. is this documented somewhere? Is this going to change? It will stay as is. It is somewhat vaguely described in p.7 under UFCS section in functions https://dlang.org/spec/function.html#pseudo-member
