Re: Unittests pass, and then an invalid memory operation happens after?
On Thu, Mar 28, 2024 at 03:56:10AM +, Liam McGillivray via
Digitalmars-d-learn wrote:
[...]
> I may be now starting to see why the use of a garbage collector is
> such a point of contention for D. Not being able to predict how the
> garbage collection process will happen seems like a major problem.
If you want it to be predictable, simply:
import core.memory;
GC.disable();
... // insert your code here
if (timeToCleanup()) {
GC.collect(); // now you know exactly when this happens
}
Of course, you'll have to know exactly how timeToCleanup() should decide
when it's time to collect. Simple possibilities are once every N units
of time, once every N iterations of some main loop, etc.. Or use a
profiler to decide.
> > As mentioned, GCs do not work this way -- you do not need to worry
> > about cascading removal of anything.
>
> Wanting to avoid the GC pauses that I hear about, I was trying to
> optimize object deletion so that the GC doesn't have to look for every
> object individually. It sounds like what I'm hearing is that I should
> just leave everything to the GC. While I can do this without really
> hurting the performance of my program (for now), I don't like this.
The whole point of a GC is that you leave everything up to it to clean
up. If you want to manage your own memory, don't use the GC. D does not
force you to use it; you can import core.stdc.stdlib and use malloc/free
to your heart's content.
> I hope that solving the unpredictable destruction pattern is a
> priority for the developers of the language. This problem in my
> program wouldn't be happening if either *all* of the objects had their
> destructors called or *none* of them did.
Unpredictable order of collection is an inherent property of GCs. It's
not going away. If you don't like it, use malloc/free instead. (Or
write your own memory management scheme.)
> Anyway, I suppose I'll have to experiment with either manually
> destroying every object at the end of every unittest, or just leaving
> more to the GC. Maybe I'll make a separate `die` function for the
> units, if you think it's a good idea.
[...]
I think you're approaching this from a totally wrong angle. (Which I
sympathize with, having come from a C/C++ background myself.) The whole
point of having a GC is that you *don't* worry about when an object is
collected. You just allocate whatever you need, and let the GC worry
about cleaning up after you. The more you let the GC do its job, the
better it will be.
Now of course there are situations where you need deterministic
destruction, such as freeing up system resources as soon as they're no
longer needed (file descriptors, OS shared memory segments allocations,
etc.). For these you would manage the memory manually (e.g. with a
struct that implements reference counting or whatever is appropriate).
As far as performance is concerned, a GC actually has higher throughput
than manually freeing objects, because in a fragmented heap situation,
freeing objects immediately when they go out of use incurs a lot of
random access RAM roundtrip costs, whereas a GC that scans memory for
references can amortize some of this cost to a single period of time.
Now somebody coming from C/C++ would immediately cringe at the thought
that a major GC collection might strike at the least opportune time. For
that, I'd say:
(1) don't fret about it until it actually becomes a problem. I.e., your
program is slow and/or has bad response times, and the profiler is
pointing to GC collections as the cause. Then you optimize appropriately
with the usual practices for GC optimization: preallocate before your
main loop, avoid frequent allocations of small objects (prefer to use
structs rather than classes), reuse previous allocations instead of
allocating new memory when you know that an existing object is no longer
used. In D, you can also selectively allocate certain troublesome
objects with malloc/free instead (mixing both types of allocations is
perfectly fine in D; we are not Java where you're forced to use the GC
no matter what).
(2) Use D's GC control mechanisms to exercise some control over when
collections happen. By default, collections ONLY ever get triggered if
you try to allocate something and the heap has run out of memory. Ergo,
if you don't allocate anything, GC collections are guaranteed not to
happen. Use GC.disable and GC.collect to control when collections
happen. In one of my projects, I got a 40% performance boost by using
GC.disable and using my own schedule of GC.collect, because the profiler
revealed that collections were happening too frequently. The exact
details how what to do will depend on your project, of course, but my
point is, there are plenty of tools at your disposal to exercise some
degree of control.
Or if (1) and (2) are not enough for your particular case, you can
always resort to the nuclear option: slap @nogc on main() and use
Re: Unittests pass, and then an invalid memory operation happens after?
On Wednesday, 27 March 2024 at 22:14:16 UTC, H. S. Teoh wrote:
What's the definition of this.map, this.faction, and
this.currentTile?
As was said, and can be found on the linked repository, they are
references to class objects.
On Thursday, 28 March 2024 at 01:47:27 UTC, Steven Schveighoffer
wrote:
On Wednesday, 27 March 2024 at 21:43:48 UTC, Liam McGillivray
wrote:
`Unit` destructor:
```
~this() {
this.alive = false;
if (this.map !is null) this.map.removeUnit(this);
if (this.faction !is null)
this.faction.removeUnit(this);
if (this.currentTile !is null)
this.currentTile.occupant = null;
}
```
The GC does not guarantee destructor order. So this code is not
valid -- e.g. you can't count on `map` to be a valid object at
this point.
Well, this was originally done so that when I explicitly call
`destroy` on a Unit object (which happens in-game) it will delete
all references to itself.
Do you think it's better for me to move this code to a `die`
function that I can call instead, which will then call for it's
own destruction after running those function calls?
The *only* thing you should be doing in a destructor is freeing
non-GC resources.
I read that the garbage collector *sometimes* but not *always*
calls destructors on deletion, which sounds crazy to me.
The GC is not guaranteed to delete memory or run destructors.
In the current implementation, it will destroy everything at
the end of the program that was allocated using the GC, but the
language does not guarantee this.
What's strange is that even if I explicitly call `destroy` for
the only two objects at the end of the last unittest, it still
happens. I assumed that each unittest was like it's own program,
but I suppose not. It's deleting objects from an earlier
unittest, right?
I may be now starting to see why the use of a garbage collector
is such a point of contention for D. Not being able to predict
how the garbage collection process will happen seems like a major
problem.
As mentioned, GCs do not work this way -- you do not need to
worry about cascading removal of anything.
Wanting to avoid the GC pauses that I hear about, I was trying to
optimize object deletion so that the GC doesn't have to look for
every object individually. It sounds like what I'm hearing is
that I should just leave everything to the GC. While I can do
this without really hurting the performance of my program (for
now), I don't like this.
I hope that solving the unpredictable destruction pattern is a
priority for the developers of the language. This problem in my
program wouldn't be happening if either *all* of the objects had
their destructors called or *none* of them did.
Anyway, I suppose I'll have to experiment with either manually
destroying every object at the end of every unittest, or just
leaving more to the GC. Maybe I'll make a separate `die` function
for the units, if you think it's a good idea.
Also, check your Github notifications. I have something for you.
Re: Two chunks but No allocation
On Wednesday, 27 March 2024 at 20:50:05 UTC, rkompass wrote:
This works:
I decided to give the full code. Maybe then it will be better
understood what I mean. I actually pointed out the indirect
solution above but it's a bit ugly and I'm sure there must be a
better way?
```d
import std.datetime.stopwatch;
import std.stdio, std.algorithm,
std.range;
auto cube(T)(T n)
=> n * n * n;
auto S1(T, R)(R a, T n)
=> n.cube - n; //
auto s1(T)(T first, size_t skip = 3)
=> recurrence!S1(0, first).drop(skip);
enum ITERATIONS = 14_000;
enum BENCHMARKS = 100;
void main()
{
real result;
long total_time = 0;
for(int i = 0; i < BENCHMARKS; i++)
{
auto sw = StopWatch(AutoStart.no);
sw.start();
result = s1(3.0).take(ITERATIONS)
.chunks(2)
.map!(r => 4/r.front)
.array // <- can't it be done without this?
.chunks(2)
.map!"a[0] - a[1]"
.fold!"a + b"(3.0);
sw.stop();
total_time += sw.peek.total!"nsecs";
}
writefln("%f (nsecs)", total_time / BENCHMARKS / 1e9);
writefln("%0.16f (result)", result);```
} /*
0.000456 (nsecs)
3.1415926535890670 (result)
//*/
```
SDB@79
Difference between chunks(stdin, 1) and stdin.rawRead?
Why in the below silly program am I reading both the \r and \n
characters when using rawRead in block a, but when looping by 1
byte chunks in block b only appear to be reading the \n
characters?
I'm on Windows 11 using DMD64 D Compiler v2.107.1 if that
matters, but I'm thinking this maybe has something to do with
stdin in general that I'm not aware of. Any pointers to
understanding what's going on would be appreciated.
import std.stdio;
void main() {
int i;
a: {
i = 0;
writeln("\nin a");
ubyte[1] buffer;
while (true) {
i++;
stdin.rawRead(buffer);
if (buffer[0] == 13) {
write("CR");
} else if (buffer[0] == 10) {
write("LF");
}
if (i > 5) {
goto b;
}
}
}
b: {
writeln("\n\nin b");
i = 0;
foreach (ubyte[] buffer; chunks(stdin, 1)) {
i++;
if (buffer[0] == 13) {
write("cr");
} else if (buffer[0] == 10) {
write("lf");
}
if (i > 5) {
goto a;
}
}
}
}
Output:
in a
CRLF
CRLF
CRLF
in b
lf
lf
lf
lf
lf
lf
in a
Re: Limits of implicit conversion of class arrays
On Monday, 25 March 2024 at 07:16:35 UTC, Per Nordlöw wrote:
On Saturday, 23 March 2024 at 11:04:04 UTC, Dmitry Olshansky
wrote:
The first and second is unsound (infamously allowed in Java).
In the general case, yes. But, do you see any errors with the
code
```d
class Base {}
class Derived : Base {}
@safe pure nothrow unittest {
Base b;
Derived d;
b = d; // pass
Base[] bs;
Derived[] ds;
bs ~= ds; // pass
bs = ds; // fail [1], should pass
bs = cast(Base[])ds; // fail [2], should pass
}
```
Yes, it's unsafe, as you can replace an element of `ds` with
something that has no relation to `Derived`.
Once you cast the slice you can populate it with Derived2
objects that are not Derived, hence breaking type safety of
the ds slice.
Again, in the general case, yes.
So what is different in this code example compared to the
general case? Hint: this has overlaps with a missing compiler
optimization in dmd (and many other statically typed languages)
enabled by a specific kind of data flow analysis. Which one?
If there is a way to end up with a `Derived` reference to point
at something that is not a `Derived` *without a cast* in system
code, or *even with a cast* in safe code, then it is an error. It
doesn't matter if you aren't actually doing it.
If you know you are not making that mistake, change it to system,
and cast to inform the compiler that you "know what you are
doing".
-Steve
Re: Unittests pass, and then an invalid memory operation happens after?
On Wednesday, 27 March 2024 at 21:43:48 UTC, Liam McGillivray
wrote:
In my current [game
project](https://github.com/LiamM32/Open_Emblem), [something
strange](https://github.com/LiamM32/Open_Emblem/issues/20) has
happened as of a recent commit. When running `dub test`, all
the unittests appear to pass, but then after the last unittest
has concluded an "Invalid memory operation" happens. Removing a
few lines replaces this error with a segfault, but either way,
these errors shouldn't be happening. Weirdly, the commit where
these errors first appear did nothing but replace a single
class-member function with a seemingly identical function
through a mixin template.
The errors seem to be related to object deletion. The traceback
output for the first error, and where GDB points to for the
second error, is the destructor for my `Unit` class.
You see, every `Unit` object is associated with a particular
`Map` object and `Faction` object, which it hold references to.
Those objects in turn each have an array of `Unit` objects that
they are associated with. In the `Unit` destructor, I have it
call the `removeUnit` function in both the associated `Map` and
`Faction` objects. The errors are happening in either the
`Unit` destructor itself, or the `removeUnit` function that it
calls. Until the commit that introduced these errors, the
`removeUnit` function was written directly in the `Map` class
in it's module, but this commit replaced it with the mixin
template `UnitArrayManagement`, which `Faction` also uses.
`Unit` destructor:
```
~this() {
this.alive = false;
if (this.map !is null) this.map.removeUnit(this);
if (this.faction !is null)
this.faction.removeUnit(this);
if (this.currentTile !is null)
this.currentTile.occupant = null;
}
```
The GC does not guarantee destructor order. So this code is not
valid -- e.g. you can't count on `map` to be a valid object at
this point.
In my estimation, the code is not correct in principle anyway --
if the `map` has a pointer to the `unit`, then neither will be
collected without both being garbage, and so there is no need to
do these calls (they are all going away presently).
The *only* thing you should be doing in a destructor is freeing
non-GC resources.
I read that the garbage collector *sometimes* but not *always*
calls destructors on deletion, which sounds crazy to me.
The GC is not guaranteed to delete memory or run destructors. In
the current implementation, it will destroy everything at the end
of the program that was allocated using the GC, but the language
does not guarantee this.
The second error, which can be achieved by removing the
instances of `writeln` in `removeUnit` (making it seemingly
identical now to the version of this function previously
defined in the `Map` class) is also strange. It seems to be a
case of the `Unit` object calling a `Map` object that no longer
exists. However, that's also strange, as the `Map` object is
supposed to delete all it's associated units on destruction.
As mentioned, GCs do not work this way -- you do not need to
worry about cascading removal of anything.
You should assume in the destructor that all references in the
type that were pointing at GC blocks are now invalid (i.e.
dangling pointers).
So why are these things even happening *after* the unittests
have been run? What else do I need to know about object
destruction? What may be happening?
The GC is cleaning up all allocated memory, in *no particular
order*.
-Steve
Re: Why is this code slow?
On Wednesday, 27 March 2024 at 08:22:42 UTC, rkompass wrote:
I apologize for digressing a little bit further - just to share
insights to other learners.
Good thing you're digressing; I am 45 years old and I still
cannot say that I am finished as a student! For me this is
version 4 and it looks like we don't need a 3rd variable other
than the function parameter and return value:
```d
auto leibniz_v4(int i) @nogc pure {
double n = 0.5*((i%2) ? -1.0 : 1.0) / (i * 2.0 + 1.0);
while(--i >= 0)
n += ((i%2) ? -1.0 : 1.0) / (i * 2.0 + 1.0);
return n * 4.0;
} /*
3.1415926535892931
3.141592653589 793238462643383279502884197169399375105
3.14159365359077420 (v1)
Avg execution time: 0.33
*/
```
SDB@79
Re: Unittests pass, and then an invalid memory operation happens after?
On Wed, Mar 27, 2024 at 09:43:48PM +, Liam McGillivray via
Digitalmars-d-learn wrote:
[...]
> ```
> ~this() {
> this.alive = false;
> if (this.map !is null) this.map.removeUnit(this);
> if (this.faction !is null) this.faction.removeUnit(this);
> if (this.currentTile !is null) this.currentTile.occupant = null;
> }
> ```
[...]
What's the definition of this.map, this.faction, and this.currentTile?
If any of them are class objects, this would be the cause of your
problem. Basically, when the dtor runs, there is no guarantee that any
referenced classed objects haven't already been collected by the GC. So
if you try to access them, it will crash with an invalid memory access.
In general, it's a bad idea to do anything that relies on the dtor being
run in a particular order, because the GC can collect dead objects in
any order. It's also illegal to perform any GC memory-related
operations inside a dtor (like allocate memory, free memory, etc.)
because the GC is not reentrant.
If you need deterministic clean up of your objects, you should do it
before the last reference to the object is deleted.
T
--
He who does not appreciate the beauty of language is not worthy to bemoan its
flaws.
Unittests pass, and then an invalid memory operation happens after?
In my current [game
project](https://github.com/LiamM32/Open_Emblem), [something
strange](https://github.com/LiamM32/Open_Emblem/issues/20) has
happened as of a recent commit. When running `dub test`, all the
unittests appear to pass, but then after the last unittest has
concluded an "Invalid memory operation" happens. Removing a few
lines replaces this error with a segfault, but either way, these
errors shouldn't be happening. Weirdly, the commit where these
errors first appear did nothing but replace a single class-member
function with a seemingly identical function through a mixin
template.
The errors seem to be related to object deletion. The traceback
output for the first error, and where GDB points to for the
second error, is the destructor for my `Unit` class.
You see, every `Unit` object is associated with a particular
`Map` object and `Faction` object, which it hold references to.
Those objects in turn each have an array of `Unit` objects that
they are associated with. In the `Unit` destructor, I have it
call the `removeUnit` function in both the associated `Map` and
`Faction` objects. The errors are happening in either the `Unit`
destructor itself, or the `removeUnit` function that it calls.
Until the commit that introduced these errors, the `removeUnit`
function was written directly in the `Map` class in it's module,
but this commit replaced it with the mixin template
`UnitArrayManagement`, which `Faction` also uses.
`Unit` destructor:
```
~this() {
this.alive = false;
if (this.map !is null) this.map.removeUnit(this);
if (this.faction !is null) this.faction.removeUnit(this);
if (this.currentTile !is null) this.currentTile.occupant
= null;
}
```
```
template UnitArrayManagement(alias Unit[] unitsArray) {
bool removeUnit(Unit unit) {
import std.algorithm.searching;
writeln("Doing `Map.removeUnit`");
Unit[] shiftedUnits = unitsArray.find(unit);
ushort unitKey = cast(ushort)(unitsArray.length -
shiftedUnits.length);
debug {
writeln("unitsArray: ");
foreach (listedUnit; unitsArray)
writeln(listedUnit.name~", ");
writeln("shiftedUnits: ");
foreach (listedUnit; shiftedUnits)
writeln(listedUnit.name~", ");
}
if (shiftedUnits.length > 0) {
debug writeln("shiftedUnits.length > 0");
unitsArray[$-shiftedUnits.length] = null;
for (ushort i=0; idebug writeln("In loop. unitKey = ", unitKey, ",
i = ", i);
unitsArray[unitKey+i] = unitsArray[unitKey+i+1];
}
unitsArray.length--;
return true;
} else return false;
}
}
```
The first error happens because I inserted some uses of `writeln`
for debugging purposes in the new version of `removeUnit`
(because I haven't figured out how to do the same thing with
GDB), in which I try to get it to print the names of all the
units in the array before deleting any of them. I suppose that it
might get a `Invalid memory operation` when trying to access a
member of a `Unit` object that no longer exists, but this
shouldn't be happening. When that other `Unit` object got
destroyed, the destructor should have called this same
`removeUnit` function to remove it's reference from the array.
I read that the garbage collector *sometimes* but not *always*
calls destructors on deletion, which sounds crazy to me. Is this
a case of one unit being deleted without the destructor and then
the next unit (of the same `Map` object) having the destructor
called?
Are destructors normally called after a program is concluded?
The second error, which can be achieved by removing the instances
of `writeln` in `removeUnit` (making it seemingly identical now
to the version of this function previously defined in the `Map`
class) is also strange. It seems to be a case of the `Unit`
object calling a `Map` object that no longer exists. However,
that's also strange, as the `Map` object is supposed to delete
all it's associated units on destruction.
I wrote these destructors so that objects wouldn't have
references floating around on their deletion, yet now I'm getting
errors from the destructors.
So why are these things even happening *after* the unittests have
been run? What else do I need to know about object destruction?
What may be happening?
Re: Two chunks but No allocation
On Wednesday, 27 March 2024 at 13:38:29 UTC, Salih Dincer wrote:
So, not works this:
```d
fib(1, 1).take(48)
//.array
.chunks(2)
.map!"a[1] / a[0]"
.back
.writeln; // 1.61803
```
Thanks...
SDB@79
This works:
```d
import std.stdio;
import std.range;
import std.algorithm;
void main() {
auto fib = (real a, real b) => recurrence!"a[n-1] + a[n-2]"(a,
b);
auto golden = fib(1, 1).drop(46).take(2).fold!((a, e) => a/e);
writefln("%.20f", golden);
writeln("0.61803398874989484820");
}
```
Two chunks but No allocation
Is it possible to process both chunks without requiring memory
allocation (not using an array)?
For example:
```d
import std;
void main()
{
auto fib = (real a, real b)
=> recurrence!"a[n-1] + a[n-2]"(a, b);
auto myFib = fib(1, 1).take(48).array;
auto goldenRadio = myFib.chunks(2).map!
//"a[1] / a[0]";/* line #9
((a) {
const m = a.front;
a.popFront();
const n = a.front;
return m/n; });//*/
goldenRadio.back.writefln!"%.21f";
writeln("0.61803398874989484820");
}
```
Of course, it also works if you remove the comment marks on line
#9 but I'm still using .array because the .chunks error tells me
that.
So, not works this:
```d
fib(1, 1).take(48)
//.array
.chunks(2)
.map!"a[1] / a[0]"
.back
.writeln; // 1.61803
```
Thanks...
SDB@79
Re: Why is this code slow?
I apologize for digressing a little bit further - just to share
insights to other learners.
I had the question, why my binary was so big (> 4M), discovered
the
`gdc -Wall -O2 -frelease -shared-libphobos` options (now >200K).
Then I tried to avoid GC, just learnt about this: The GC in the
Leibnitz code is there only for the writeln. With a change to
(again standard C) printf the
`@nogc` modifier can be applied, the binary then gets down to
~17K, a comparable size of the C counterpart.
Another observation regarding precision:
The iteration proceeds in the wrong order. Adding small
contributions first and bigger last leads to less loss when
summing up the small parts below the final real/double LSB limit.
So I'm now at this code (abolishing the avarage of 20 interations
as unnesseary)
```d
// import std.stdio; // writeln will lead to the garbage
collector to be included
import core.stdc.stdio: printf;
import std.datetime.stopwatch;
const int ITERATIONS = 1_000_000_000;
@nogc pure double leibniz(int it) { // sum up the small values
first
double n = 0.5*((it%2) ? -1.0 : 1.0) / (it * 2.0 + 1.0);
for (int i = it-1; i >= 0; i--)
n += ((i%2) ? -1.0 : 1.0) / (i * 2.0 + 1.0);
return n * 4.0;
}
@nogc void main() {
double result;
double total_time = 0;
auto sw = StopWatch(AutoStart.yes);
result = leibniz(ITERATIONS);
sw.stop();
total_time = sw.peek.total!"nsecs";
printf("%.16f\n", result);
printf("Execution time: %f\n", total_time / 1e9);
}
```
result:
```
3.1415926535897931
Execution time: 1.068111
```
Re: request assistance resolving curl error
On 3/26/24 8:44 PM, Andrea Fontana wrote: On Tuesday, 26 March 2024 at 07:13:24 UTC, confuzzled wrote: I think you should use the HTTP interface, did you check this docs? https://dlang.org/phobos/std_net_curl.html#.HTTP https://dlang.org/phobos/std_net_curl.html#.HTTP.addRequestHeader Andrea Thanks Andrea. I noticed it and even tried HTTP before but could not find a means for setting CurlOptions. As a result, I chose to stick with Curl. Following your comment, however, I went back and combed the source and found it buried in HTTP.handle. Thanks again. --confuzzled
