Re: Populating nested AAs; "This wouldn't be so verbose in Ada 2020"
On Sunday, 8 December 2019 at 06:42:22 UTC, Paul Backus wrote:
You can use the `require` function [1] for this:
with (userHistory.require(host).require(user)) {
if (isOk) ++ok; // renamed to avoid shadowing
else ++evil;
}
Many "methods" for built-in AAs are located in the `object`
module. It makes them a bit tricky to find in the
documentation, if you don't already know they're there.
https://dlang.org/phobos/object.html#.require
Thanks! I wasn't aware of the 'object' doc. Looks like the
language documentation does mention require() in "Inserting if
not present", but combining it with 'with' is a neat trick.
Re: Populating nested AAs; "This wouldn't be so verbose in Ada 2020"
On Sunday, 8 December 2019 at 04:17:45 UTC, mipri wrote:
Hello,
I've got this code:
struct UserStats {
int ok, evil;
}
// module-level variable
UserStats[string][string] userHistory;
and this code that populates it:
// loop over DB query
if (host !in userHistory)
userHistory[host] = typeof(userHistory[host]).init;
if (user !in userHistory[host])
userHistory[host][user] =
typeof(userHistory[host][user]).init;
if (ok) ++userHistory[host][user].ok;
else ++userHistory[host][user].evil;
You can use the `require` function [1] for this:
with (userHistory.require(host).require(user)) {
if (isOk) ++ok; // renamed to avoid shadowing
else ++evil;
}
Many "methods" for built-in AAs are located in the `object`
module. It makes them a bit tricky to find in the documentation,
if you don't already know they're there.
https://dlang.org/phobos/object.html#.require
Populating nested AAs; "This wouldn't be so verbose in Ada 2020"
Hello,
I've got this code:
struct UserStats {
int ok, evil;
}
// module-level variable
UserStats[string][string] userHistory;
and this code that populates it:
// loop over DB query
if (host !in userHistory)
userHistory[host] = typeof(userHistory[host]).init;
if (user !in userHistory[host])
userHistory[host][user] =
typeof(userHistory[host][user]).init;
if (ok) ++userHistory[host][user].ok;
else ++userHistory[host][user].evil;
The joke of the thread title is that in pseudo-Ada2020, those
assignments would look like
User_History (host) = typeof (@).init;
User_History (host) (user) = typeof (@).init;
as @ is a shortcut for the left hand side of an assignment.
It's what Ada has instead of += and friends.
Of course, I don't propose that @ be added to D; I just want to
know if there's a good idiom for new assignments to nested AAs
like this. A nice alternative for my specific use is
struct UserStats {
static int[Tuple!(string, string)] ok, evil;
}
and
// loop over DB query
if (ok) ++UserStats.ok[tuple(host, user)];
else ++UserStats.evil[tuple(host, user)];
But of course this would be less convenient if f.e. I wanted to
pattern my data structures after a JSON production result.
Re: Using map result type
On Sunday, 8 December 2019 at 01:10:21 UTC, AA wrote:
I'd like to accept the return type of map. From some previous
questions that I should accept a template?
In general this is what you want to do with any kind of range
code, because you're not working with definite types, but with
types that have certain properties. And an advantage of this is
that you can decide to do more or less efficient things at
compile time based on additional properties of the types you're
working with. Just open up phobos and look for 'static if':
https://github.com/dlang/phobos/blob/master/std/algorithm/searching.d#L3858
So if you you give minPos an array, it'll just foreach() over
the array and return a slice. In which case minPos is exactly
as efficient as the code you would've written by hand instead
of using minPos, because it expands to that same code.
And if you give minPos something else, it'll still work.
So for something like:
```
void mapAccepter(Range)(Range r)
{
import std.array : array;
import std.stdio : writeln;
auto collected = r.array;
writeln(collected);
}
void main()
{
import std.algorithm.iteration : map;
int[] nums = [1, 2, 3];
auto evenness = map!(n => n % 2 == 0)(nums);
mapAccepter(evenness);
}
```
1) Is there any way I can make `mapAccepter` not a templated
function?
Not really. Even if you try to use typeof() to get the
(internal) type that std.alogorithm.map returns, you'll run
into errors like
Error: function x253.mapAccepter(MapResult!(__lambda5, int[]) r)
is not callable using argument types (MapResult!(__lambda1,
int[]))
when you try to use it.
2) Is there any way if I need to make `mapAccepter` templated
to constrain Range to be a range of booleans.
Certainly. Static reflection's what this kind of generic code
is all about. See all the constraints in the std.algorithm pages
in the library documentation? Just look at this one, randomly
picked:
https://dlang.org/phobos/std_algorithm_searching.html#.endsWith
if (isBidirectionalRange!Range
&& (Needles.length > 1)
&& is(typeof(.endsWith!pred(doesThisEnd,
withOneOfThese[0])) : bool)
&& is(typeof(.endsWith!pred(doesThisEnd,
withOneOfThese[1..$])) : uint));
So for your code:
void mapAccepter(Range)(Range r)
if (is(ElementType!Range == bool))
{
import std.array : array;
import std.stdio : writeln;
auto collected = r.array;
writeln(collected);
}
Or if you do this often, maybe something like this would do:
enum RangeOfBools(T) = is(ElementType!T == bool) &&
isInputRange!T;
void mapAccepter(Range)(Range r) if (RangeOfBools!Range)
{
import std.array : array;
import std.stdio : writeln;
auto collected = r.array;
writeln(collected);
}
Using map result type
I'd like to accept the return type of map. From some previous
questions that I should accept a template?
So for something like:
```
void mapAccepter(Range)(Range r)
{
import std.array : array;
import std.stdio : writeln;
auto collected = r.array;
writeln(collected);
}
void main()
{
import std.algorithm.iteration : map;
int[] nums = [1, 2, 3];
auto evenness = map!(n => n % 2 == 0)(nums);
mapAccepter(evenness);
}
```
1) Is there any way I can make `mapAccepter` not a templated
function?
2) Is there any way if I need to make `mapAccepter` templated to
constrain Range to be a range of booleans.
Re: GC.collect inflating memory usage?
On 07/12/2019 12:20, cc wrote: > Given the following program: [...] > But when both FREE and COLLECT are enabled, things seem to spiral out of > control: > // FREE, COLLECT > Stats(16, 1048560, 16) > 848 4096 > 40960832 40964096 > 81920832 81924096 > 122880832 122884096 > 163840832 163844096 > 204800832 204804096 > 245760832 245764096 > 286720832 286724096 > 327680832 327684096 > 368640832 368644096 > Elapsed: 29143 ms > > I wouldn't normally call GC.collect on every frame in my application, > but I'm curious why this is happening and if there is unnecessary bloat > being added somehow when I do choose to call GC.free manually and > garbage collection later occurs in a long-running program. Ideally I'd > like to free up as many objects and arrays as soon as they become unused > to avoid lengthy collections reducing performance. I know that > std.container.array is one alternative to using D's GC-managed dynamic > arrays, but could I run into the same issue when trying to manually > deallocate class objects as well? > > Using DMD32 D Compiler v2.089.0-dirty > Seems like a bug introduced in dmd 2.086, I've created a bugzilla issue: https://issues.dlang.org/show_bug.cgi?id=20438 I suspect there is something broken with respect to the free-lists inside the GC when manually freeing memory :-/
Re: Profiling the memory in D
On Wednesday, 4 December 2019 at 22:51:45 UTC, Steven
Schveighoffer wrote:
I localized that the leak was actually being caused by
websockets. I want to write down my experience because I did some
weird stuff which seems to be working but I want to learn how it
actually make sense and works.
I had a fancy work flow which caused by that bug
https://github.com/vibe-d/vibe.d/issues/2169 which disallowed to
open multiple sockets from my main process.
I solved that by lunching multiple process and using one web
socket by one process. And communicated this process and my main
process via zmqd.
My suggestion is; don't do that. Don't be super creative with
current Vibe websockets. I had this unidentifable leak which took
to much time to localize.
The bug I created around one year ago is solved now so I left
creating processes approach and put web sockets in to a list.
Next problem I had while listening 300 websocket was I got some
crushes within
webSocket.dataAvailableForRead() function, I am not sure if it is
a bug or my misusage some how so I haven't created a bug yet. But
my question to vibe forum can be seen
https://forum.rejectedsoftware.com/groups/rejectedsoftware.vibed/thread/112309/
I solved that ( I hope ) by using something like:
if ( webSocket.waitForData(0.msecs) &&
webSocket.dataAvailableForRead() )
I know it looks so wierd to wait for 0.msecs but things seems to
be working :/ .
The last problem I had was with closing the socket because this
sockets are getting disconnected time to time. I need to close
and reopen. When I call
webSocket.close() directly, after my program runs about 1 day it
was freezing in a function called epoll while calling
webSocket.close().
I also find another weird solution to that problem :
while ( true )
{
auto future = vibe.core.concurrency.async( {
socket.socket.close(); return true;} );
vibe.core.core.sleep(100.msecs);
if ( future.ready() )
break;
writeln( " Couldn't close the socket retring ");
}
sockets.remove(uniqStreamName);// BTW order of this removal
matters if you remove
//from your list before closing the ticket you are screwed.
Which seems to be working with 300 websockets around 2 days
without any leak nor crush neither freeze.
As I pointed in the beginning I don't have any question or
problems now but I am very open to feedback if you guys have any.
Erdemdem
Re: Thin UTF8 string wrapper
On Saturday, 7 December 2019 at 15:57:14 UTC, Jonathan M Davis wrote: There may have been some tweaks to std.encoding here and there, but for the most part, it's pretty ancient. Looking at the history, it's Seb who marked some if it as being a replacement for std.utf, which is just plain wrong. Ouch! I must say it was a surprise to read, precisely because std.encoding seemed weird and clunky. Good to know that it's misleading. Unfortunately that adds to the list I have of weirdly misleading docs that seem to have crept in over the last months/years :-( std.utf.validate does need a replacement, but doing so gets pretty complicated. And looking at std.encoding.isValid, I'm not sure that what it does is any better from simply wrapping std.utf.validate and returning a bool based on whether an exception was thrown. Unfortunately I'm dealing with a use case where exception throwing (and indeed, anything that generates garbage) is preferred to be avoided. That's why I was looking for a function that returned a bool ;-) Depending on the string, it would actually be faster to use validate, because std.encoding.isValid iterates through the entire string regardless. The way it checks validity is also completely different from what std.utf does. Either way, some of the std.encoding internals do seem to be an alternate implementation of what std.utf has, but outside of std.encoding itself, std.utf is what Phobos uses for UTF-8, UTF-16, and UTF-32, not std.encoding. Thanks -- good to know. I did do a PR at one point to add isValidUTF to std.utf so that we could replace std.utf.validate, but Andrei didn't like the implementation, so it didn't get merged, and I haven't gotten around to figuring out how to implement it more cleanly. Thanks for the attempt, at least! While I get the reasons it was rejected, it feels a bit of a shame -- surely it's easier to do a more major under-the-hood rewrite with the public API (and tests) already in place ... :-\
Re: Thin UTF8 string wrapper
On Saturday, December 7, 2019 5:23:30 AM MST Joseph Rushton Wakeling via Digitalmars-d-learn wrote: > On Saturday, 7 December 2019 at 03:23:00 UTC, Jonathan M Davis > > wrote: > > The module to look at here is std.utf, not std.encoding. > > Hmmm, docs may need updating then -- several functions in > `std.encoding` explicitly state they are replacements for > `std.utf`. Did you mean `std.uni`? > It is honestly a bit confusing which of these 3 modules to use, > especially as they each offer different (and useful) tools. For > example, `std.utf.validate` is less useful than > `std.encoding.isValid`, because it throws rather than returning a > bool and giving the user the choice of behaviour. `std.uni` > doesn't seem to have any equivalent for either. > > Thanks in any case for the as-ever characteristically detailed > and useful advice :-) There may have been some tweaks to std.encoding here and there, but for the most part, it's pretty ancient. Looking at the history, it's Seb who marked some if it as being a replacement for std.utf, which is just plain wrong. Phobos in general uses std.utf for dealing with UTF-8, UTF-16, and UTF-32, not std.encoding. std.encoding is an old module that's had some tweaks done to it but which probably needs a pretty serious overhaul. The only thing that I've ever use it for is BOM stuff. std.utf.validate does need a replacement, but doing so gets pretty complicated. And looking at std.encoding.isValid, I'm not sure that what it does is any better from simply wrapping std.utf.validate and returning a bool based on whether an exception was thrown. Depending on the string, it would actually be faster to use validate, because std.encoding.isValid iterates through the entire string regardless. The way it checks validity is also completely different from what std.utf does. Either way, some of the std.encoding internals do seem to be an alternate implementation of what std.utf has, but outside of std.encoding itself, std.utf is what Phobos uses for UTF-8, UTF-16, and UTF-32, not std.encoding. I did do a PR at one point to add isValidUTF to std.utf so that we could replace std.utf.validate, but Andrei didn't like the implementation, so it didn't get merged, and I haven't gotten around to figuring out how to implement it more cleanly. - Jonathan M Davis
Re: Thin UTF8 string wrapper
On Saturday, 7 December 2019 at 03:23:00 UTC, Jonathan M Davis wrote: The module to look at here is std.utf, not std.encoding. Hmmm, docs may need updating then -- several functions in `std.encoding` explicitly state they are replacements for `std.utf`. Did you mean `std.uni`? It is honestly a bit confusing which of these 3 modules to use, especially as they each offer different (and useful) tools. For example, `std.utf.validate` is less useful than `std.encoding.isValid`, because it throws rather than returning a bool and giving the user the choice of behaviour. `std.uni` doesn't seem to have any equivalent for either. Thanks in any case for the as-ever characteristically detailed and useful advice :-)
GC.collect inflating memory usage?
Given the following program:
//version=FREE;
//version=COLLECT;
import std.stdio;
import std.datetime.stopwatch;
import core.memory;
immutable int[] intZ =
[1,2,3,4,4,6,6,8,8,65,8,23,76,2,57,264,23,4,4,6,6,8,8,65,8,23,76,2,57,264,23,4,4,6,6,8,8,65,8,23,76,2,57,264,23,4,4,6,6,8,8,65,8,23,76,2,57,264,23,4,4,6,6,8,8,65,8,23,76,2,57,264,23,4,4,6,6,8,8,65,8,23,76,2,57,264,23,4,4,6,6,8,8,65,8,23,76,2,57,264,23,4,4,6,6,8,8,65,8,23,76,2,57,264,23,4,4,6,6,8,8,65,8,23,76,2,57,264,23];
void main() {
writeln(GC.stats);
enum max = 10;
StopWatch sw;
sw.start();
foreach (i; 0 .. max) {
bool doprint = !(i % (max/10));
int[] z = intZ.dup;
if (doprint) writef("%7d ", GC.stats.usedSize);
version(FREE) GC.free(cast(void*) z.ptr);
version(COLLECT) GC.collect();
if (doprint) writefln("%7d", GC.stats.usedSize);
}
sw.stop();
writefln("Elapsed: %d ms", sw.peek.total!"msecs");
}
When compiled with neither the FREE or COLLECT versions, I get
results like this typically:
Stats(16, 1048560, 16)
848 848
883104 883104
711072 711072
539040 539040
367008 367008
191696 191696
1966419664
887200 887200
715168 715168
540672 540672
Elapsed: 11 ms
When only the FREE line is enabled, I see results like this:
// FREE
Stats(16, 1048560, 16)
848 32
848 32
848 32
848 32
848 32
848 32
848 32
848 32
848 32
848 32
Elapsed: 12 ms
When only the COLLECT line is enabled, I see results like this:
// COLLECT
Stats(16, 1048560, 16)
848 4096
4928 4096
4928 4096
4928 4096
9024 8192
4928 4096
4928 4096
4928 4096
4928 4096
4928 4096
Elapsed: 1130 ms
But when both FREE and COLLECT are enabled, things seem to spiral
out of control:
// FREE, COLLECT
Stats(16, 1048560, 16)
848 4096
40960832 40964096
81920832 81924096
122880832 122884096
163840832 163844096
204800832 204804096
245760832 245764096
286720832 286724096
327680832 327684096
368640832 368644096
Elapsed: 29143 ms
I wouldn't normally call GC.collect on every frame in my
application, but I'm curious why this is happening and if there
is unnecessary bloat being added somehow when I do choose to call
GC.free manually and garbage collection later occurs in a
long-running program. Ideally I'd like to free up as many
objects and arrays as soon as they become unused to avoid lengthy
collections reducing performance. I know that
std.container.array is one alternative to using D's GC-managed
dynamic arrays, but could I run into the same issue when trying
to manually deallocate class objects as well?
Using DMD32 D Compiler v2.089.0-dirty
