Re: Tips on TCP socket to postgresql middleware
On Monday, 21 February 2022 at 04:46:53 UTC, Chris Piker wrote: On Sunday, 20 February 2022 at 18:00:26 UTC, eugene wrote: Yes, here is my engine with example (echo client/server pair): - [In D (for Linux & FreeBSD)] The code is terse and clean, thanks for sharing :) Nice to hear, thnx! Actually I wrote D variant just to demonstrate the idea to my collegue who is OOP/py guy and for whom it was hard to understand C code. I'm adverse to reading it closely since there was no license file and don't want to accidentally violate copyright. :) I think WTFPL will do :) I noticed there were no dub files in the package. Not surprised. Dub is such a restrictive tool compared to say, setup.py/.cfg in python. I have very little experience in D and did not think about dub at all.
Re: Tips on TCP socket to postgresql middleware
On Monday, 21 February 2022 at 04:48:56 UTC, Chris Piker wrote: On Sunday, 20 February 2022 at 18:36:21 UTC, eugene wrote: I often use two connections, one for perform main task (upload some data and alike) and the second for getting notifications from PG, 'cause it very incovinient to do both in a single connection. Ah, a very handy tip. It would be convoluted to multiplex notifications on the data connection. I am remembering psql client behavior - it sees notifications only after some request. It is really inconvinient to perform regular tasks and be ready to peek up notifications at any moment in one connection.
Re: Tips on TCP socket to postgresql middleware
On Sunday, 20 February 2022 at 18:36:21 UTC, eugene wrote: I often use two connections, one for perform main task (upload some data and alike) and the second for getting notifications from PG, 'cause it very incovinient to do both in a single connection. Ah, a very handy tip. It would be convoluted to multiplex notifications on the data connection.
Re: Tips on TCP socket to postgresql middleware
On Sunday, 20 February 2022 at 18:00:26 UTC, eugene wrote: Yes, here is my engine with example (echo client/server pair): - [In D (for Linux & FreeBSD)](http://zed.karelia.ru/0/e/edsm-2022-02-20.tar.gz) The code is terse and clean, thanks for sharing :) I'm adverse to reading it closely since there was no license file and don't want to accidentally violate copyright. I noticed there were no dub files in the package. Not surprised. Dub is such a restrictive tool compared to say, setup.py/.cfg in python.
Re: Tips on TCP socket to postgresql middleware
On Sunday, 20 February 2022 at 17:58:41 UTC, Ali Çehreli wrote: Another one is to set the message box sizes to throttle. Message sizes and rates are relatively well know so it will be easy to pick a throttle point that's unlikely to backup the source yet provide for some quick DB maintenance in the middle of a testing session. In case you haven't seen yet, the recipe for std.concurrency that works for me is summarized here: https://www.youtube.com/watch?v=dRORNQIB2wA=1735s Thanks! I like your simple exception wrapping pattern, will use that.
Re: Is there a way to not escape slashes when parsing JSON?
On Monday, 21 February 2022 at 03:42:55 UTC, bachmeier wrote:
I tried this
```d
import std.json, std.stdio;
void main() {
writeln(parseJSON(`{"a": "path/file"}`,
JSONOptions.doNotEscapeSlashes));
}
```
but the output is
```
{"a":"path\/file"}
```
Is there a way to avoid the escaping of the forward slash? Is
there some reason I should want to escape the forward slash?
The options are applied on parsing or output but do not stay with
the item! So just because you parsed without allowing escapes on
slashes doesn't mean the output will use that option.
2 ways I found:
```d
// 1. allocate a string to display
writeln(parseJson(...).toString(JSONOptions.doNotEscapeSlashes));
// 2. wrap so you can hook the output range version of toString
struct NoEscapeJson
{
JSONValue json;
void toString(Out)(Out outputrange) const
{
json.toString(outputrange,
JSONOptions.doNotEscapeSlashes);
}
}
writeln(NoEscapeJson(parseJson(...)));
```
-Steve
Is there a way to not escape slashes when parsing JSON?
I tried this
```
import std.json, std.stdio;
void main() {
writeln(parseJSON(`{"a": "path/file"}`,
JSONOptions.doNotEscapeSlashes));
}
```
but the output is
```
{"a":"path\/file"}
```
Is there a way to avoid the escaping of the forward slash? Is
there some reason I should want to escape the forward slash?
Crosscompiling LDC's druntime for Android on Windows
I'm trying to cross compile LDC's druntime to enable more debug print statements but I'm having trouble getting it built correctly for aarch64. I'm following the azure pipeline's commands for how it's being built here: https://github.com/ldc-developers/ldc/blob/master/.azure-pipelines/2-posix-build_cross_android.yml#L64 It includes threadasm.S that does have an aarch64 version, it just isn't being selected for some reason. I get the following errors when trying to link: ``` ld: error: C:/Program Files/LDC 1.28/bin/../lib-android_aarch64\libdruntime-ldc-debug.a(threadasm.S.o) is incompatible with aarch64linux ld: error: C:/Program Files/LDC 1.28/bin/../lib-android_aarch64\libphobos2-ldc-debug.a(adler32.c.o) is incompatible with aarch64linux ld: error: C:/Program Files/LDC 1.28/bin/../lib-android_aarch64\libphobos2-ldc-debug.a(compress.c.o) is incompatible with aarch64linux ld: error: C:/Program Files/LDC 1.28/bin/../lib-android_aarch64\libphobos2-ldc-debug.a(crc32.c.o) is incompatible with aarch64linux ld: error: C:/Program Files/LDC 1.28/bin/../lib-android_aarch64\libphobos2-ldc-debug.a(deflate.c.o) is incompatible with aarch64linux ld: error: C:/Program Files/LDC 1.28/bin/../lib-android_aarch64\libphobos2-ldc-debug.a(infback.c.o) is incompatible with aarch64linux ld: error: C:/Program Files/LDC 1.28/bin/../lib-android_aarch64\libphobos2-ldc-debug.a(inflate.c.o) is incompatible with aarch64linux ``` I manually set the cflags to include -march=armv8-a instead of v7 which it was by default I guess from the android cmake toolchain file. I don't see what else needs to be set in order to change what binary is being built. For d code, I just set the mtriple but it is using the ndk's clang for the c files.
Re: Tips on TCP socket to postgresql middleware
On Saturday, 19 February 2022 at 20:13:01 UTC, Chris Piker wrote: In general it should buffer data in RAM to avoid exerting back pressure on the input socket and to allow for dropped connections to the PG database. If I get it right you want to restore connection if it was closed by server for some reason. I use special SM for that purpose, see [this picture](http://zed.karelia.ru/0/e/db-link.jpg) In each state where this SM has to send/recv data, it takes sending/receiving SM from a pool and commands them to perform the task. Upon reaching IDLE state this machine send some messsge to the user (another SM) of the connection and seats in this state until the user detects connection lost (in which case it sends M2 to DB-LINK SM). Then DB-LINK goes to WAIT state, where it starts a timer and when it expires, it goes to CONN state, where it tries to reconnect (using sending SM - when connection is ready we get POLLOUT on socket). You can have as many such connectors as you want, so you have multiple connections within single thread. I often use two connections, one for perform main task (upload some data and alike) and the second for getting notifications from PG, 'cause it very incovinient to do both in a single connection.
Re: Tips on TCP socket to postgresql middleware
On Sunday, 20 February 2022 at 16:55:44 UTC, Chris Piker wrote: But I would like to return to your idea in a couple months so that I can try a fiber based implementation instead. I thougt about implementing my engine using fibers but... it seemed to me they are not very convinient because coroutines yield returns to the caller, but I want to return to a single event loop (after processing an event). Usually I design more or less complex (network) programs using event-driven paradigm (reactor pattern) plus state machines. In other words programs designed this way are, so to say, hierarchical team of state machines, interacting with each other as well as with outer world (signals, timers, events from sockets etc) It sounds like you might have a rigorous way of defining and keeping track of your state machines. I could probably learn quite a bit from reading your source code, or the source for similarly implemented programs. Are there examples you would recommend? Yes, here is my engine with example (echo client/server pair): - [In C (for Linux)](http://zed.karelia.ru/mmedia/bin/edsm-g2-rev-h.tar.gz) - [In D (for Linux & FreeBSD)](http://zed.karelia.ru/0/e/edsm-2022-02-20.tar.gz) edsm = 'event driven state machines' As to the program you are writing - I wrote a couple of dozens of programs more or less similar to what you are going to do (data acqusition) using the engine above (C) for production systems and they all serve very well.
Re: Tips on TCP socket to postgresql middleware
On 2/19/22 12:13, Chris Piker wrote: >* gotchas you've run into in your multi-threaded (or just concurrent) > programs, I use the exact scenario that you describe: Multiple threads process data and pass the results to a "writer" thread that persist it in a file. The main gotcha is your thread disappearing without a trace. The most common reason is it throws an exception and dies. Another one is to set the message box sizes to throttle. Otherwise, producers could produce more than the available memory before the consumer could consume it. Unlike the main thread, there is nobody to catch an report this "uncaught" exception. https://dlang.org/phobos/std_concurrency.html#.setMaxMailboxSize You need to experiment with different number of threads, the buffer size that you mention, different lengths of message boxes, etc. For example, I could not gain more benefit in my program beyond 3 threads (but still set the number to 4 :p Humans are crazy.). In case you haven't seen yet, the recipe for std.concurrency that works for me is summarized here: https://www.youtube.com/watch?v=dRORNQIB2wA=1735s Ali
Re: keyword as struct field
On Sunday, 20 February 2022 at 17:02:21 UTC, H. S. Teoh wrote:
The D convention is to append a `_` to the identifier. From
https://dlang.org/dstyle.html:
But this doesn't fix the issue, because mir-ion will throw an
exception that there is no version_ field that required and but
json string does have "version".
I already fixed the issue with mir-ion @serdeKeys("version")
annotation to struct field. So i'm good. But in Nim backticks are
a _little_ bit other thing then just simple convention. Not sure
about is's implementation, but at least for situation like that
it would work. And it's operator overloading feature uses them
too.
Re: keyword as struct field
On Sun, Feb 20, 2022 at 04:52:30PM +, partypooper via Digitalmars-d-learn wrote: > On Sunday, 20 February 2022 at 15:33:17 UTC, Andrey Zherikov wrote: > > On Sunday, 20 February 2022 at 11:08:55 UTC, partypooper wrote: > > > keyword as struct field > > > > I believe this is the case for the most languages - keyword is not > > allowed as a variable name. > > Yes, but as example in Nim you can surround identifier with backticks > to omit this issue. The D convention is to append a `_` to the identifier. From https://dlang.org/dstyle.html: If a name would conflict with a keyword, and it is desirable to use the keyword rather than pick a different name, a single underscore ‘_’ should be appended to it. Names should not be capitalized differently in order to avoid conflicting with keywords. T -- Questions are the beginning of intelligence, but the fear of God is the beginning of wisdom.
Re: Tips on TCP socket to postgresql middleware
On Sunday, 20 February 2022 at 15:20:17 UTC, eugene wrote: Most people will probably say this is crazy, but as to PG, one can do without libraries. I am doing so during years (in C, not D) and did not expierienced extremely complex troubles. I mean I do not use libpq - instead I implement some subset of the protocol, which is needed for particular program. Very interesting. I need to stand-up this program and two others in one week, so it looks like dpq2 and message passing is the good short term solution to reduce implementation effort. But I would like to return to your idea in a couple months so that I can try a fiber based implementation instead. Usually I design more or less complex (network) programs using event-driven paradigm (reactor pattern) plus state machines. In other words programs designed this way are, so to say, hierarchical team of state machines, interacting with each other as well as with outer world (signals, timers, events from sockets etc) It sounds like you might have a rigorous way of defining and keeping track of your state machines. I could probably learn quite a bit from reading your source code, or the source for similarly implemented programs. Are there examples you would recommend?
Re: keyword as struct field
On Sunday, 20 February 2022 at 15:33:17 UTC, Andrey Zherikov wrote: On Sunday, 20 February 2022 at 11:08:55 UTC, partypooper wrote: keyword as struct field I believe this is the case for the most languages - keyword is not allowed as a variable name. Yes, but as example in Nim you can surround identifier with backticks to omit this issue.
Re: Function Parameters without Names?
On 2/19/22 22:09, Salih Dincer wrote:
> The following doesn't work in a function outside the class:
> ```d
> TYPE foo(TYPE) { return 42; }
> ```
> The compiler gives the following error:
> ```undefined identifier TYPE```
I am not sure whether you are asking a question so apologies if I am
stating the obvious. For (TYPE) to be a template parameter list, there
must be the function parameter list as well:
TYPE foo(TYPE)() { return 42; }
By the way, the fact that we can return the "int" 42 even when TYPE is
e.g. 'short' is thanks to "value range propagation". The compilr sees
that the manifest constant 42 fits in a short and compiles the code
without complaint.
On the other hand, the following modification does not compile:
TYPE foo(TYPE)() {
int r = 42;
return r; // <-- Compilation ERROR
// Error: cannot implicitly convert expression `r` of type `int` to `short`
}
void main() {
foo!short();
}
Getting back to the main topic, you may or may not need the function
parameter list for a member function:
struct S(TYPE) {
// This is not a template but a function
// with the parameter name not omitted.
// (TYPE is already known at this point.)
TYPE foo(TYPE) {
return 42;
}
// This is a member function template
TYPE bar(TYPE2)() {
// ...
}
}
Ali
Re: keyword as struct field
On Sunday, 20 February 2022 at 11:08:55 UTC, partypooper wrote: Hello, I'm new to D. Title is self described, is it possible to use keyword as struct field? Maybe it is XYproblem, so here I will describe a little what I need. I'm parsing some json currently with [mir-ion](https://code.dlang.org/packages/mir-ion) (better/simpler suggestions for only json de/serialization?), that has "version" as one of it's keys, so I can't use that as field in my struct. Maybe some annotation in mir library to parse "version" into other struct field name, can't find though (its docs pretty concise). Unfortunately I'm not able to help you with mir-ion. But I see two questions here: keyword as struct field I believe this is the case for the most languages - keyword is not allowed as a variable name. usage of `version` keyword as a variable name IMO having `version` keyword in D is unexpected language design. No one will complain about keywords that are widely used in the industry (like `struct`, `int` etc). But `version`?! I hit this problem multiple times already and the only solution for me was to use `version_` instead.
Re: Tips on TCP socket to postgresql middleware
On Saturday, 19 February 2022 at 20:13:01 UTC, Chris Piker wrote: * general tips on which libraries to examine Most people will probably say this is crazy, but as to PG, one can do without libraries. I am doing so during years (in C, not D) and did not expierienced extremely complex troubles. I mean I do not use libpq - instead I implement some subset of the protocol, which is needed for particular program. What I do not like in all these libs for working with widely used services (postgres, redis etc) is the fact that they all hide inside them i/o stuff, including TCP-connect. Why have connect() in each library? It is universal thing, as well as read() and write(). If I want several connection to DBMS in a program, libraries like libpq compel me to use multithreading. But what if I want to do many-many-many things concurrently in a single thread? Usually I design more or less complex (network) programs using event-driven paradigm (reactor pattern) plus state machines. In other words programs designed this way are, so to say, hierarchical team of state machines, interacting with each other as well as with outer world (signals, timers, events from sockets etc)
Re: keyword as struct field
On Sunday, 20 February 2022 at 14:00:45 UTC, Basile B. wrote: I have a library solution based on opDispatch + __traits(getMember): Thank you, didn't know about opDispatch, so you have taught me something new today. But seems for my current need it can't be apllied (or I still don't understand how to). Anyway I have found in the mir-ion docs examples that it has @serdeKeys and @serdeKeyOut annotations that can do exactly what I need, bad part it is hard to find them or any documentation about them.
Re: keyword as struct field
On Sunday, 20 February 2022 at 11:08:55 UTC, partypooper wrote:
Hello, I'm new to D.
Title is self described, is it possible to use keyword as
struct field?
Maybe it is XYproblem, so here I will describe a little what I
need.
I'm parsing some json currently with
[mir-ion](https://code.dlang.org/packages/mir-ion)
(better/simpler suggestions for only json de/serialization?),
that has "version" as one of it's keys, so I can't use that as
field in my struct. Maybe some annotation in mir library to
parse "version" into other struct field name, can't find though
(its docs pretty concise).
I have a library solution based on opDispatch +
__traits(getMember):
```d
/**
* Mixin template allowing to use a field as if its identifier is
a D keyword.
* Note that this only works with `__traits(getMember)`.
* Params:
* keywordFieldPairs = An array of keyword and field pairs.
*/
template FieldOfKeyword(string[2][] keywordFieldPairs)
{
template opDispatch(string member)
{
static foreach (pair; keywordFieldPairs)
static if (member == pair[0])
mixin("alias opDispatch = ", pair[1], ";" );
}
}
///
unittest
{
struct Foo
{
mixin FieldOfKeyword!([["scope", "scope_"],["class",
"class_"]]);
string scope_;
string class_;
}
Foo foo;
__traits(getMember, foo, "scope") = "The Universe";
assert(__traits(getMember, foo, "scope") == "The Universe");
__traits(getMember, foo, "class") = "Atom";
assert(__traits(getMember, foo, "class") == "Atom");
}
```
never used it TBH.
keyword as struct field
Hello, I'm new to D. Title is self described, is it possible to use keyword as struct field? Maybe it is XYproblem, so here I will describe a little what I need. I'm parsing some json currently with [mir-ion](https://code.dlang.org/packages/mir-ion) (better/simpler suggestions for only json de/serialization?), that has "version" as one of it's keys, so I can't use that as field in my struct. Maybe some annotation in mir library to parse "version" into other struct field name, can't find though (its docs pretty concise).
Re: curl error msg
On Saturday, 19 February 2022 at 13:12:32 UTC, MichaelBi wrote: when running the example in the std.net.curl on my windows, got following msg: std.net.curl.CurlException@std\net\curl.d(4239): Failed to load curl, tried "libcurl.dll", "curl.dll" does it mean i don't have those dll files? thx. Are you running it from a WSL session? I get libcurl.dll errors (although a graphical popup) when I run my thing from in there, but it works well from a normal command console and/or Powershell.
Re: curl error msg
On Saturday, 19 February 2022 at 13:12:32 UTC, MichaelBi wrote: when running the example in the std.net.curl on my windows, got following msg: std.net.curl.CurlException@std\net\curl.d(4239): Failed to load curl, tried "libcurl.dll", "curl.dll" does it mean i don't have those dll files? thx. A libcurl.dll comes with the installer. It can be found by your program if it's located in the same folder as the executable or if the %PATH% environment variable contains the directory where the installer puts the DLL in. You may find this path in the environment variable but it has to match 32/64bit. If your 64bit program tries to load the 32bit variant it will fail to load. It's likely that your %PATH% environment variable contains the 32bit path (eg. for DMD `C:\install-dir\dmd2\windows\bin` but not the path to `\bin64`. So you have to adjust the environment variable or copy the DLL from this path to your program executable.
Re: Function Parameters without Names?
On Saturday, 19 February 2022 at 23:37:01 UTC, Vijay Nayar wrote: What is the main motivator to allow parameters with no names? 1) `extern(C) void* malloc(size_t);` No need for parameter name at all as that is only a declaration. You don't have an implementation thus don't need a name for the parameter. 2) As Ali says, unused parameters (i.e. in template specializations, virtual function overrides, dummy parameters for overload resolution purposes, etc.)
