Re: Non-blocking keyboard input
On Wednesday, 27 December 2023 at 13:27:53 UTC, Adam D Ruppe
wrote:
On Wednesday, 27 December 2023 at 05:07:04 UTC, Joe wrote:
??? Surely there there is
a one liner library solution for this?
It is not one line because it needs a bit of setup (and
teardown, but the objects' destructors do that for you) but it
is close:
http://arsd-official.dpldocs.info/arsd.terminal.html#single-key
`input.getch` waits for a single line, but you can use
`if(input.kbhit())` to see if it would block before calling it.
This shouldn't be hard... yet it is.
Better be careful, the mods are out in force deleting posts
this week that tell the hard truth. But yeah, the stdlib in D
has very little activity:
https://github.com/dlang/phobos/graphs/contributors?
So you can't expect much from it. My arsd libs provide a broad
set of functionality missing from it: stuff like this
terminal/console stuff, window creation, basic guis, web
servers, etc.
If you want to try them, you can use it from the dub system,
but I recommend just `git clone
https://github.com/adamdruppe/arsd.git` in your working
directory then import what you want and use `dmd -i` to
automatically include them in the build.
This does not actually work on my computer. It still blocks.
int itr = 0;
for(;;)
{
itr++;
writeln("1. closeKBThread = ", closeKBThread, ", iter = ",
itr);
if (closeKBThread) return;
string op = "";
string istr = "";
while (!closeKBThread)
{
writeln("2. ", input.kbhit(), " ",
closeKBThread);
if (input.kbhit())
{
istr ~= input.getch(false);
break;
}
}
writeln("final istr = ", istr);
1. closeKBThread = false, iter = 1
2. false false
2. false false
2. false false
2. false false
final istr = f
1. closeKBThread = false, iter = 2
2. false false < now blocking
All this is just junk of me trying to figure out what was going
on, but literally input.kbhit blocks after the first run(which
really means it's always blocking)
My original code was
while(true)
{
if (input.kbhit()) { istr ~= input.getch(); break; }
}
and I've been trying all kinds of stuff to figure out what was
going on but I believe it is the kbhit function itself. It calls
getch(true) and blocks on that as if I were just using getch
itself.
The issue is the same, the code does not block then after I hit a
key and it goes through an iteration of the outside loop it then
blocks waiting for the next input.
Non-blocking keyboard input
??? Surely there there is a one liner library solution for this? I have a program that spawns a thread for debugging information and uses the keyboard input which allows me to display the information. If I use getchar or readline then it blocks the thread. This is generally fine because that is all the thread does. The problem is that it also blocks the program termination as the main thread will not exit while the thread is running which is is because it's waiting on keyboard input stuck on getchar or fgetc or whatever. If I terminate the threads using thread_term then it terminates the program but the program then does not return the return code that it was successfully finished(because it was prematurely terminated by thread_term. Surely there is some type of peek for keyboards in D? I can't seem to get kbhit or because the library is not included(well, I tried to use one from dmc but it seemed to be invalid. snn.lib IIRC). This shouldn't be hard... yet it is.
D atomics
How does one really use atomics in D?
shared string[int] AA;
AA[s[2]].atomicStore(s[1]);
causes a hangup.
If I just assign it "works":
AA[s[2]] = s[1];
I've had similar issues where:
atomicOp!"+="(X[Y],1);
fails and I have to use
X[Y].atomicStore(1);
or
atomicOp!"+="(count,1); // fails (does not increment count)
so I have to create a new variable
int rcount = atomicFetchAdd(count,1);
and this will increment count.
In fact, I don't know what works or what doesn't except by trial
and error. I don't even know how to go about thinking about these
issues. In my mind, when using multiple threads and shared
variables I try to use atomics to avoid any race conditions.
Usually this involves incrementing or storing values. I try to
use atomics that replace the single threaded concept but they do
not always seem to work.
Are there any known issues or is it just me? Should it just work?
is there any good reference that gets ones mind right for
multithreading? I have some idea but I rarely do multithreading
until lately(I guess because I now have the HP to care). I think
the main issue I tend to have is trying to convert single
threaded code in to multiple threads and missing certain key
issues.
E.g.,
I was doing a parallel for and it wasn't really working(it seems
the task pool stalls the entire pool until all tasks are
complete):
for(s; parallel(arr))
{
// do stuff with s.
}
So I converted to using threads:
for(s; arr)
{
new Thread({
// do stuff with s.
}).start();
} // slightly more complex as I limited the total number of
threads
But as I spun up threads I would get different threads working on
the same s. I imagined that this likely had something to do with
the variable being on the stack being reused. so I wrapped the
thread in a function:
for(ss; arr)
{
void foo(typeof(ss) s)
{
new Thread({
// do stuff with s.
}).start();
}
foo(ss);
}
and this seemed to solve the problem. I imagine this is because
ss is duped by the function call essentially detaching it from
the loop code so it is unique across the thread.
But I guess I need to learn exactly how to deal with all these
problems rather than just hacking away.
Re: range shortened method not enabled, compile with compiler switch `-preview=shortenedMethods`
On Sunday, 24 September 2023 at 12:59:57 UTC, Steven Schveighoffer wrote: On Sunday, 24 September 2023 at 10:00:31 UTC, Joe wrote: For absolutely no reason I started getting this error. Last night I compiled the project and it worked just fine. This morning I made a single insignificant change and tried to compile and got that error. Only possible thing is that for some reason some change in updating the compiler may have no propagated correctly till after the reboot. [...] This sounds like a compiler issue. Are you on the latest compiler? Which compiler? Is there a set of code you can post or upload which causes the problem? -Steve I accidentally deleted my D folder containing the compiler and other things. (It as a weird issue where I was trying to add a command to an app and because the %1 parameter didn't have quotes I guess it deleted the D folder as it shortened it to D) I had a backup and didn't change anything since then except for updating the compilers. Still had the same visual D version. So I copied it over and updated dmd with visual D. Everything worked fine that day and I messed with the code several times. Woke up and wanted to make a simple change and code would not compile. The changes were insignificant. It gave me that error and when I added that switch then I got one about pragma(crt_constuctor) needing extern(C). Anyways, tried adding the extern(C) in the code but it wouldn't accept the changes. BUT I just tried to compile it again and no issues. Probably drive corruption issues or possibly some type of caching issue. But I was able to make changes now and compile it like it should work. E.g., it's possible that when I copied over the old D directory that it had an old compiler(I think it was from '22). I did update with visual D but maybe there was caching going on that was causing the conflict. In any case everything seems to be back to normal. I really didn't do much to fix anything so it was probably all caused by deleting that D folder accidentally.
Re: Vibe.d download function, how to get callback when done or error?
On Saturday, 23 September 2023 at 20:20:31 UTC, Christian Köstlin wrote: On 23.09.23 14:07, j...@bloow.edu wrote: I'm using download(url, filename) to download files in vibe.d. The issue is that I do not know when the download is finished or errors. There is a callback for the streaming side but not for the file download. A small test program shows, that if the function return normally the transfer was done (and the file saved). The function raises an exception if there is e,g, an http error status communicated. I am not sure what happens if the download is interrupted in the middle. I guess there will be an exception, but the file might be written partially. Kind regards, Christian I can't do any testing because overnight for no explicable reason the app will not compile any more. If it is synchronous then I should be able to do what I need without issue. When I initially was using it and the way the app works it seemed like it was running in parallel because I didn't see it execute synchronously because the tasks were relatively short and I just thought it was asynchronous for some reason(I didn't give it any thought at the time until I got errors in the download and needed to make the code more robust but I can't do anything until I figure out why the code no longer compiles(I made a post about it but it's not showing up yet).
Re: Vibe.d download function, how to get callback when done or error?
On Saturday, 23 September 2023 at 15:09:13 UTC, Elias wrote: On Saturday, 23 September 2023 at 12:07:38 UTC, Joe wrote: I'm using download(url, filename) to download files in vibe.d. The issue is that I do not know when the download is finished or errors. There is a callback for the streaming side but not for the file download. You don’t need a callback for that. It’s done when the download() returns. Pretty sure it is not synchronous. I'm running it many times. Maybe it is synchronous and I just didn't notice.
range shortened method not enabled, compile with compiler switch `-preview=shortenedMethods`
For absolutely no reason I started getting this error. Last night I compiled the project and it worked just fine. This morning I made a single insignificant change and tried to compile and got that error. Only possible thing is that for some reason some change in updating the compiler may have no propagated correctly till after the reboot. when I add the switch I then get. function `core.time.TickDuration.time_initializer` must be `extern(C)` for `pragma(crt_constructor)` dmd-2.105.2 EVERYTHING was working fine yesterday. All I did today was load up the project and change an integer value and I couldn't compile it ;/ Maybe corruption somewhere, I don't know. Anyways, I added extern(C) to that method in the location it says the file exists but no change. I removed the pragma, no change. Doesn't seem to actually be using that file. I have no idea what is going on. All I can do is try to reinstall the compiler and maybe redo the dub package. I'm using visual D + vibe.d + dub. Nothing fancy.
Vibe.d download function, how to get callback when done or error?
I'm using download(url, filename) to download files in vibe.d. The issue is that I do not know when the download is finished or errors. There is a callback for the streaming side but not for the file download.
Re: Help on array pointers
On Monday, 18 September 2023 at 02:49:37 UTC, vino wrote: On Sunday, 17 September 2023 at 18:28:36 UTC, Joe wrote: On Friday, 15 September 2023 at 16:55:34 UTC, Vino wrote: [...] [...] char[] invalid = (cast(char*)malloc(char.sizeof * len))[0..len]; This is not the way to go about it. You are mixing "pointer arrays" with "arrays". [...] Thank you very much, I am still newbie for programming and currently concentrating on Arrays/Struct/Pointers/Memory management. It will make more sense over time. It's actually really simple but it takes time to get used to thinking along the lines of the computer languages and such. My suggestion is simply do a lot of examples. Just type them in. As you type you think about what you are typing and how it relates. Then you will start to notice the patterns and such. I wouldn't try to create new programs from scratch because it's easy to make mistakes that can be very hard to debug. D might not be the best language to start learning programming. E.g., might be better to go with javascript, lua, or python and get a nice rapid development environment. Even though D is fast to compile it's not that fast. Maybe use Repl which lets you code line by line essentially. All languages more or less are the same but just look different so it you want to find out one that has the most convenience. you don't want to bite off too much as it will slow you down. Ultimately it's just like anything else though in that you are learning a language and all it's nuances. You ultimately have to learn how to translate your thoughts and desires in to the correct syntax so you can make the language useful and act as a tool. If you try to us it as a tool before you learn to mimic others it will be very slow and frustrating. Because most languages are 99% the same with just different syntax(just about every major programming language can do what ever other one can, it's just some are better/easier at doing certain things than others). You might try learning several languages at once as it can actually speed things up. Learn how to write simple programs(hello worlds) then modify those to do more and then learn control structures(branching(if's), loops(for)), variables, etc. Build up slowly and everything will make sense. Build up fast and you'll get lost at some point and then spend a lot of time trying to figure out how to make sense of it. Ideally you want an ide and language that gives you precise info on errors that you make else it can be hard to track down mistakes because sometimes errors can be hidden. D is not good for. Even Q-basis might be better to start with or try to find Turbo Pascal if it still exists. What makes D great is it's generic/template programming and that is a more complex topic that you probably won't get much in to for a few years(although it is not really hard it will seem like it at first). D has a lot of features that hard core programmers like and so you are not going to be using those features for a while so it might not be worth using D to learn off of. Not saying you can't do it but it might be slow. You could always do it in parallel. E.g., learn to write simple programs in many languages then build up. E.g., for i=1:10 // Matlab for(auto i = 1; i < 10; i++) // D for(int i = 1; i < 10; i++) // C for(var i = 1; i < 10; i++) // JS for i from 1 to 10 // maple loop(50, ...) // JSFX for i = 1,10 // lua etc. The underlying idea is that one wants a way to do the same thing over and over and so loops exist to handle that pattern. Each language uses different ways to express a loop. Just like languages, all languages uses nouns, verbs, etc but express them differently.
Re: Help on array pointers
On Friday, 15 September 2023 at 16:55:34 UTC, Vino wrote:
On Friday, 15 September 2023 at 15:27:00 UTC, Vino wrote:
On Friday, 15 September 2023 at 02:25:09 UTC, Joe wrote:
On Thursday, 14 September 2023 at 14:21:09 UTC, Vino wrote:
[...]
A pointer is a type that points to something. It's literally
that simple. Every piece of data and code exist somewhere in
memory. Every piece of memory has an address. The address is
what a pointer contains. Sometimes we want a type that is the
address itself rather than a value/number.
[...]
Hi Joe,
Thank you very much for the explanation can you please correct
me if my understanding is incorrect
```
byte[] z; // Creates an array of bytes. That is, the compiler
will create a pointer to an array of memory and track it's
length and deal with memory allocation and all that.
```
If we use the above method then :
The compiler takes care of initilizing the array and free the
memory after the usage.
And this is the recommended method.
```
We can use a pointer as an array also, this is the "old school
way of creating arrays".
int qlen = 5;
int* q = cast(int*)malloc(int.sizeof*qlen);
```
If we use the above method then :
We need to manual initilize the array.
Ensure that the memory is freed after the usage using
try/final block.
By default the memory allocation for arrays in D is based on
GC (expect for std.array containers) if we want to
reduce/avoid GC then we need to use the old school way of
creating the arrays.
In case of using the old school ways then can you guide me
what is wrong in my earlier code that I am getting the below
error and how do I correct the same.
Error
```
Invalid Name passed: /T
double free or corruption (out)
Error: program killed by signal 6
```
Hi All,
At last was able to resolve the issue, but not sure
whether this is the reight solution.
Code:
```
import std.stdio: writeln;
import std.exception: enforce;
import std.range: empty;
import std.format: format;
auto ref testNames(in string[] names) {
enforce(!empty(names), "Names cannot be Empty or Null");
import core.stdc.stdlib;
import std.algorithm: any, canFind;
string[] _names;
size_t len = 19;
char[] invalid = (cast(char*)malloc(char.sizeof *
len))[0..len];
invalid[0 ..len] = 0; //Array Initlization
try {
version(Posix) {
invalid =
['\'','\"',':',';','*','&','/','[',']','-','+','$','#','<','>','{','}','(',')'];
}
foreach(i; names.dup) {
auto result = i.any!(a => invalid.canFind(a));
if(result) { throw new Exception("Invalid Name
passed: %s".format(i)); }
else {_names = names.dup; return _names; }
}
} catch(Exception e) { writeln(e.msg); }
finally { invalid = null; free(invalid.ptr); } //
added invalid = null resolved the issue
return _names;
}
void main () {
writeln(testNames(["/T"]));
}
From,
Vino
char[] invalid = (cast(char*)malloc(char.sizeof * len))[0..len];
This is not the way to go about it. You are mixing "pointer
arrays" with "arrays".
You are creating a pointer array then turning that in to an
array. There is no need to do that. Basically you are copying
what the compiler already does for you.
When you are using arrays([] language) you don't have to worry
about anything. Just use them as arrays directly and let the
compiler deal with memory management. The entire point of
"managed arrays" is to avoid having to manually deal with memory
which can cause problems if one is not careful.
Of course you have to make sure your array used used correctly
but that should be obvious.
char[] invalid = (cast(char*)malloc(char.sizeof * len))[0..len];
is exactly the same as
char[] invalid. EXCEPT that you've forced it to be initialized
with a length of len and went from D in to C then back to D just
to create an array. You are making it much more complex than it
needs to be. There are good tutorials on D arrays that you should
look over. You will see how easy they are.
You could just do char[] invalid = new char[](len); for the same
effect but is nicer. New understands char, malloc only
understands bytes. You should not use malloc in D unless you
explicitly know you need to use it. Malloc is primitive memory
management.
When you use D's symantics for arrays it is much easier... which
is the entire point. You don't even need to free the arrays. D
knows they are local variable sand will free them when out of
scope. It will grow them when they need to grow, etc.
invalid = null; free(invalid.ptr);
That does nothing. you set invalid.ptr to null and are freeing
null so you are not freeing the array.
You don't have to though, the array will be free out of scope(not
in final though so exceptions could cause memory leak) because it
is a D array.
Do not
Re: Help on array pointers
On Thursday, 14 September 2023 at 14:21:09 UTC, Vino wrote:
Hi All,
Request your help to guide me in understanding about
pointers, the below code works,I have few question which i need
your help for better understanding.
Questions:1
```
char[] invalid = (cast(char*)malloc(char.sizeof *
length))[0..length];
```
The above statement allocate memory for char type and the size
of the allocated memory is char.sizeof * length so what is the
use of this "[0..length]";
Question:2
```
char[]* invalidptr =
```
Is this the right way to create a array pointer.
Question: 3
```
ST1: char[] invalid = (cast(char*)malloc(char.sizeof *
length))[0..length];
ST2: char[]* invalid = (cast(char*)malloc(char.sizeof *
length))[0..length];
```
What is the difference between the above to statement.
Question: 4
Who do we free the memory allocated.
Code:
```
auto ref testNames(in string[] names) {
enforce(!empty(names), "Names cannot be Empty or Null");
import core.stdc.stdlib;
import std.algorithm: any, canFind;
size_t length = 20;
char[] invalid = (cast(char*)malloc(char.sizeof *
length))[0..length];
char[]* invalidptr =
version(Windows) { (*invalidptr) =
['\'','\"',':',';','*','&','[',']','-','+','$','#','<','>','{','}','(',')']; }
foreach(i; names.dup) {
auto result = i.any!(a => (*invalidptr).canFind(a));
if(result) { throw new Exception("Invalid Name passed:
%s".format(i)); }
}
string[] _names = names.dup;
return _names;
}
```
From,
Vino
A pointer is a type that points to something. It's literally that
simple. Every piece of data and code exist somewhere in memory.
Every piece of memory has an address. The address is what a
pointer contains. Sometimes we want a type that is the address
itself rather than a value/number.
https://run.dlang.io/gist/19c63d325ee412df23bdbefabce111b9
import std, std.stdio, core.stdc.stdlib;
void main()
{
float x = 43.534f; // Creates a float type that hold
sthe value 43.534
writeln("Value of x = ", x); // Displays it's value
writeln("Address of x = ", ); // Displays it's address
writeln();
float* y = null; // creates a pointer of type float note
that it is point to no memory location(null) and so it is an
error to store a value to the location
writeln("Value of y = ", y); // Displays it's
address(remember, y is a ptr to a float, not a float)
y = new float(); // This allocates free/unused memory(of
float type) to y so we can store a value.
*y = 50.34f; // assigns it the value 50.34. We have to
use * because if we didn't we would be trying to change the
address of y
writeln("Value of y = ", y); // Displays it's
address(remember, y is a ptr to a float, not a float)
writeln("Address of y = ", ); // Writes it's address(y
is a pointer but it also is stored in a location in memory and
hence has an address
writeln("Dereferenced value of y = ", *y); // Displays
the value y is pointing to interpreting it as a float(the base
type of the pointer)
writeln("Dereferences the value of y as if it were an int
= ", *(cast(int*)y)); // Displays the value y is pointing to
interpreting it as a int. We have to force the compilier to
reinterpret it as an int*.
writeln();
byte[] z; // Creates an array of bytes. That is, the
compiler will create a pointer to an array of memory and track
it's length and deal with memory allocation and all that.
writeln("Value of z = ", z); // Displays it's
address(remember, z is a ptr to a float, not a float)
writeln("Address of z = ", ); // Note that z is an
array but it is a pointer to and gets the address where the
"array" is stored.
writeln("Value of z's pointer = ", z.ptr); // Displays
it's address(remember, y is a ptr to a float, not a float)
writeln("Length of z = ", z.length); // Writes it's
address(y is a pointer but it also is stored in a location in
memory and hence has an address
writeln();
z ~= 4; // We can store/append a value to our array. The
compiler will take care of dealing with allocating memory and all
that.
writeln("Value of z = ", z); // Displays it's
address(remember, z is a ptr to a float, not a float)
writeln("Address of z = ", ); // Note that z is an
array but it is a pointer to and gets the address where the
"array" is stored.
writeln("Value of z's pointer = ", z.ptr); // Displays
it's address(remember, y is a ptr to a float, not a float)
writeln("Length of z = ", z.length); // Writes it's
address(y is a pointer but it also is stored in a location in
memory and hence has an address
z ~= 54; // We can store/append a value to our array. The
compiler will take
Re: parallel threads stalls until all thread batches are finished.
On Monday, 28 August 2023 at 22:43:56 UTC, Ali Çehreli wrote: On 8/28/23 15:37, j...@bloow.edu wrote: > Basically everything is hard coded to use totalCPU's parallel() is a function that dispatches to a default TaskPool object, which uses totalCPUs. It's convenient but as you say, not all problems should use it. In such cases, you would create your own TaskPool object and call .parallel on it: https://youtu.be/dRORNQIB2wA?t=1611s Ali Thanks. Seems to work. Didn't realize it was that easy ;)
Re: parallel threads stalls until all thread batches are finished.
On Monday, 28 August 2023 at 10:33:15 UTC, Christian Köstlin
wrote:
On 26.08.23 05:39, j...@bloow.edu wrote:
On Friday, 25 August 2023 at 21:31:37 UTC, Ali Çehreli wrote:
On 8/25/23 14:27, j...@bloow.edu wrote:
> "A work unit is a set of consecutive elements of range to be
processed
> by a worker thread between communication with any other
thread. The
> number of elements processed per work unit is controlled by
the
> workUnitSize parameter. "
>
> So the question is how to rebalance these work units?
Ok, your question brings me back from summer hibernation. :)
This is what I do:
- Sort the tasks in decreasing time order; the ones that will
take the most time should go first.
- Use a work unit size of 1.
The longest running task will start first. You can't get
better than that. When I print some progress reporting, I see
that most of the time N-1 tasks have finished and we are
waiting for that one longest running task.
Ali
"back to sleep"
I do not know the amount of time they will run. They are files
that are being downloaded and I neither know the file size nor
the download rate(in fact, the actual download happens
externally).
While I could use work unit of size 1 then problem then is I
would be downloading N files at once and that will cause other
problems if N is large(and sometimes it is).
There should be a "work unit size" and a "max simultaneous
workers". Then I could set the work unit size to 1 and say the
max simultaneous workers to 8 to get 8 simultaneous downloads
without stalling.
I think thats what is implemented atm ...
`taskPool` creates a `TaskPool` of size `defaultPoolThreads`
(defaulting to totalCPUs - 1). The work unit size is only there
to optimize for small workloads where task / thread switching
would be a big performance problem (I guess). So in your case a
work unit size of 1 should be good.
Did you try this already?
Kind regards,
Christian
Well, I have 32 cores so that would spawn 64-1 threads with hyper
threading so not really a solution as it is too many simultaneous
downs IMO.
"These properties get and set the number of worker threads in the
TaskPool instance returned by taskPool. The default value is
totalCPUs - 1. Calling the setter after the first call to
taskPool does not changes number of worker threads in the
instance returned by taskPool. "
I guess I could try to see if I can change this but I don't know
what the "first call" is(and I'm using parallel to create it).
Seems that the code should simply be made more robust. Probably a
just a few lines of code to change/add at most. Maybe the
constructor and parallel should take an argument to set the
"totalCPUs" which defaults to getting the total number rather
than it being hard coded.
I currently don't need or have 32+ downlaods to test ATM so...
this() @trusted
{
this(totalCPUs - 1);
}
/**
Allows for custom number of worker threads.
*/
this(size_t nWorkers) @trusted
{
Basically everything is hard coded to use totalCPU's and that is
the ultimate problem. Not all tasks should use all CPU's.
What happens when we get 128 cores? or even 32k at some point?
It shouldn't be a hard coded value, it's really that simple and
where the problem originates because someone didn't think ahead.
Re: parallel threads stalls until all thread batches are finished.
On Friday, 25 August 2023 at 21:43:26 UTC, Adam D Ruppe wrote: On Wednesday, 23 August 2023 at 13:03:36 UTC, Joe wrote: to download files from the internet. Are they particularly big files? You might consider using one of the other libs that does it all in one thread. (i ask about size cuz mine ive never tested doing big files at once, i usually use it for smaller things, but i think it can do it) The reason why this causes me problems is that the downloaded files, which are cashed to a temporary file, stick around and do not free up space(think of it just as using memory) and this can cause some problems some of the time. this is why im a lil worried about my thing, like do they have to be temporary files or can it be memory that is recycled? The downloading is simply a wrapper that provides some caching to a ram drive and management of other things and doesn't have any clue how or what is being downloaded. It passes a link to something like youtube-dl or yt-dlp and has it do the downloaded. Everything works great except for the bottle neck when things are not balancing out. It's not a huge deal since it does work and, for the most part, gets everything downloaded but sorta defeats the purpose of having multiple downloads(which is much faster since each download seems to be throttled). Increasing the work unit size will make the problem worse while reducing it to 1 will flood the downloads(e.g., having 200 or even 2000 downloads at once). Ultimately this seems like a design flaw in ThreadPool which should auto rebalance the threads and not treat the number of threads as identical to the worker unit size(well, length/workerunitsize). e.g., suppose we have 1000 tasks and set worker unit size to 100. This gives 10 workers and 10 workers will be spawned(not sure if this is limited to total number of cpu threads or not) What would be nice is to be able to set worker unit size to 1 and this gives 1000 workers but limit concurent workers to, say 10. So we would have at any time 10 workers each working on 1 element. When one gets finished it can be repurposed for any unfinished tasks. The second case is preferable since there should be no issues with balancing but one still gets 10 workers. The stalling comes from the algorithm design and not anything innate in the problem or workload itself.
Re: parallel threads stalls until all thread batches are finished.
On Friday, 25 August 2023 at 21:31:37 UTC, Ali Çehreli wrote: On 8/25/23 14:27, j...@bloow.edu wrote: > "A work unit is a set of consecutive elements of range to be processed > by a worker thread between communication with any other thread. The > number of elements processed per work unit is controlled by the > workUnitSize parameter. " > > So the question is how to rebalance these work units? Ok, your question brings me back from summer hibernation. :) This is what I do: - Sort the tasks in decreasing time order; the ones that will take the most time should go first. - Use a work unit size of 1. The longest running task will start first. You can't get better than that. When I print some progress reporting, I see that most of the time N-1 tasks have finished and we are waiting for that one longest running task. Ali "back to sleep" I do not know the amount of time they will run. They are files that are being downloaded and I neither know the file size nor the download rate(in fact, the actual download happens externally). While I could use work unit of size 1 then problem then is I would be downloading N files at once and that will cause other problems if N is large(and sometimes it is). There should be a "work unit size" and a "max simultaneous workers". Then I could set the work unit size to 1 and say the max simultaneous workers to 8 to get 8 simultaneous downloads without stalling.
Re: parallel threads stalls until all thread batches are finished.
On Wednesday, 23 August 2023 at 14:43:33 UTC, Sergey wrote:
On Wednesday, 23 August 2023 at 13:03:36 UTC, Joe wrote:
I use
foreach(s; taskPool.parallel(files, numParallel))
{ L(s); } // L(s) represents the work to be done.
If you make for example that L function return “ok” in case
file successfully downloaded, you can try to use TaskPool.amap.
The other option - use std.concurrency probably.
I think I might know what is going on but not sure:
The tasks are split up in batches and each batch gets a thread.
What happens then is some long task will block it's entire batch
and there will be no re-balancing of the batches.
"A work unit is a set of consecutive elements of range to be
processed by a worker thread between communication with any other
thread. The number of elements processed per work unit is
controlled by the workUnitSize parameter. "
So the question is how to rebalance these work units?
E.g., when a worker thread is done with it's batch it should look
to help finish that batch rather than terminating and leaving all
the work for the last thread.
This seems like a flaw in the design. E.g., if one happens to
have n batches and every batch but one has tasks that finish
instantly then essentially one has no parallelization.
Re: parallel threads stalls until all thread batches are finished.
On Wednesday, 23 August 2023 at 14:43:33 UTC, Sergey wrote:
On Wednesday, 23 August 2023 at 13:03:36 UTC, Joe wrote:
I use
foreach(s; taskPool.parallel(files, numParallel))
{ L(s); } // L(s) represents the work to be done.
If you make for example that L function return “ok” in case
file successfully downloaded, you can try to use TaskPool.amap.
The other option - use std.concurrency probably.
Any idea why it is behaving the way it is?
parallel threads stalls until all thread batches are finished.
I use
foreach(s; taskPool.parallel(files, numParallel))
{ L(s); } // L(s) represents the work to be done.
to download files from the internet.
Everything works. The issue is this:
the foreach will download 8 files at once. BUT it will not start
the next batch of 8 *until* ALL of the previous 8 are done. It
seems that taskPool.parallel will not immediately start a new
thread once a task is done
E.g., I get
L(s1);
L(s2);
...
L(s8);
--- // nothing below is executed until all L(s1) through L(s8)
are finished.
L(s9);
L(s10);
...
My expectation is that, say, when the first task is complete, say
L(s4), that L(s9) is then executed.
The reason why this causes me problems is that the downloaded
files, which are cashed to a temporary file, stick around and do
not free up space(think of it just as using memory) and this can
cause some problems some of the time. Also, the point of parallel
tasks is to allow paralleling but the way the code is working is
that it starts the tasks in parallel but then essentially stalls
the paralleling a large portion of the time. E.g.,
If there are a bunch of small downloads but one large one, then
that one large download stalls the everything. E.g., say L(s5) is
a very long download while all others are very quick. Then L(s5)
will prevent downloading anything afterwards until it is
finished(I'll get L(s1) through L(s8) but nothing else until
L(s5) is finished).
What's going on and how to reconcile?
Re: Code organization, dub, etc.
On Monday, 13 March 2023 at 13:58:29 UTC, Adam D Ruppe wrote: I'm not particularly interested in defending dub - i consider it a useless piece of crap that I only suffer through cuz some users demanded it For the record, I wasn't trying to attack dub (or dfmt). I was more interested in determining whether DLang (the Foundation and/or the community) would consider working with the CMake and clang-format communities to get them to support D in their products, or whether they prefer to stick with what they have in terms of tooling. I think I heard someone at DConf 22 ask about getting Intellij support for D and Walter said that JetBrains would ask for funding to do that. I don't think CMake support/development is in the same boat, is it?
Re: Code organization, dub, etc.
On Monday, 13 March 2023 at 13:32:04 UTC, Mike Parker wrote: The package registry is full of libraries, yes. That's what it's primarily for. There aren't a lot of executables uploaded there because they're usually better distributed in other ways. But plenty of people are using dub to build them. One way to handle multiple executables is to write a simple script that makes multiple calls to dub with the configurations you need. That's essentially the same as using a Makefile (or CMake with custom commands) to build your project. What I had in mind when I mentioned "multi-executable projects" was something like Postgres or say, a tenth of that, e.g., a server executable, two or more client and utility executables and one or more libraries, all of it spread over a few directories, possibly to a depth of say, two from the root. Can dub handle something like that, e.g., can it handle nested dub.json's, or wouldn't it make much more sense to build such a thing with make, cmake, meson?
Re: Code organization, dub, etc.
On Monday, 13 March 2023 at 12:56:57 UTC, Bradley Chatha wrote: For better or for worse we're stuck with dub as the standard package manager + build tool one-in-all for most of our open source libraries. Yeah, it seems like it's *only* for libraries (and a few single-exe utilities). Looking at code.dlang.org, under "Stand-alone applications/Server software", the top rated item is "handy-httpd" which according to its dub.json builds a library! And the second place "voxelman" is builds three libraries and one executable, which appears to be a "launcher" to access the libraries as plugins.
Re: Passing and returning arguments by ref
Thanks, Ali.
On Friday, 3 March 2023 at 18:09:01 UTC, Ali Çehreli wrote:
Think may be due to D not having reference variables. Sometimes
one needs to use pointers.
Ah! I'm about five chapters away from Pointers ;-).
Actually, I had tried changing B.x1() to:
`ref X x1() { return [0]; }`
but the compiler didn't accept it.
It's a bit weird that by taking the address of calling B.x1() and
thus getting an X*, I had to *dereference* that to pass it to the
helper function of A.mfd() which actually takes a `ref C`
argument.
Passing and returning arguments by ref
Let's say we have two classes, A and B. The latter has a dynamic
array of X and type X has an add() method that can be used to
append elements (of type C, another struct) to X's own dynamic
array of C. So it's something like the following:
```d
struct C {}
struct X { C[] cs;
void add(C c) { cs ~= c; }
}
class A {
X x;
void mfd(ref B b) {
// manipulate A.x and one B.xs (returned by B.x1()
// by calling another member function
// that calls X.add(some C)
}
}
class B {
X[] xs;
this() { xs.length = 1; }
ref X x1() { return xs[0]; }
}
```
After A and B are instantiated and A.mfd() is called, the changes
to A.x are visible in main. However, although the changes to the
X returned by B.x1() are visible while A.mfd() executes, the
B.xs[0] is unchanged after it returns.
My understanding was that since A, B and X[] are all reference
types, this ought to work, but obviously something is missing.
std.math log and family
I've discovered that the 'log' function as well several similar functions are only defined as taking a real argument and returning a real, unlike most other std.math functions, which have triple definitions (also supporting double and float types). This created a problem when I tried compiling code like the following: --- import std.math; alias double function(double) Funcptr; Funcptr sinptr = Funcptr tanptr = Funcptr logptr = --- dmd (and ldc2) report test.d(7): Error: cannot implicitly convert expression & log of type real function(real x) pure nothrow @nogc @safe to double function(double) [ldc2 also reports test.d(6): Error: cannot implicitly convert expression & tan of type real function(real x) pure nothrow @nogc @trusted to double function(double) but apparently this is an LDC-only problem] I'd like to know if the lack of double/float versions of 'log', 'log10', etc. are intentional, i.e., there's some rationale behind it, or an oversight. I'd also like to know the proper/best way to deal with the error, considering the real code embeds the 'Funcptr's in an array of structs. Is it better to write a function 'mylog' that internally casts the argument and the return value, or is there some safe way to cast away the 'log' function at the point of initializing the array? Also, is it preferable to post the first question (intentional/oversight) in the Phobos forum? And what is the preferred way of reporting the LDC problem, via GitHub?
Re: Dlang tour - Unittesting example
On Tuesday, 2 October 2018 at 13:24:09 UTC, Basile B. wrote:
The problem is the NaN madness.
Since several values are NaN there's this strange stuff:
void main()
{
import std.stdio;
import std.math : isNaN;
double d;
writeln(d.init);// nan
writeln(d); // nan
writeln(d.nan); // nan
assert(d.isNaN);
assert(d == d.nan); // fails
assert(d == d.init); // fails
}
the last assert is just crazy.
OK, so changing the example from
// .init a special built-in property that
// returns the initial value of type.
assert(vec.x == double.init);
to
import std.math : isNaN;
assert(vec.x.isNaN);
doesn't cause the crash. Although I'm a bit puzzled still, I
thought (coming from Python) that when you ran the unittest
version it would report which tests passed and which failed, not
run through main().
Re: Dlang tour - Unittesting example
On Tuesday, 2 October 2018 at 12:25:19 UTC, Joe wrote: On Tuesday, 2 October 2018 at 09:59:28 UTC, bauss wrote: On Tuesday, 2 October 2018 at 04:13:01 UTC, Joe wrote: There appears to be a problem with the example at https://tour.dlang.org/tour/en/gems/unittesting If compiled with -unittest, the resulting program crashes. It happens with ldc2 on Linux but it can also be seen if you click on "Export" and run it with dmd -unittest. I think it's more likely a problem with your OS. I am unable to reproduce that with either of dmd or ldc. Well then it's also a problem with run.dlang.io, since as I said it also happens when you run it there. I forgot to mention: at the end, it reports "1/1 unittests FAILED". I see three tests--two in the struct and the separate one--but the assertion failure is in line 49 (the standalone) so apparently the other two are not being run (or reported).
Re: Dlang tour - Unittesting example
On Tuesday, 2 October 2018 at 09:59:28 UTC, bauss wrote: On Tuesday, 2 October 2018 at 04:13:01 UTC, Joe wrote: There appears to be a problem with the example at https://tour.dlang.org/tour/en/gems/unittesting If compiled with -unittest, the resulting program crashes. It happens with ldc2 on Linux but it can also be seen if you click on "Export" and run it with dmd -unittest. I think it's more likely a problem with your OS. I am unable to reproduce that with either of dmd or ldc. Well then it's also a problem with run.dlang.io, since as I said it also happens when you run it there.
Dlang tour - Unittesting example
There appears to be a problem with the example at https://tour.dlang.org/tour/en/gems/unittesting If compiled with -unittest, the resulting program crashes. It happens with ldc2 on Linux but it can also be seen if you click on "Export" and run it with dmd -unittest.
Re: Access to structures defined in C
On Tuesday, 18 September 2018 at 13:47:50 UTC, Atila Neves wrote:
On Tuesday, 18 September 2018 at 02:39:39 UTC, Joe wrote:
The second type is like that shown above. The first is a
simpler array of pointers to int, e.g.,
int *yp = {2, 4, 0};
int *yq = {10, 12, 0};
This is valid C in the sense that it compiles, but I doubt it
does what you think it does. This is equivalent code:
int *yp = 2;
int *yq = 10;
Sorry, Atila, I got confused looking at my two cases. I should
have said "an array of ints", e.g.,
int yp[] = {2, 4, 0};
int yq[] = {10, 12, 0};
int *ys[] = {yp, yq, 0};
This isn't even valid C code.
It is, because C treats 'yp' as a pointer.
In D, I first declared these as
int[] yp = [2, 4];
int[] yq = [10, 12];
__gshared int*[] ys = [ , ];
D dynamic arrays are not equivalent to C arrays.
It's hard to see what you're trying to do with the code you
posted. Have you tried instead to use a tool to translate the C
headers?
At this point, I've translated everything, even the code above. I
had to use 'immutable(int [])' in the second and higher level
arrays like 'ys' so that they could refer to 'yp' and 'yq'
(without the address operators).
I still have to make several changes to the library code because
these arrays were declared as dynamic arrays of pointers, of size
1, to bypass bounds checking. Now they're proper dynamic arrays,
and I can use foreach on them and get other D benefits.
However, I still would like to have a deeper understanding of the
"static variable yp cannot be read at compile time" error
messages which went away when I declared yp immutable.
Re: Access to structures defined in C
On Sunday, 10 June 2018 at 17:59:12 UTC, Joe wrote:
That worked but now I have a more convoluted case: a C array of
pointers to int pointers, e.g.,
int **xs[] = {x1, x2, 0};
int *x1[] = {x1a, 0};
int *x2[] = {x2a, x2b, 0};
...
int x2a[] = { 1, 3, 5, 0};
Only the first line is exposed (and without the
initialization). So I tried:
extern(C) __gshared extern int**[1] xs;
After a long hiatus, I'm back to working on something related to
the above, but now that various other C pieces have been
converted to D I'm down to converting these static arrays to D.
There are two arrays that are giving me trouble. The second type
is like that shown above. The first is a simpler array of
pointers to int, e.g.,
int *yp = {2, 4, 0};
int *yq = {10, 12, 0};
int *ys[] = {yp, yq, 0};
In D, I first declared these as
int[] yp = [2, 4];
int[] yq = [10, 12];
__gshared int*[] ys = [ , ];
The compiler (ldc2) gave errors like "cannot take address of
thread-local variable yp at compile time" or "static variable yp
cannot be read at compile time" (when I replaced the "" by
"yp[0]". Eventually, I managed to get them to compile without
errors by using the following:
immutable int[] yp = [2, 4];
immutable int[] yq = [10, 12];
__gshared immutable(int[])[] ys = [ yp, yq ];
I still haven't tested them (or linked them) so I don't know what
other changes I'll have to make to the library (now in D) where
ys is declared as:
__gshared extern int*[] ys;
Presumably changing it to
__gshared extern immutable(int[])[] ys;
should do the trick (everything is still within extern (C))? But
I suspect I'll have to change several array access statements as
well.
However, I've been unable to compile the other case, which now
looks like this:
immutable int x1a[] = [ 9, 7, 5];
immutable(int[])[] x1 = [ x1a ];
immutable(int[])[] x2a = [ 1, 3, 5];
...
immutable(int[])[] x2 = [ x2a, x2b ];
__gshared immutable(int[])[][] xs = [ x1, x2 ];
The error is like "static variable x2a cannot be read at compile
time". It seems like I would have to wrap that immutable(int[])
within another immutable, as
immutable(immutable(int[])[]) x2 ...
and extend that to xs as well.
I'll try it later but I'd like some confirmation or better yet,
pointers to where this is explained in some comprehensible way,
i.e., on what can you apply an address operator or array
subscript and when, is the behavior differerent for immutable,
const, static, thread-local and why?
Re: Linking a C program with D library
On Wednesday, 15 August 2018 at 06:39:50 UTC, Mike Parker wrote: String literals are implicitly convertible to const(char)* and are guaranteed to be nul-terminated like a C string, so this works: [...] Does that help? Yes, indeed. I think I possibly read about literal strings being nul-terminated somewhere but it must've slipped my mind.
Re: Linking a C program with D library
On Wednesday, 15 August 2018 at 01:56:34 UTC, Mike Parker wrote:
The correct thing to do is to keep the original C function
signatures in the converted code, i.e. don't change char* to
string[]. And don't use anything from Phobos internally that
requires linking. In other words, treat your converted code as
C code in a D module as much as possible. Only when the
conversion is complete and everything is in D do you start
pulling in the D features.
I understand that, Mike. However if I'm not mistaken given
something in C like
char* strs[] = { "This", "is a", "test"};
AFAIK, even with -betterC and an extern (C), the literals will
still be understood by D as type "string", and there is no other
way around it, right?
I could put the array and the function in its own C file for the
time being, but instead chose to replace the toStringz by a small
hack: use memcpy to copy the string to a stack fixed, big enough
array and a NUL terminator.
Linking a C program with D library
I'm attempting a piecemeal conversion of some C programs. I've converted a few that depend on C modules that are in a library and now I'm sort of in the middle of converting those C modules. One of them has an array of strings, i.e., array of char*, which most easily translated to D's string[]. However, the array is used by a function that expects a const char*, so I had to use toStringz to pass those values. Now there are some C programs that haven't been converted yet, but they depend on the function above. When linking those programs, the linker complains about an undefined reference to the mangled name of std.string.toStringz. What is the most appropriate way around this? Is there a way to declare an array of const char* and initialize it to literal strings? Or can the mangled name (and Phobos library) somehow be made visible to the linker?
?? How to subscribe to Multicast Broadcasts ??
Hello All!
I've been trying every possible combination and cannot get
anything working. (>_<)
This is I think the closest I've got, I think the problem may be
with the 3 argument. I'm not sure how to join the Multicast IP
membership?
(this code currently does not work - throws error:
'Unable to set socket option: An invalid argument was supplied'.)
Socket listen_socket = new Socket(AddressFamily.INET,
SocketType.DGRAM, ProtocolType.UDP);
listen_socket.setOption(SocketOptionLevel.IGMP,
SocketOption.IPV6_JOIN_GROUP, 1);
auto adr = getAddress("0.0.0.0", "56000");
listen_socket.bind(adr[0]);
This is how the messages are sent (which works fine):
Socket udp_broadcast_soc = new UdpSocket();
udp_broadcast_soc.setOption(SocketOptionLevel.SOCKET,
SocketOption.BROADCAST, 1);
auto adr = getAddress("239.192.0.10", 54000);
udp_broadcast_soc.sendTo(, adr[0]);
Please Please Please Help, I am desperate!
Many Thanks in Advance for your time & attention,
-joe
Re: How to call a C function from D that takes a FILE * as an argument?
On Wednesday, 4 July 2018 at 02:16:00 UTC, Seb wrote:
Hmm, calling e.g. fprintf with stdout should just work:
---
void main()
{
import core.stdc.stdio;
fprintf(stdout, "Hello %s", "world".ptr);
}
---
Could you maybe provide your whole code?
This short test program shows the error:
---
import std.stdio;
void main()
{
extern (C) void list(FILE *fd);
list(stdout);
}
---
Now I fixed this by changing the import to core.stdc.stdio. I
guess the problem is if you import std.stdio (which brings in the
other one), there are two slightly incompatible stdout's and the
D takes precedence.
Re: How to call a C function from D that takes a FILE * as an argument?
On Wednesday, 4 July 2018 at 01:58:15 UTC, Seb wrote: So just add the declaration to your D file: --- extern(C) void myCfunction(FILE* stream); --- I do have a similar declaration in D. It appears the problem is that the C program I'm trying to convert passes stdout as the argument and the D compiler complains somewhat like the following: Error: function foo.main.myCfunction (shared(_IO_FILE)* stream) is not callable using argument types (File) So I guess the question is what to pass instead of stdout.
How to call a C function from D that takes a FILE * as an argument?
The subject basically says it all. The C function uses the argument to call fprintf and also passes it to other functions where it's used to call fileno, fprintf or putc.
Re: Access to structures defined in C
On Sunday, 10 June 2018 at 17:59:12 UTC, Joe wrote:
That worked but now I have a more convoluted case: a C array of
pointers to int pointers, e.g.,
int **xs[] = {x1, x2, 0};
int *x1[] = {x1a, 0};
int *x2[] = {x2a, x2b, 0};
...
int x2a[] = { 1, 3, 5, 0};
Only the first line is exposed (and without the
initialization). So I tried:
extern(C) __gshared extern int**[1] xs;
The D compiler accepts that, but just about any manipulation
gets screamed at, usually with Error: only one index allowed to
index int. Note that I'm trying to access the ints, i.e., in C
something like xs[1][0][2] to access the 5 in x2a. Do I have to
mimic the intermediate C arrays?
I don't know why I didn't try this first. It seems that the D
equivalent of C's xs[1][0][2] is simply xs.ptr[[1][0][2].
Re: Access to structures defined in C
On Wednesday, 14 March 2018 at 02:17:57 UTC, Adam D. Ruppe wrote:
The type system would *like* to know, certainly for correct
range errors, but if you declare it as the wrong length and use
the .ptr, it still works like it does in C:
extern(C) __gshared extern char*[1] files; // still works
import core.stdc.stdio;
void main() {
printf("%s\n", files.ptr[2]); // ptr bypasses the range
check
}
That worked but now I have a more convoluted case: a C array of
pointers to int pointers, e.g.,
int **xs[] = {x1, x2, 0};
int *x1[] = {x1a, 0};
int *x2[] = {x2a, x2b, 0};
...
int x2a[] = { 1, 3, 5, 0};
Only the first line is exposed (and without the initialization).
So I tried:
extern(C) __gshared extern int**[1] xs;
The D compiler accepts that, but just about any manipulation gets
screamed at, usually with Error: only one index allowed to index
int. Note that I'm trying to access the ints, i.e., in C
something like xs[1][0][2] to access the 5 in x2a. Do I have to
mimic the intermediate C arrays?
But if you can match the length, that's ideal.
Unfortunately, although the C array lengths are known at C
compile time, they're not made available otherwise so I'm afraid
the [1] trick will have to do for now.
Re: unittests, dub and libraries
On Wednesday, 28 March 2018 at 03:07:23 UTC, Jonathan M Davis wrote: Run dub test The problem is that an executable needs a main, and a library doesn't have one, whereas when you're testing a library, you need an executable. So, a main must be inserted - e.g. with the -main flag to dmd. Just building the unittest build doesn't insert one. However, dub test _does_ deal with that for you. Thanks.
unittests, dub and libraries
I'm trying to build a very simple library. For now it just has a single class, constructor, destructor and one method. I added a unit test right after the method, declared the targetType to be "library" and a buildType of "unittest" (with options "unittests", "debugMode", "debugInfo"). When I run dub run -b unittest it builds the library, but then says: Target is a library. Skipping execution. If I compile with ldc2 -unittest the linker throws the error: (.text+0x20): undefined reference to `main' If I add an empty main function to the source file, ldc2/gcc manage to create an executable that can be invoked manually and it runs through the unit test. Is this the best that can be done?
Re: core.stdc.stdlib._compare_fp_t and qsort
On Sunday, 18 March 2018 at 19:01:11 UTC, Joe wrote: I managed to get it working by declaring a D dynamic array, appending n_recs pointers to it and using it as argument to sort. Unfortunately, I then had to copy from the dynamic array to the fixed array in order to continue using the latter. Any shortcuts around this? I just saw your other reply. Passing recs[0..n_recs] does the trick. Thanks.
Re: core.stdc.stdlib._compare_fp_t and qsort
On Sunday, 18 March 2018 at 18:11:02 UTC, Dmitry Olshansky wrote: Well since recs is array of pointers this looks like a null pointer in your data. The usual ways to fix that is either print stuff or poke around in debugger to see if a Record* is null or .name is null. The problem is that although the "recs" array is declared as having 10 pointers, not all pointers are used. In the qsort case, the second argument limited the sort to just the first n_recs (valid) pointers. There doesn't seem to be a way to tell std.algorithm.sorting.sort to only look at part of the fixed array. I managed to get it working by declaring a D dynamic array, appending n_recs pointers to it and using it as argument to sort. Unfortunately, I then had to copy from the dynamic array to the fixed array in order to continue using the latter. Any shortcuts around this?
Re: core.stdc.stdlib._compare_fp_t and qsort
On Sunday, 18 March 2018 at 13:10:08 UTC, Dmitry Olshansky wrote:
Do this to get the usual ptr + length:
sort!((a, b) => to!string((*a).name) <
to!string((*b).name))(recs[]);
Also to!string would be computed on each compare anew. May want
to use schwartzSort to avoid that, on 10 elements there is no
real difference though.
The 10 elements are just because it's a small test program.
What does changing "recs" to "recs[]" as the argument actually
do? Does it duplicate the fixed array on the fly? [I guess I
have to study more!]
The change does pass the compiler, but at runtime it causes a
segfault. The next to last frame in the backtrace shows (having
changed from to!string to fromStringz and using a string template
instead of a lambda):
#6 0x55565760 in
std.algorithm.sorting.sort!("fromStringz((*a).name.ptr) <
fromStringz((*b).name.ptr)", 0,
testd.Record*[]).sort(testd.Record*[]) (r=...)
Then it goes through quickSortImpl, shortSort and sort5, moving
on to either std.functional.binaryFun or processing of the
lambda, with a and b equal to 0, ending with a segfault in a ??
call from fromStringz or in memcpy called from object._dup (in
the to!string case).
Re: core.stdc.stdlib._compare_fp_t and qsort
On Monday, 12 March 2018 at 03:50:42 UTC, Joe wrote: On Monday, 12 March 2018 at 03:13:08 UTC, Seb wrote: Out of interest: I wonder what's your usecase for using qsort. Or in other words: why you can't use the high-level std.algorithm.sorting.sort? This is only temporary. I will be using std.algorithm.sorting.sort. I was converting a C program and it annoyed me that I couldn't get the qsort invocation past the D compiler. Now that I'm trying to use std.algorithm.sorting, I'm again puzzled by what I need to use for the "less" predicate. My first try was: sort!((a, b) => to!string((*a).name) < to!string((*b).name))(recs); This results in the error: Error: template std.algorithm.sorting.sort cannot deduce function from argument types !((a, b) => to!string((*a).name) < to!string((*b).name))(Record*[10]), candidates are: /usr/lib/ldc/x86_64-linux-gnu/include/d/std/algorithm/sorting.d(1851):std.algorithm.sorting.sort(alias less = "a < b", SwapStrategy ss = SwapStrategy.unstable, Range)(Range r) if ((ss == SwapStrategy.unstable && (hasSwappableElements!Range || hasAssignableElements!Range) || ss != SwapStrategy.unstable && hasAssignableElements!Range) && isRandomAccessRange!Range && hasSlicing!Range && hasLength!Range) which is not very helpful. It's a bit different from the "no function match" that Adam complains about, but I presume it's because now we're dealing with templates, and although the compiler finds a single candidate, it's not satisfactory but it can't tell the user anything further. I've tried using fromStringz((*x).name.ptr) instead of to!string (I'm still unclear to what extent can templates be used within templates). I also tried using an explicit cast(Record *)x because I'm also unsure that type information is passed down. Neither change helped.
Access to structures defined in C
What is the correct way to declare and access storage managed by
C?
For example, say a C module defines an array of filenames, e.g.,
char *files[] = { "one", "two", "three", 0};
The C header of course declares this as:
extern char *files[];
The way to declare it in a D module appears to be:
extern (C) {
__gshared extern char *[] files;
}
However, trying to index into this, e.g.,
char *filename = files[1];
does not work. In gdb if I try to examine 'filename' I see
something like a reverse text string but treated as an
(inaccessible) address and calling fromStringz causes a segfault.
Re: core.stdc.stdlib._compare_fp_t and qsort
On Monday, 12 March 2018 at 03:13:08 UTC, Seb wrote: Out of interest: I wonder what's your usecase for using qsort. Or in other words: why you can't use the high-level std.algorithm.sorting.sort? This is only temporary. I will be using std.algorithm.sorting.sort. I was converting a C program and it annoyed me that I couldn't get the qsort invocation past the D compiler.
Re: core.stdc.stdlib._compare_fp_t and qsort
On Monday, 12 March 2018 at 01:45:54 UTC, Adam D. Ruppe wrote:
I just reformatted it but now the difference should be visible:
`extern(C)` is missing on your callback.
The scope things might make a difference too, but I know for
sure extern(C) is necessary on your callback function.
I saw the extern(C) and I believe I tried it before my previous
post, but dismissed it because I saw no difference in compiler
behavior. In any case declaring
extern (C) int compar(const (void *) p1, const (void *) p2) {
... // as before
}
still gives the error:
function core.stdc.stdlib.qsort (scope void* base, ulong nmemb,
ulong size, extern (C) int function(scope const(void*), scope
const(void*)) @system compar) is not callable using argument
types (void*, ulong, ulong, extern (C) int function(const(void*)
p1, const(void*) p2))
It only went away with
extern (C) int compar(scope const (void *) p1, scope const (void
*) p2)
Could you explain or direct me to something that elucidates why
the "scope" qualifiers are needed? And is there something else
that is necessary inside compar() or are the "scope"s just for
decorative purposes?
Joe
Re: core.stdc.stdlib._compare_fp_t and qsort
On Sunday, 11 March 2018 at 23:26:04 UTC, Stefan Koch wrote: You have to pass a pointer to the function. Otherwise it'll be a parenthsis-less call. use : qsort(recs, num_recs, (Record *).sizeof, ); After passing a pointer, getting some other error messages, I changed the call to qsort(cast(void *)recs, num_recs, cast(size_t)(Record *).sizeof, ); and the latest error is: Error: function core.stdc.stdlib.qsort (scope void* base, ulong nmemb, ulong size, extern (C) int function(scope const(void*), scope const(void*)) @system compar) is not callable using argument types (void*, ulong, ulong, int function(const(void*) p1, const(void*) p2)) I fail to see which argument is causing the problem now. Joe
core.stdc.stdlib._compare_fp_t and qsort
I'm getting a compiler error in a qsort() call as follows:
qsort(recs, num_recs, (Record *).sizeof, compar);
Record is a struct, recs is a fixed array of pointers to Record's
and num_recs is a size_t that holds the number of valid records.
compar is this:
int compar(const void *p1, const void *p2)
{
import core.stdc.string : strcmp;
const Record **rp1 = cast(Record **)p1;
const Record **rp2 = cast(Record **)p2;
return strcmp((*rp1).name.ptr, (*rp2).name.ptr);
}
The error is: Error: function testd.compar (const(void*) p1,
const(void*) p2) is not callable using argument types ()
I don't quite understand what those parentheses mean: is it
implying "no arguments" and if so, where would one provide
arguments?
Joe
Re: Understanding the AST...
On Thursday, 22 February 2018 at 14:53:11 UTC, Seb wrote: On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote: Hello everybody! Last week end I found this post ( https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library/ ) on the Blog and thought to myself awesome. [...] BTW I know it's not as powerful as DMD (and not the real thing), but often the AST XML dump from DScanner helps to deepen the understanding: https://github.com/dlang-community/D-Scanner#ast-dump You can even play with libdparse on the web: https://run.dlang.io/is/qZsGDD Hello Seb, I had a look at the resources you provided and they are quite useful. Thank you. However, while technically a lexer would be enough to solve the problem at hand, I think I'm going to want that full front-end a bit later. The more information, the better. I rather have the option to ignore something I don't need than to need something I don't have :)
Re: Understanding the AST...
On Thursday, 22 February 2018 at 13:44:51 UTC, RazvanN wrote: On Thursday, 22 February 2018 at 13:21:04 UTC, joe wrote: [...] Indeed, @Stefan is right. The ParseTimeVisitor only contains information available at parse time. If you are interested in the parent you have 2 options: either (1) use the ParseTimeVisitor and implement the AST traversal logic yourself or (2) you can use the SemanticTimeTransitiveVisitor in which case the parent is not going to be null. In the case of (2) you need to also do some semantic analysis (so you need the whole dmd library, not just the parsing one). Here's an example on using the dmd library (including semantic) [1]. You can copy paste that example and add a few lines of code where you instantiate your visitor (which will inherit SemanticTimeTransitiveVisitor). [...] awesome, that helps a lot! Thanks both of you :)
Re: Negative index range violation
On Thursday, 22 February 2018 at 02:41:30 UTC, Steven Schveighoffer wrote: On 2/21/18 7:30 PM, SrMordred wrote: But with a slice negative indexes are never allowed, even on a pointer. youd have to do (c-1)[0 .. 1]; Nice! Thank you both! In D Slice article it says "You can even use negative indexes!" so I thought that the [-1..x] should work too :) Hah! I never thought of doing a slice with negative indexes ;) /SNIP -Steve At night I dream about doing something like this auto pos = haystack.find(needle); auto something = haystack[pos..+3]; // meaning [pos..pos+3] auto somethingElse = haystack[pos..-3]; // and [pos..pos-3] respectively :)
Re: Understanding the AST...
On Monday, 12 February 2018 at 08:47:58 UTC, RazvanN wrote:
Hi Joe,
/SNIP
On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote:
[...]
The FuncDeclaration node contains all the information for that.
For example, you can access fd.parent to see if the function is
declared at top-level (in which case, the parent is going to be
a module
declaration ) or if it is a nested function (in a class, in a
struct, in a function).
Every AST node contains information about the position in the
AST, all you
have to do is find how to get that information: which field to
access or which
member function to call.
/SNIP
Cheers,
RazvanN
Follow up question...
Why is *.parent always null?
e.g.:
extern(C++) class MyVisitor(AST): ParseTimeTransitiveVisitor!AST
{
override void visit(AST.Import i)
{
assert(i.parent is null); // always true
}
override void visitFuncBody(AST.FuncDeclaration f)
{
assert(f.parent is null); // always true
}
}
Re: Understanding the AST...
On Monday, 12 February 2018 at 08:47:58 UTC, RazvanN wrote: Hi Joe, I suggest you watch this video which explains how the parse time visitors work: https://www.youtube.com/watch?v=tK072jcoWv4 . On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote: [...] The FuncDeclaration node contains all the information for that. For example, you can access fd.parent to see if the function is declared at top-level (in which case, the parent is going to be a module declaration ) or if it is a nested function (in a class, in a struct, in a function). Every AST node contains information about the position in the AST, all you have to do is find how to get that information: which field to access or which member function to call. [...] The function average length visitor inherits a transitive visitor which means that the AST traversal logic is already implemented for you. All you have to do is override the visiting methods of interest and do whatever suits you : print stuff, alter the ast, stop the visitation or continue the visitation (by calling super.visit(ASTnode)). [...] I know that my explanations might not be very explicit, but if you have an example please post it and we can work on it. Cheers, RazvanN Hello RazvanN, thank you very much for taking the time to reply and also your effort in making this happen. I watched the video you linked and read your reply over and over, yet I still have a hard time to wrap my head around this idea. Like for example DHTML DOM is very easy for me to grasp. It's like riding the car down the country road and I know where I am and which town I'm going to be next, etc. This AST thing is more like a teleporter room on the Enterprise. Scotty activates the teleporter and a canister appears an a spot labeled imports. He repeats and a canister appears on a spot labeled functions, etc. I will try again...
Understanding the AST...
Hello everybody! Last week end I found this post ( https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library/ ) on the Blog and thought to myself awesome. So I built the library and everything went smooth. Thanks for the effort of all the involved people who made that possible! I've had a look at the 2 examples, too, the avg. function lines ( https://github.com/dlang/dmd/blob/master/src/examples/avg.d ) and the import ( https://github.com/dlang/dmd/blob/master/src/examples/impvisitor.d ) ones and for a start I decided to make a program that prints the outline of a module. Turns out I don't really understand how to access the data in the AST. For everything there's a visitor method and overriding a few of them to print return statements and some such works as advertised. However, I have no idea where I am in the tree when any of those methods are called. Like for example in FunctionLengthVisitor(AST).visitFuncBody(AST.FuncDeclaration fd). I have a function declaration object which tells me everything about what's inside the function, but how do I know what or where this function belongs to, where can I get that information ? I don't see anything about UDAs either, nor the doc comment. I understand when visitor.getAvgLen is called with the parsed module, the accept function calls a visitor overload for each member. But this sounds to me like I'd have to do a lot of book keeping in my visitor to keep track of things which are already present in the AST. Any insight to this would be much appreciated :)
Further questions on interfacing to Postgres libpq
An example test program that I'm using to learn D to C interfacing (specifically calling the libpq library) has a call to a C function declared as follows: void PQprint(FILE *fout, /* output stream */ const PGresult *res, const PQprintOpt *po); PQprintOpt is a struct whose first six members are declared as 'pqbool' which is in turn declared as "typedef char pqbool;" in the distributed Postgres header file. I've defined an "alias pqbool = char;" in the D file, which is pretty straightforward. The second of the six members has the name "align", which is a D keyword. So I renamed it "align_" and I presume that won't cause any problems. To deal with the first argument to PQprint, I added "import core.stdc.stdio : FILE;". The question is how to pass the D "stdout" as that argument. The D compiler tells me I can't pass it as is (as was done in C), because in D it's of type "File".
Re: Using Postgres connection functions
On Saturday, 20 January 2018 at 04:54:47 UTC, Adam D. Ruppe wrote: Same as above. The general pattern is: C_Type[] name = new C_Type[](requested_size); // pass as `name.ptr`. This becomes a C_Type* Thanks, Adam. Perhaps something like this ought to make its way into the "D for C Programmers" page.
Re: Using Postgres connection functions
On Saturday, 13 January 2018 at 05:28:17 UTC, Joe wrote:
Going beyond the connection, there are various other libpq
functions that use a similar pattern of values passed using
multiple parallel C arrays, e.g.,
PGresult *PQexecParams(PGconn *conn,
const char *command,
int nParams,
const Oid *paramTypes,
const char * const *paramValues,
const int *paramLengths,
const int *paramFormats,
int resultFormat);
Each of the `paramXxxxs' arguments are arrays (Oid is an alias
for uint).
[...]
Focusing on the above function, suppose the first two parameter
arrays are defined in a C program as:
Oid paramTypes[] = {23, 25};
char *paramValues[] = {"1", "abcde"};
which could be expressed in D as:
Oid [] paramTypes = [23, 25];
string [] paramValues = ["1", "abcde"];
I know the paramTypes could be passed as null, letting PG deduce
the data types but suppose we have some reason to accumulate
types, etc., in D slices. I know the paramValues can be
manipulated in a manner similar to the one shown in my first
post, namely something like this:
extern(C) char * [2] pvs;
foreach (i, val; paramValues)
pvs[i] = cast(char *)toStringz(val);
And then use "cast(const char **)pvs" for the paramValues
argument. The only feedback that I received that was explicit to
this approach was in response to my initial post, in which Adam
D. Ruppe said that what I did was "decent".
So I have two lingering doubts:
1. Is malloc() the only way to allocate the arrays, either of
Oid's, int's or char *'s, for passing to the libpq functions?
IOW, isn't there a way to take advantage of D's 'new' (and thus
the GC)?
2. How to convert a slice of Oid's or int's to an array of
pointers suitable by processing by the libpq C function? A
technique similar to the previous one seems to work, e.g.,
extern(C) Oid [2] pts;
foreach (i, typ; paramTypes)
pts[i] = typ;
But I'm not sure if it's really working (when I mistakenly had a
* in the pts declaration, at one point it also seemed to work).
Re: Using Postgres connection functions
On Monday, 15 January 2018 at 02:28:29 UTC, Matthias Klumpp wrote: In any case, please don't start another Postgres library and consider contributing to one of the existing ones, so that we maybe have one really awesome, 100% complete library at some point. If, on the other hand, your goal is to learn about the low-level Postgres interface and not just to have a Postgres interface for an application you develop, by all means, play with it :-) At this point, I am indeed learning about low-level Postgres interfaces (but not so low-level as the client-server protocol) as a way to understand the challenges of interfacing D to C. However, as part of the Pyrseas open source project, which I maintain, I had started to create a Postgres interface in Python inspired by The Third Manifesto, as opposed to ORMs like SQLAlchemy (see https://pyrseas.wordpress.com/2013/03/07/a-pythonic-ttm-inspired-interface-to-postgresql-requirements/). I got criticized for "reinventing the wheel" but I believe TTM, if properly done, is quite different from an ORM. I understand your concern about not starting another PG library. From the cursory investigation of the existing libraries, I think they span a spectrum, with ddb at one end (low-level protocol), then derelict-pq (low-level binding over libpq), ddbc at the opposite end (multi-DBMS support) and several others in between. So I guess the real problem is with the proliferation in the latter group.
Re: Using Postgres connection functions
On Saturday, 13 January 2018 at 10:10:41 UTC, Jacob Carlborg wrote: There's a native D library, ddb [1], for connecting to Postgres. Then you don't have to worry about null-terminated strings. There are several D libraries that I would consider "native": derelict-pq, dpq, dpq2 and ddb. The latter perhaps has the distinction that it doesn't use libpq, but rather implements the Postgres FE/BE protocol. That's a bit *too* native for my taste. It means the library maintainer has to keep up with changes to the internal protocol, which although published, the Postgres group doesn't have to maintain compatibility from version to version. For example, they haven't dropped the PQsetdbLogin function even though the PQconnectdb and PQconnectdbParams functions are obviously preferred. OTOH, there used to be an AsciiRow message format in the protocol, that was dropped, unceremoniously (not even mentioned in the release notes).
Re: Using Postgres connection functions
Going beyond the connection, there are various other libpq functions that use a similar pattern of values passed using multiple parallel C arrays, e.g., PGresult *PQexecParams(PGconn *conn, const char *command, int nParams, const Oid *paramTypes, const char * const *paramValues, const int *paramLengths, const int *paramFormats, int resultFormat); Each of the `paramXxxxs' arguments are arrays (Oid is an alias for uint). PGresult *PQprepare(PGconn *conn, const char *stmtName, const char *query, int nParams, const Oid *paramTypes); PGresult *PQexecPrepared(PGconn *conn, const char *stmtName, int nParams, const char * const *paramValues, const int *paramLengths, const int *paramFormats, int resultFormat); My point is that there seems to be a need to have a generic or generalized mechanism for passing these argument arrays from D to C.
Re: Using Postgres connection functions
On Saturday, 13 January 2018 at 04:26:06 UTC, Adam D. Ruppe wrote: If and only if the values are known at compile time, you can do: const char** keywords = ["hostaddr".ptr, "port".ptr, "dbname".ptr, null].ptr; or even do it inline: PQconnectdbParams(["hostaddr".ptr, "port".ptr, "dbname".ptr, null].ptr, ditto_for_values, 1); The keywords are (or could be) known at compile time, but almost by definition, the associated values are only known at runtime.
Using Postgres connection functions
I'm trying to learn how to use D to connect (and send queries) to
Postgres, i.e., libpq in C. Postgres has three families of
connection functions: PQsetdbLogin which takes multiple
individual arguments (all as const char *), PQconnectdb which
takes a single connection string (which Postgres then has to
parse into keywords/values) and PQconnectdbParams (introduced
with PG 9.0 in 2010) which takes two arrays of char pointers, one
for the keywords and another for the values, i.e., it's
pre-parsed, and I believe, more convenient for callers since it's
most likely that they get the parameter values from command line
arguments, environment values or UI dialogs, so it saves the
connection string formatting on the caller side and the
parsing/decomposing on the library side.
In spite of the latter considerations, it appears most D
libraries for Postgres only support connection string usage (only
derelict-pq has a declaration for PQconnectdbParams--but I
haven't tested it).
In any case, I tried to use PQconnectdbParams by first declaring
it, as per the Postgres manual, with
extern(C)
{
struct PGconn;
PGconn *PQconnectdbParams(const char * const * keywords,
const char * const * values,
int expand_dbname);
}
This caused ldc2, on Linux, to complain as follows:
connection.d(30): Error: found 'const' when expecting ')'
connection.d(30): Error: semicolon expected following function
declaration
connection.d(30): Error: declaration expected, not '*'
It only compiled after I removed the second 'const' in the first
and second arguments.
The hard thing was figuring out how to pass the keyword/values
arrays, defined as:
string[] keywords = ["hostaddr", "port", "dbname"];
string[] values = ["127.0.0.1", "5432", "testdb"];
to the D invocation of PQconnectdbParams. I ended up with the
following, which looks like covering a patient with lots of
bandaids and looking the other way (and I had to modify the
arrays above with a fixed length).
extern(C) char * [keywords.length + 1] kws;
extern(C) char * [keywords.length + 1] vals;
foreach (i, k; keywords) {
kws[i] = cast(char *)toStringz(k);
vals[i] = cast(char *)toStringz(values[i]);
}
kws[keywords.length] = null; // Postgres wants the NULL
termination
vals[keywords.length] = null;
conn = PQconnectdbParams(cast(const char **)kws,
cast(const char **)vals, 1);
I assume that in a real program I'd have to malloc the C arrays,
since I wouldn't know beforehand how many parameters there would
be (or I could define them with a static size of about 30 since
that's how many connection parameters are recognized currently).
So my question is: is there an easier or better way of passing
two arrays of C null-terminated strings to an extern(C) function?
