Re: Non-blocking keyboard input
On Wednesday, 27 December 2023 at 13:27:53 UTC, Adam D Ruppe wrote: On Wednesday, 27 December 2023 at 05:07:04 UTC, Joe wrote: ??? Surely there there is a one liner library solution for this? It is not one line because it needs a bit of setup (and teardown, but the objects' destructors do that for you) but it is close: http://arsd-official.dpldocs.info/arsd.terminal.html#single-key `input.getch` waits for a single line, but you can use `if(input.kbhit())` to see if it would block before calling it. This shouldn't be hard... yet it is. Better be careful, the mods are out in force deleting posts this week that tell the hard truth. But yeah, the stdlib in D has very little activity: https://github.com/dlang/phobos/graphs/contributors? So you can't expect much from it. My arsd libs provide a broad set of functionality missing from it: stuff like this terminal/console stuff, window creation, basic guis, web servers, etc. If you want to try them, you can use it from the dub system, but I recommend just `git clone https://github.com/adamdruppe/arsd.git` in your working directory then import what you want and use `dmd -i` to automatically include them in the build. This does not actually work on my computer. It still blocks. int itr = 0; for(;;) { itr++; writeln("1. closeKBThread = ", closeKBThread, ", iter = ", itr); if (closeKBThread) return; string op = ""; string istr = ""; while (!closeKBThread) { writeln("2. ", input.kbhit(), " ", closeKBThread); if (input.kbhit()) { istr ~= input.getch(false); break; } } writeln("final istr = ", istr); 1. closeKBThread = false, iter = 1 2. false false 2. false false 2. false false 2. false false final istr = f 1. closeKBThread = false, iter = 2 2. false false < now blocking All this is just junk of me trying to figure out what was going on, but literally input.kbhit blocks after the first run(which really means it's always blocking) My original code was while(true) { if (input.kbhit()) { istr ~= input.getch(); break; } } and I've been trying all kinds of stuff to figure out what was going on but I believe it is the kbhit function itself. It calls getch(true) and blocks on that as if I were just using getch itself. The issue is the same, the code does not block then after I hit a key and it goes through an iteration of the outside loop it then blocks waiting for the next input.
Non-blocking keyboard input
??? Surely there there is a one liner library solution for this? I have a program that spawns a thread for debugging information and uses the keyboard input which allows me to display the information. If I use getchar or readline then it blocks the thread. This is generally fine because that is all the thread does. The problem is that it also blocks the program termination as the main thread will not exit while the thread is running which is is because it's waiting on keyboard input stuck on getchar or fgetc or whatever. If I terminate the threads using thread_term then it terminates the program but the program then does not return the return code that it was successfully finished(because it was prematurely terminated by thread_term. Surely there is some type of peek for keyboards in D? I can't seem to get kbhit or because the library is not included(well, I tried to use one from dmc but it seemed to be invalid. snn.lib IIRC). This shouldn't be hard... yet it is.
D atomics
How does one really use atomics in D? shared string[int] AA; AA[s[2]].atomicStore(s[1]); causes a hangup. If I just assign it "works": AA[s[2]] = s[1]; I've had similar issues where: atomicOp!"+="(X[Y],1); fails and I have to use X[Y].atomicStore(1); or atomicOp!"+="(count,1); // fails (does not increment count) so I have to create a new variable int rcount = atomicFetchAdd(count,1); and this will increment count. In fact, I don't know what works or what doesn't except by trial and error. I don't even know how to go about thinking about these issues. In my mind, when using multiple threads and shared variables I try to use atomics to avoid any race conditions. Usually this involves incrementing or storing values. I try to use atomics that replace the single threaded concept but they do not always seem to work. Are there any known issues or is it just me? Should it just work? is there any good reference that gets ones mind right for multithreading? I have some idea but I rarely do multithreading until lately(I guess because I now have the HP to care). I think the main issue I tend to have is trying to convert single threaded code in to multiple threads and missing certain key issues. E.g., I was doing a parallel for and it wasn't really working(it seems the task pool stalls the entire pool until all tasks are complete): for(s; parallel(arr)) { // do stuff with s. } So I converted to using threads: for(s; arr) { new Thread({ // do stuff with s. }).start(); } // slightly more complex as I limited the total number of threads But as I spun up threads I would get different threads working on the same s. I imagined that this likely had something to do with the variable being on the stack being reused. so I wrapped the thread in a function: for(ss; arr) { void foo(typeof(ss) s) { new Thread({ // do stuff with s. }).start(); } foo(ss); } and this seemed to solve the problem. I imagine this is because ss is duped by the function call essentially detaching it from the loop code so it is unique across the thread. But I guess I need to learn exactly how to deal with all these problems rather than just hacking away.
Re: range shortened method not enabled, compile with compiler switch `-preview=shortenedMethods`
On Sunday, 24 September 2023 at 12:59:57 UTC, Steven Schveighoffer wrote: On Sunday, 24 September 2023 at 10:00:31 UTC, Joe wrote: For absolutely no reason I started getting this error. Last night I compiled the project and it worked just fine. This morning I made a single insignificant change and tried to compile and got that error. Only possible thing is that for some reason some change in updating the compiler may have no propagated correctly till after the reboot. [...] This sounds like a compiler issue. Are you on the latest compiler? Which compiler? Is there a set of code you can post or upload which causes the problem? -Steve I accidentally deleted my D folder containing the compiler and other things. (It as a weird issue where I was trying to add a command to an app and because the %1 parameter didn't have quotes I guess it deleted the D folder as it shortened it to D) I had a backup and didn't change anything since then except for updating the compilers. Still had the same visual D version. So I copied it over and updated dmd with visual D. Everything worked fine that day and I messed with the code several times. Woke up and wanted to make a simple change and code would not compile. The changes were insignificant. It gave me that error and when I added that switch then I got one about pragma(crt_constuctor) needing extern(C). Anyways, tried adding the extern(C) in the code but it wouldn't accept the changes. BUT I just tried to compile it again and no issues. Probably drive corruption issues or possibly some type of caching issue. But I was able to make changes now and compile it like it should work. E.g., it's possible that when I copied over the old D directory that it had an old compiler(I think it was from '22). I did update with visual D but maybe there was caching going on that was causing the conflict. In any case everything seems to be back to normal. I really didn't do much to fix anything so it was probably all caused by deleting that D folder accidentally.
Re: Vibe.d download function, how to get callback when done or error?
On Saturday, 23 September 2023 at 20:20:31 UTC, Christian Köstlin wrote: On 23.09.23 14:07, j...@bloow.edu wrote: I'm using download(url, filename) to download files in vibe.d. The issue is that I do not know when the download is finished or errors. There is a callback for the streaming side but not for the file download. A small test program shows, that if the function return normally the transfer was done (and the file saved). The function raises an exception if there is e,g, an http error status communicated. I am not sure what happens if the download is interrupted in the middle. I guess there will be an exception, but the file might be written partially. Kind regards, Christian I can't do any testing because overnight for no explicable reason the app will not compile any more. If it is synchronous then I should be able to do what I need without issue. When I initially was using it and the way the app works it seemed like it was running in parallel because I didn't see it execute synchronously because the tasks were relatively short and I just thought it was asynchronous for some reason(I didn't give it any thought at the time until I got errors in the download and needed to make the code more robust but I can't do anything until I figure out why the code no longer compiles(I made a post about it but it's not showing up yet).
Re: Vibe.d download function, how to get callback when done or error?
On Saturday, 23 September 2023 at 15:09:13 UTC, Elias wrote: On Saturday, 23 September 2023 at 12:07:38 UTC, Joe wrote: I'm using download(url, filename) to download files in vibe.d. The issue is that I do not know when the download is finished or errors. There is a callback for the streaming side but not for the file download. You don’t need a callback for that. It’s done when the download() returns. Pretty sure it is not synchronous. I'm running it many times. Maybe it is synchronous and I just didn't notice.
range shortened method not enabled, compile with compiler switch `-preview=shortenedMethods`
For absolutely no reason I started getting this error. Last night I compiled the project and it worked just fine. This morning I made a single insignificant change and tried to compile and got that error. Only possible thing is that for some reason some change in updating the compiler may have no propagated correctly till after the reboot. when I add the switch I then get. function `core.time.TickDuration.time_initializer` must be `extern(C)` for `pragma(crt_constructor)` dmd-2.105.2 EVERYTHING was working fine yesterday. All I did today was load up the project and change an integer value and I couldn't compile it ;/ Maybe corruption somewhere, I don't know. Anyways, I added extern(C) to that method in the location it says the file exists but no change. I removed the pragma, no change. Doesn't seem to actually be using that file. I have no idea what is going on. All I can do is try to reinstall the compiler and maybe redo the dub package. I'm using visual D + vibe.d + dub. Nothing fancy.
Vibe.d download function, how to get callback when done or error?
I'm using download(url, filename) to download files in vibe.d. The issue is that I do not know when the download is finished or errors. There is a callback for the streaming side but not for the file download.
Re: Help on array pointers
On Monday, 18 September 2023 at 02:49:37 UTC, vino wrote: On Sunday, 17 September 2023 at 18:28:36 UTC, Joe wrote: On Friday, 15 September 2023 at 16:55:34 UTC, Vino wrote: [...] [...] char[] invalid = (cast(char*)malloc(char.sizeof * len))[0..len]; This is not the way to go about it. You are mixing "pointer arrays" with "arrays". [...] Thank you very much, I am still newbie for programming and currently concentrating on Arrays/Struct/Pointers/Memory management. It will make more sense over time. It's actually really simple but it takes time to get used to thinking along the lines of the computer languages and such. My suggestion is simply do a lot of examples. Just type them in. As you type you think about what you are typing and how it relates. Then you will start to notice the patterns and such. I wouldn't try to create new programs from scratch because it's easy to make mistakes that can be very hard to debug. D might not be the best language to start learning programming. E.g., might be better to go with javascript, lua, or python and get a nice rapid development environment. Even though D is fast to compile it's not that fast. Maybe use Repl which lets you code line by line essentially. All languages more or less are the same but just look different so it you want to find out one that has the most convenience. you don't want to bite off too much as it will slow you down. Ultimately it's just like anything else though in that you are learning a language and all it's nuances. You ultimately have to learn how to translate your thoughts and desires in to the correct syntax so you can make the language useful and act as a tool. If you try to us it as a tool before you learn to mimic others it will be very slow and frustrating. Because most languages are 99% the same with just different syntax(just about every major programming language can do what ever other one can, it's just some are better/easier at doing certain things than others). You might try learning several languages at once as it can actually speed things up. Learn how to write simple programs(hello worlds) then modify those to do more and then learn control structures(branching(if's), loops(for)), variables, etc. Build up slowly and everything will make sense. Build up fast and you'll get lost at some point and then spend a lot of time trying to figure out how to make sense of it. Ideally you want an ide and language that gives you precise info on errors that you make else it can be hard to track down mistakes because sometimes errors can be hidden. D is not good for. Even Q-basis might be better to start with or try to find Turbo Pascal if it still exists. What makes D great is it's generic/template programming and that is a more complex topic that you probably won't get much in to for a few years(although it is not really hard it will seem like it at first). D has a lot of features that hard core programmers like and so you are not going to be using those features for a while so it might not be worth using D to learn off of. Not saying you can't do it but it might be slow. You could always do it in parallel. E.g., learn to write simple programs in many languages then build up. E.g., for i=1:10 // Matlab for(auto i = 1; i < 10; i++) // D for(int i = 1; i < 10; i++) // C for(var i = 1; i < 10; i++) // JS for i from 1 to 10 // maple loop(50, ...) // JSFX for i = 1,10 // lua etc. The underlying idea is that one wants a way to do the same thing over and over and so loops exist to handle that pattern. Each language uses different ways to express a loop. Just like languages, all languages uses nouns, verbs, etc but express them differently.
Re: Help on array pointers
On Friday, 15 September 2023 at 16:55:34 UTC, Vino wrote: On Friday, 15 September 2023 at 15:27:00 UTC, Vino wrote: On Friday, 15 September 2023 at 02:25:09 UTC, Joe wrote: On Thursday, 14 September 2023 at 14:21:09 UTC, Vino wrote: [...] A pointer is a type that points to something. It's literally that simple. Every piece of data and code exist somewhere in memory. Every piece of memory has an address. The address is what a pointer contains. Sometimes we want a type that is the address itself rather than a value/number. [...] Hi Joe, Thank you very much for the explanation can you please correct me if my understanding is incorrect ``` byte[] z; // Creates an array of bytes. That is, the compiler will create a pointer to an array of memory and track it's length and deal with memory allocation and all that. ``` If we use the above method then : The compiler takes care of initilizing the array and free the memory after the usage. And this is the recommended method. ``` We can use a pointer as an array also, this is the "old school way of creating arrays". int qlen = 5; int* q = cast(int*)malloc(int.sizeof*qlen); ``` If we use the above method then : We need to manual initilize the array. Ensure that the memory is freed after the usage using try/final block. By default the memory allocation for arrays in D is based on GC (expect for std.array containers) if we want to reduce/avoid GC then we need to use the old school way of creating the arrays. In case of using the old school ways then can you guide me what is wrong in my earlier code that I am getting the below error and how do I correct the same. Error ``` Invalid Name passed: /T double free or corruption (out) Error: program killed by signal 6 ``` Hi All, At last was able to resolve the issue, but not sure whether this is the reight solution. Code: ``` import std.stdio: writeln; import std.exception: enforce; import std.range: empty; import std.format: format; auto ref testNames(in string[] names) { enforce(!empty(names), "Names cannot be Empty or Null"); import core.stdc.stdlib; import std.algorithm: any, canFind; string[] _names; size_t len = 19; char[] invalid = (cast(char*)malloc(char.sizeof * len))[0..len]; invalid[0 ..len] = 0; //Array Initlization try { version(Posix) { invalid = ['\'','\"',':',';','*','&','/','[',']','-','+','$','#','<','>','{','}','(',')']; } foreach(i; names.dup) { auto result = i.any!(a => invalid.canFind(a)); if(result) { throw new Exception("Invalid Name passed: %s".format(i)); } else {_names = names.dup; return _names; } } } catch(Exception e) { writeln(e.msg); } finally { invalid = null; free(invalid.ptr); } // added invalid = null resolved the issue return _names; } void main () { writeln(testNames(["/T"])); } From, Vino char[] invalid = (cast(char*)malloc(char.sizeof * len))[0..len]; This is not the way to go about it. You are mixing "pointer arrays" with "arrays". You are creating a pointer array then turning that in to an array. There is no need to do that. Basically you are copying what the compiler already does for you. When you are using arrays([] language) you don't have to worry about anything. Just use them as arrays directly and let the compiler deal with memory management. The entire point of "managed arrays" is to avoid having to manually deal with memory which can cause problems if one is not careful. Of course you have to make sure your array used used correctly but that should be obvious. char[] invalid = (cast(char*)malloc(char.sizeof * len))[0..len]; is exactly the same as char[] invalid. EXCEPT that you've forced it to be initialized with a length of len and went from D in to C then back to D just to create an array. You are making it much more complex than it needs to be. There are good tutorials on D arrays that you should look over. You will see how easy they are. You could just do char[] invalid = new char[](len); for the same effect but is nicer. New understands char, malloc only understands bytes. You should not use malloc in D unless you explicitly know you need to use it. Malloc is primitive memory management. When you use D's symantics for arrays it is much easier... which is the entire point. You don't even need to free the arrays. D knows they are local variable sand will free them when out of scope. It will grow them when they need to grow, etc. invalid = null; free(invalid.ptr); That does nothing. you set invalid.ptr to null and are freeing null so you are not freeing the array. You don't have to though, the array will be free out of scope(not in final though so exceptions could cause memory leak) because it is a D array. Do not
Re: Help on array pointers
On Thursday, 14 September 2023 at 14:21:09 UTC, Vino wrote: Hi All, Request your help to guide me in understanding about pointers, the below code works,I have few question which i need your help for better understanding. Questions:1 ``` char[] invalid = (cast(char*)malloc(char.sizeof * length))[0..length]; ``` The above statement allocate memory for char type and the size of the allocated memory is char.sizeof * length so what is the use of this "[0..length]"; Question:2 ``` char[]* invalidptr = ``` Is this the right way to create a array pointer. Question: 3 ``` ST1: char[] invalid = (cast(char*)malloc(char.sizeof * length))[0..length]; ST2: char[]* invalid = (cast(char*)malloc(char.sizeof * length))[0..length]; ``` What is the difference between the above to statement. Question: 4 Who do we free the memory allocated. Code: ``` auto ref testNames(in string[] names) { enforce(!empty(names), "Names cannot be Empty or Null"); import core.stdc.stdlib; import std.algorithm: any, canFind; size_t length = 20; char[] invalid = (cast(char*)malloc(char.sizeof * length))[0..length]; char[]* invalidptr = version(Windows) { (*invalidptr) = ['\'','\"',':',';','*','&','[',']','-','+','$','#','<','>','{','}','(',')']; } foreach(i; names.dup) { auto result = i.any!(a => (*invalidptr).canFind(a)); if(result) { throw new Exception("Invalid Name passed: %s".format(i)); } } string[] _names = names.dup; return _names; } ``` From, Vino A pointer is a type that points to something. It's literally that simple. Every piece of data and code exist somewhere in memory. Every piece of memory has an address. The address is what a pointer contains. Sometimes we want a type that is the address itself rather than a value/number. https://run.dlang.io/gist/19c63d325ee412df23bdbefabce111b9 import std, std.stdio, core.stdc.stdlib; void main() { float x = 43.534f; // Creates a float type that hold sthe value 43.534 writeln("Value of x = ", x); // Displays it's value writeln("Address of x = ", ); // Displays it's address writeln(); float* y = null; // creates a pointer of type float note that it is point to no memory location(null) and so it is an error to store a value to the location writeln("Value of y = ", y); // Displays it's address(remember, y is a ptr to a float, not a float) y = new float(); // This allocates free/unused memory(of float type) to y so we can store a value. *y = 50.34f; // assigns it the value 50.34. We have to use * because if we didn't we would be trying to change the address of y writeln("Value of y = ", y); // Displays it's address(remember, y is a ptr to a float, not a float) writeln("Address of y = ", ); // Writes it's address(y is a pointer but it also is stored in a location in memory and hence has an address writeln("Dereferenced value of y = ", *y); // Displays the value y is pointing to interpreting it as a float(the base type of the pointer) writeln("Dereferences the value of y as if it were an int = ", *(cast(int*)y)); // Displays the value y is pointing to interpreting it as a int. We have to force the compilier to reinterpret it as an int*. writeln(); byte[] z; // Creates an array of bytes. That is, the compiler will create a pointer to an array of memory and track it's length and deal with memory allocation and all that. writeln("Value of z = ", z); // Displays it's address(remember, z is a ptr to a float, not a float) writeln("Address of z = ", ); // Note that z is an array but it is a pointer to and gets the address where the "array" is stored. writeln("Value of z's pointer = ", z.ptr); // Displays it's address(remember, y is a ptr to a float, not a float) writeln("Length of z = ", z.length); // Writes it's address(y is a pointer but it also is stored in a location in memory and hence has an address writeln(); z ~= 4; // We can store/append a value to our array. The compiler will take care of dealing with allocating memory and all that. writeln("Value of z = ", z); // Displays it's address(remember, z is a ptr to a float, not a float) writeln("Address of z = ", ); // Note that z is an array but it is a pointer to and gets the address where the "array" is stored. writeln("Value of z's pointer = ", z.ptr); // Displays it's address(remember, y is a ptr to a float, not a float) writeln("Length of z = ", z.length); // Writes it's address(y is a pointer but it also is stored in a location in memory and hence has an address z ~= 54; // We can store/append a value to our array. The compiler will take
Re: parallel threads stalls until all thread batches are finished.
On Monday, 28 August 2023 at 22:43:56 UTC, Ali Çehreli wrote: On 8/28/23 15:37, j...@bloow.edu wrote: > Basically everything is hard coded to use totalCPU's parallel() is a function that dispatches to a default TaskPool object, which uses totalCPUs. It's convenient but as you say, not all problems should use it. In such cases, you would create your own TaskPool object and call .parallel on it: https://youtu.be/dRORNQIB2wA?t=1611s Ali Thanks. Seems to work. Didn't realize it was that easy ;)
Re: parallel threads stalls until all thread batches are finished.
On Monday, 28 August 2023 at 10:33:15 UTC, Christian Köstlin wrote: On 26.08.23 05:39, j...@bloow.edu wrote: On Friday, 25 August 2023 at 21:31:37 UTC, Ali Çehreli wrote: On 8/25/23 14:27, j...@bloow.edu wrote: > "A work unit is a set of consecutive elements of range to be processed > by a worker thread between communication with any other thread. The > number of elements processed per work unit is controlled by the > workUnitSize parameter. " > > So the question is how to rebalance these work units? Ok, your question brings me back from summer hibernation. :) This is what I do: - Sort the tasks in decreasing time order; the ones that will take the most time should go first. - Use a work unit size of 1. The longest running task will start first. You can't get better than that. When I print some progress reporting, I see that most of the time N-1 tasks have finished and we are waiting for that one longest running task. Ali "back to sleep" I do not know the amount of time they will run. They are files that are being downloaded and I neither know the file size nor the download rate(in fact, the actual download happens externally). While I could use work unit of size 1 then problem then is I would be downloading N files at once and that will cause other problems if N is large(and sometimes it is). There should be a "work unit size" and a "max simultaneous workers". Then I could set the work unit size to 1 and say the max simultaneous workers to 8 to get 8 simultaneous downloads without stalling. I think thats what is implemented atm ... `taskPool` creates a `TaskPool` of size `defaultPoolThreads` (defaulting to totalCPUs - 1). The work unit size is only there to optimize for small workloads where task / thread switching would be a big performance problem (I guess). So in your case a work unit size of 1 should be good. Did you try this already? Kind regards, Christian Well, I have 32 cores so that would spawn 64-1 threads with hyper threading so not really a solution as it is too many simultaneous downs IMO. "These properties get and set the number of worker threads in the TaskPool instance returned by taskPool. The default value is totalCPUs - 1. Calling the setter after the first call to taskPool does not changes number of worker threads in the instance returned by taskPool. " I guess I could try to see if I can change this but I don't know what the "first call" is(and I'm using parallel to create it). Seems that the code should simply be made more robust. Probably a just a few lines of code to change/add at most. Maybe the constructor and parallel should take an argument to set the "totalCPUs" which defaults to getting the total number rather than it being hard coded. I currently don't need or have 32+ downlaods to test ATM so... this() @trusted { this(totalCPUs - 1); } /** Allows for custom number of worker threads. */ this(size_t nWorkers) @trusted { Basically everything is hard coded to use totalCPU's and that is the ultimate problem. Not all tasks should use all CPU's. What happens when we get 128 cores? or even 32k at some point? It shouldn't be a hard coded value, it's really that simple and where the problem originates because someone didn't think ahead.
Re: parallel threads stalls until all thread batches are finished.
On Friday, 25 August 2023 at 21:43:26 UTC, Adam D Ruppe wrote: On Wednesday, 23 August 2023 at 13:03:36 UTC, Joe wrote: to download files from the internet. Are they particularly big files? You might consider using one of the other libs that does it all in one thread. (i ask about size cuz mine ive never tested doing big files at once, i usually use it for smaller things, but i think it can do it) The reason why this causes me problems is that the downloaded files, which are cashed to a temporary file, stick around and do not free up space(think of it just as using memory) and this can cause some problems some of the time. this is why im a lil worried about my thing, like do they have to be temporary files or can it be memory that is recycled? The downloading is simply a wrapper that provides some caching to a ram drive and management of other things and doesn't have any clue how or what is being downloaded. It passes a link to something like youtube-dl or yt-dlp and has it do the downloaded. Everything works great except for the bottle neck when things are not balancing out. It's not a huge deal since it does work and, for the most part, gets everything downloaded but sorta defeats the purpose of having multiple downloads(which is much faster since each download seems to be throttled). Increasing the work unit size will make the problem worse while reducing it to 1 will flood the downloads(e.g., having 200 or even 2000 downloads at once). Ultimately this seems like a design flaw in ThreadPool which should auto rebalance the threads and not treat the number of threads as identical to the worker unit size(well, length/workerunitsize). e.g., suppose we have 1000 tasks and set worker unit size to 100. This gives 10 workers and 10 workers will be spawned(not sure if this is limited to total number of cpu threads or not) What would be nice is to be able to set worker unit size to 1 and this gives 1000 workers but limit concurent workers to, say 10. So we would have at any time 10 workers each working on 1 element. When one gets finished it can be repurposed for any unfinished tasks. The second case is preferable since there should be no issues with balancing but one still gets 10 workers. The stalling comes from the algorithm design and not anything innate in the problem or workload itself.
Re: parallel threads stalls until all thread batches are finished.
On Friday, 25 August 2023 at 21:31:37 UTC, Ali Çehreli wrote: On 8/25/23 14:27, j...@bloow.edu wrote: > "A work unit is a set of consecutive elements of range to be processed > by a worker thread between communication with any other thread. The > number of elements processed per work unit is controlled by the > workUnitSize parameter. " > > So the question is how to rebalance these work units? Ok, your question brings me back from summer hibernation. :) This is what I do: - Sort the tasks in decreasing time order; the ones that will take the most time should go first. - Use a work unit size of 1. The longest running task will start first. You can't get better than that. When I print some progress reporting, I see that most of the time N-1 tasks have finished and we are waiting for that one longest running task. Ali "back to sleep" I do not know the amount of time they will run. They are files that are being downloaded and I neither know the file size nor the download rate(in fact, the actual download happens externally). While I could use work unit of size 1 then problem then is I would be downloading N files at once and that will cause other problems if N is large(and sometimes it is). There should be a "work unit size" and a "max simultaneous workers". Then I could set the work unit size to 1 and say the max simultaneous workers to 8 to get 8 simultaneous downloads without stalling.
Re: parallel threads stalls until all thread batches are finished.
On Wednesday, 23 August 2023 at 14:43:33 UTC, Sergey wrote: On Wednesday, 23 August 2023 at 13:03:36 UTC, Joe wrote: I use foreach(s; taskPool.parallel(files, numParallel)) { L(s); } // L(s) represents the work to be done. If you make for example that L function return “ok” in case file successfully downloaded, you can try to use TaskPool.amap. The other option - use std.concurrency probably. I think I might know what is going on but not sure: The tasks are split up in batches and each batch gets a thread. What happens then is some long task will block it's entire batch and there will be no re-balancing of the batches. "A work unit is a set of consecutive elements of range to be processed by a worker thread between communication with any other thread. The number of elements processed per work unit is controlled by the workUnitSize parameter. " So the question is how to rebalance these work units? E.g., when a worker thread is done with it's batch it should look to help finish that batch rather than terminating and leaving all the work for the last thread. This seems like a flaw in the design. E.g., if one happens to have n batches and every batch but one has tasks that finish instantly then essentially one has no parallelization.
Re: parallel threads stalls until all thread batches are finished.
On Wednesday, 23 August 2023 at 14:43:33 UTC, Sergey wrote: On Wednesday, 23 August 2023 at 13:03:36 UTC, Joe wrote: I use foreach(s; taskPool.parallel(files, numParallel)) { L(s); } // L(s) represents the work to be done. If you make for example that L function return “ok” in case file successfully downloaded, you can try to use TaskPool.amap. The other option - use std.concurrency probably. Any idea why it is behaving the way it is?
parallel threads stalls until all thread batches are finished.
I use foreach(s; taskPool.parallel(files, numParallel)) { L(s); } // L(s) represents the work to be done. to download files from the internet. Everything works. The issue is this: the foreach will download 8 files at once. BUT it will not start the next batch of 8 *until* ALL of the previous 8 are done. It seems that taskPool.parallel will not immediately start a new thread once a task is done E.g., I get L(s1); L(s2); ... L(s8); --- // nothing below is executed until all L(s1) through L(s8) are finished. L(s9); L(s10); ... My expectation is that, say, when the first task is complete, say L(s4), that L(s9) is then executed. The reason why this causes me problems is that the downloaded files, which are cashed to a temporary file, stick around and do not free up space(think of it just as using memory) and this can cause some problems some of the time. Also, the point of parallel tasks is to allow paralleling but the way the code is working is that it starts the tasks in parallel but then essentially stalls the paralleling a large portion of the time. E.g., If there are a bunch of small downloads but one large one, then that one large download stalls the everything. E.g., say L(s5) is a very long download while all others are very quick. Then L(s5) will prevent downloading anything afterwards until it is finished(I'll get L(s1) through L(s8) but nothing else until L(s5) is finished). What's going on and how to reconcile?
Re: Code organization, dub, etc.
On Monday, 13 March 2023 at 13:58:29 UTC, Adam D Ruppe wrote: I'm not particularly interested in defending dub - i consider it a useless piece of crap that I only suffer through cuz some users demanded it For the record, I wasn't trying to attack dub (or dfmt). I was more interested in determining whether DLang (the Foundation and/or the community) would consider working with the CMake and clang-format communities to get them to support D in their products, or whether they prefer to stick with what they have in terms of tooling. I think I heard someone at DConf 22 ask about getting Intellij support for D and Walter said that JetBrains would ask for funding to do that. I don't think CMake support/development is in the same boat, is it?
Re: Code organization, dub, etc.
On Monday, 13 March 2023 at 13:32:04 UTC, Mike Parker wrote: The package registry is full of libraries, yes. That's what it's primarily for. There aren't a lot of executables uploaded there because they're usually better distributed in other ways. But plenty of people are using dub to build them. One way to handle multiple executables is to write a simple script that makes multiple calls to dub with the configurations you need. That's essentially the same as using a Makefile (or CMake with custom commands) to build your project. What I had in mind when I mentioned "multi-executable projects" was something like Postgres or say, a tenth of that, e.g., a server executable, two or more client and utility executables and one or more libraries, all of it spread over a few directories, possibly to a depth of say, two from the root. Can dub handle something like that, e.g., can it handle nested dub.json's, or wouldn't it make much more sense to build such a thing with make, cmake, meson?
Re: Code organization, dub, etc.
On Monday, 13 March 2023 at 12:56:57 UTC, Bradley Chatha wrote: For better or for worse we're stuck with dub as the standard package manager + build tool one-in-all for most of our open source libraries. Yeah, it seems like it's *only* for libraries (and a few single-exe utilities). Looking at code.dlang.org, under "Stand-alone applications/Server software", the top rated item is "handy-httpd" which according to its dub.json builds a library! And the second place "voxelman" is builds three libraries and one executable, which appears to be a "launcher" to access the libraries as plugins.
Re: Passing and returning arguments by ref
Thanks, Ali. On Friday, 3 March 2023 at 18:09:01 UTC, Ali Çehreli wrote: Think may be due to D not having reference variables. Sometimes one needs to use pointers. Ah! I'm about five chapters away from Pointers ;-). Actually, I had tried changing B.x1() to: `ref X x1() { return [0]; }` but the compiler didn't accept it. It's a bit weird that by taking the address of calling B.x1() and thus getting an X*, I had to *dereference* that to pass it to the helper function of A.mfd() which actually takes a `ref C` argument.
Passing and returning arguments by ref
Let's say we have two classes, A and B. The latter has a dynamic array of X and type X has an add() method that can be used to append elements (of type C, another struct) to X's own dynamic array of C. So it's something like the following: ```d struct C {} struct X { C[] cs; void add(C c) { cs ~= c; } } class A { X x; void mfd(ref B b) { // manipulate A.x and one B.xs (returned by B.x1() // by calling another member function // that calls X.add(some C) } } class B { X[] xs; this() { xs.length = 1; } ref X x1() { return xs[0]; } } ``` After A and B are instantiated and A.mfd() is called, the changes to A.x are visible in main. However, although the changes to the X returned by B.x1() are visible while A.mfd() executes, the B.xs[0] is unchanged after it returns. My understanding was that since A, B and X[] are all reference types, this ought to work, but obviously something is missing.
std.math log and family
I've discovered that the 'log' function as well several similar functions are only defined as taking a real argument and returning a real, unlike most other std.math functions, which have triple definitions (also supporting double and float types). This created a problem when I tried compiling code like the following: --- import std.math; alias double function(double) Funcptr; Funcptr sinptr = Funcptr tanptr = Funcptr logptr = --- dmd (and ldc2) report test.d(7): Error: cannot implicitly convert expression & log of type real function(real x) pure nothrow @nogc @safe to double function(double) [ldc2 also reports test.d(6): Error: cannot implicitly convert expression & tan of type real function(real x) pure nothrow @nogc @trusted to double function(double) but apparently this is an LDC-only problem] I'd like to know if the lack of double/float versions of 'log', 'log10', etc. are intentional, i.e., there's some rationale behind it, or an oversight. I'd also like to know the proper/best way to deal with the error, considering the real code embeds the 'Funcptr's in an array of structs. Is it better to write a function 'mylog' that internally casts the argument and the return value, or is there some safe way to cast away the 'log' function at the point of initializing the array? Also, is it preferable to post the first question (intentional/oversight) in the Phobos forum? And what is the preferred way of reporting the LDC problem, via GitHub?
Re: Dlang tour - Unittesting example
On Tuesday, 2 October 2018 at 13:24:09 UTC, Basile B. wrote: The problem is the NaN madness. Since several values are NaN there's this strange stuff: void main() { import std.stdio; import std.math : isNaN; double d; writeln(d.init);// nan writeln(d); // nan writeln(d.nan); // nan assert(d.isNaN); assert(d == d.nan); // fails assert(d == d.init); // fails } the last assert is just crazy. OK, so changing the example from // .init a special built-in property that // returns the initial value of type. assert(vec.x == double.init); to import std.math : isNaN; assert(vec.x.isNaN); doesn't cause the crash. Although I'm a bit puzzled still, I thought (coming from Python) that when you ran the unittest version it would report which tests passed and which failed, not run through main().
Re: Dlang tour - Unittesting example
On Tuesday, 2 October 2018 at 12:25:19 UTC, Joe wrote: On Tuesday, 2 October 2018 at 09:59:28 UTC, bauss wrote: On Tuesday, 2 October 2018 at 04:13:01 UTC, Joe wrote: There appears to be a problem with the example at https://tour.dlang.org/tour/en/gems/unittesting If compiled with -unittest, the resulting program crashes. It happens with ldc2 on Linux but it can also be seen if you click on "Export" and run it with dmd -unittest. I think it's more likely a problem with your OS. I am unable to reproduce that with either of dmd or ldc. Well then it's also a problem with run.dlang.io, since as I said it also happens when you run it there. I forgot to mention: at the end, it reports "1/1 unittests FAILED". I see three tests--two in the struct and the separate one--but the assertion failure is in line 49 (the standalone) so apparently the other two are not being run (or reported).
Re: Dlang tour - Unittesting example
On Tuesday, 2 October 2018 at 09:59:28 UTC, bauss wrote: On Tuesday, 2 October 2018 at 04:13:01 UTC, Joe wrote: There appears to be a problem with the example at https://tour.dlang.org/tour/en/gems/unittesting If compiled with -unittest, the resulting program crashes. It happens with ldc2 on Linux but it can also be seen if you click on "Export" and run it with dmd -unittest. I think it's more likely a problem with your OS. I am unable to reproduce that with either of dmd or ldc. Well then it's also a problem with run.dlang.io, since as I said it also happens when you run it there.
Dlang tour - Unittesting example
There appears to be a problem with the example at https://tour.dlang.org/tour/en/gems/unittesting If compiled with -unittest, the resulting program crashes. It happens with ldc2 on Linux but it can also be seen if you click on "Export" and run it with dmd -unittest.
Re: Access to structures defined in C
On Tuesday, 18 September 2018 at 13:47:50 UTC, Atila Neves wrote: On Tuesday, 18 September 2018 at 02:39:39 UTC, Joe wrote: The second type is like that shown above. The first is a simpler array of pointers to int, e.g., int *yp = {2, 4, 0}; int *yq = {10, 12, 0}; This is valid C in the sense that it compiles, but I doubt it does what you think it does. This is equivalent code: int *yp = 2; int *yq = 10; Sorry, Atila, I got confused looking at my two cases. I should have said "an array of ints", e.g., int yp[] = {2, 4, 0}; int yq[] = {10, 12, 0}; int *ys[] = {yp, yq, 0}; This isn't even valid C code. It is, because C treats 'yp' as a pointer. In D, I first declared these as int[] yp = [2, 4]; int[] yq = [10, 12]; __gshared int*[] ys = [ , ]; D dynamic arrays are not equivalent to C arrays. It's hard to see what you're trying to do with the code you posted. Have you tried instead to use a tool to translate the C headers? At this point, I've translated everything, even the code above. I had to use 'immutable(int [])' in the second and higher level arrays like 'ys' so that they could refer to 'yp' and 'yq' (without the address operators). I still have to make several changes to the library code because these arrays were declared as dynamic arrays of pointers, of size 1, to bypass bounds checking. Now they're proper dynamic arrays, and I can use foreach on them and get other D benefits. However, I still would like to have a deeper understanding of the "static variable yp cannot be read at compile time" error messages which went away when I declared yp immutable.
Re: Access to structures defined in C
On Sunday, 10 June 2018 at 17:59:12 UTC, Joe wrote: That worked but now I have a more convoluted case: a C array of pointers to int pointers, e.g., int **xs[] = {x1, x2, 0}; int *x1[] = {x1a, 0}; int *x2[] = {x2a, x2b, 0}; ... int x2a[] = { 1, 3, 5, 0}; Only the first line is exposed (and without the initialization). So I tried: extern(C) __gshared extern int**[1] xs; After a long hiatus, I'm back to working on something related to the above, but now that various other C pieces have been converted to D I'm down to converting these static arrays to D. There are two arrays that are giving me trouble. The second type is like that shown above. The first is a simpler array of pointers to int, e.g., int *yp = {2, 4, 0}; int *yq = {10, 12, 0}; int *ys[] = {yp, yq, 0}; In D, I first declared these as int[] yp = [2, 4]; int[] yq = [10, 12]; __gshared int*[] ys = [ , ]; The compiler (ldc2) gave errors like "cannot take address of thread-local variable yp at compile time" or "static variable yp cannot be read at compile time" (when I replaced the "" by "yp[0]". Eventually, I managed to get them to compile without errors by using the following: immutable int[] yp = [2, 4]; immutable int[] yq = [10, 12]; __gshared immutable(int[])[] ys = [ yp, yq ]; I still haven't tested them (or linked them) so I don't know what other changes I'll have to make to the library (now in D) where ys is declared as: __gshared extern int*[] ys; Presumably changing it to __gshared extern immutable(int[])[] ys; should do the trick (everything is still within extern (C))? But I suspect I'll have to change several array access statements as well. However, I've been unable to compile the other case, which now looks like this: immutable int x1a[] = [ 9, 7, 5]; immutable(int[])[] x1 = [ x1a ]; immutable(int[])[] x2a = [ 1, 3, 5]; ... immutable(int[])[] x2 = [ x2a, x2b ]; __gshared immutable(int[])[][] xs = [ x1, x2 ]; The error is like "static variable x2a cannot be read at compile time". It seems like I would have to wrap that immutable(int[]) within another immutable, as immutable(immutable(int[])[]) x2 ... and extend that to xs as well. I'll try it later but I'd like some confirmation or better yet, pointers to where this is explained in some comprehensible way, i.e., on what can you apply an address operator or array subscript and when, is the behavior differerent for immutable, const, static, thread-local and why?
Re: Linking a C program with D library
On Wednesday, 15 August 2018 at 06:39:50 UTC, Mike Parker wrote: String literals are implicitly convertible to const(char)* and are guaranteed to be nul-terminated like a C string, so this works: [...] Does that help? Yes, indeed. I think I possibly read about literal strings being nul-terminated somewhere but it must've slipped my mind.
Re: Linking a C program with D library
On Wednesday, 15 August 2018 at 01:56:34 UTC, Mike Parker wrote: The correct thing to do is to keep the original C function signatures in the converted code, i.e. don't change char* to string[]. And don't use anything from Phobos internally that requires linking. In other words, treat your converted code as C code in a D module as much as possible. Only when the conversion is complete and everything is in D do you start pulling in the D features. I understand that, Mike. However if I'm not mistaken given something in C like char* strs[] = { "This", "is a", "test"}; AFAIK, even with -betterC and an extern (C), the literals will still be understood by D as type "string", and there is no other way around it, right? I could put the array and the function in its own C file for the time being, but instead chose to replace the toStringz by a small hack: use memcpy to copy the string to a stack fixed, big enough array and a NUL terminator.
Linking a C program with D library
I'm attempting a piecemeal conversion of some C programs. I've converted a few that depend on C modules that are in a library and now I'm sort of in the middle of converting those C modules. One of them has an array of strings, i.e., array of char*, which most easily translated to D's string[]. However, the array is used by a function that expects a const char*, so I had to use toStringz to pass those values. Now there are some C programs that haven't been converted yet, but they depend on the function above. When linking those programs, the linker complains about an undefined reference to the mangled name of std.string.toStringz. What is the most appropriate way around this? Is there a way to declare an array of const char* and initialize it to literal strings? Or can the mangled name (and Phobos library) somehow be made visible to the linker?
?? How to subscribe to Multicast Broadcasts ??
Hello All! I've been trying every possible combination and cannot get anything working. (>_<) This is I think the closest I've got, I think the problem may be with the 3 argument. I'm not sure how to join the Multicast IP membership? (this code currently does not work - throws error: 'Unable to set socket option: An invalid argument was supplied'.) Socket listen_socket = new Socket(AddressFamily.INET, SocketType.DGRAM, ProtocolType.UDP); listen_socket.setOption(SocketOptionLevel.IGMP, SocketOption.IPV6_JOIN_GROUP, 1); auto adr = getAddress("0.0.0.0", "56000"); listen_socket.bind(adr[0]); This is how the messages are sent (which works fine): Socket udp_broadcast_soc = new UdpSocket(); udp_broadcast_soc.setOption(SocketOptionLevel.SOCKET, SocketOption.BROADCAST, 1); auto adr = getAddress("239.192.0.10", 54000); udp_broadcast_soc.sendTo(, adr[0]); Please Please Please Help, I am desperate! Many Thanks in Advance for your time & attention, -joe
How the Heck Do I Subscribe to Receiving Multicast Messages Broadcast to Organization's IP ??
Hello All! I've been trying every possible combination and cannot get anything working. (>_<) This is I think the closest I've got, I think the problem may be with the 3 argument. I'm not sure how to join the Multicast IP membership? (this code currently does not work - throws error: 'Unable to set socket option: An invalid argument was supplied'.) Socket listen_socket = new Socket(AddressFamily.INET, SocketType.DGRAM, ProtocolType.UDP); listen_socket.setOption(SocketOptionLevel.IGMP, SocketOption.IPV6_JOIN_GROUP, 1); auto adr = getAddress("0.0.0.0", "56000"); listen_socket.bind(adr[0]); This is how the messages are sent (which works fine): Socket udp_broadcast_soc = new UdpSocket(); udp_broadcast_soc.setOption(SocketOptionLevel.SOCKET, SocketOption.BROADCAST, 1); auto adr = getAddress("239.192.0.10", 54000); udp_broadcast_soc.sendTo(, adr[0]); Please Please Please Help, I am desperate! Many Thanks in Advance for your time & attention, -joe
Re: How to call a C function from D that takes a FILE * as an argument?
On Wednesday, 4 July 2018 at 02:16:00 UTC, Seb wrote: Hmm, calling e.g. fprintf with stdout should just work: --- void main() { import core.stdc.stdio; fprintf(stdout, "Hello %s", "world".ptr); } --- Could you maybe provide your whole code? This short test program shows the error: --- import std.stdio; void main() { extern (C) void list(FILE *fd); list(stdout); } --- Now I fixed this by changing the import to core.stdc.stdio. I guess the problem is if you import std.stdio (which brings in the other one), there are two slightly incompatible stdout's and the D takes precedence.
Re: How to call a C function from D that takes a FILE * as an argument?
On Wednesday, 4 July 2018 at 01:58:15 UTC, Seb wrote: So just add the declaration to your D file: --- extern(C) void myCfunction(FILE* stream); --- I do have a similar declaration in D. It appears the problem is that the C program I'm trying to convert passes stdout as the argument and the D compiler complains somewhat like the following: Error: function foo.main.myCfunction (shared(_IO_FILE)* stream) is not callable using argument types (File) So I guess the question is what to pass instead of stdout.
How to call a C function from D that takes a FILE * as an argument?
The subject basically says it all. The C function uses the argument to call fprintf and also passes it to other functions where it's used to call fileno, fprintf or putc.
Re: Access to structures defined in C
On Sunday, 10 June 2018 at 17:59:12 UTC, Joe wrote: That worked but now I have a more convoluted case: a C array of pointers to int pointers, e.g., int **xs[] = {x1, x2, 0}; int *x1[] = {x1a, 0}; int *x2[] = {x2a, x2b, 0}; ... int x2a[] = { 1, 3, 5, 0}; Only the first line is exposed (and without the initialization). So I tried: extern(C) __gshared extern int**[1] xs; The D compiler accepts that, but just about any manipulation gets screamed at, usually with Error: only one index allowed to index int. Note that I'm trying to access the ints, i.e., in C something like xs[1][0][2] to access the 5 in x2a. Do I have to mimic the intermediate C arrays? I don't know why I didn't try this first. It seems that the D equivalent of C's xs[1][0][2] is simply xs.ptr[[1][0][2].
Re: Access to structures defined in C
On Wednesday, 14 March 2018 at 02:17:57 UTC, Adam D. Ruppe wrote: The type system would *like* to know, certainly for correct range errors, but if you declare it as the wrong length and use the .ptr, it still works like it does in C: extern(C) __gshared extern char*[1] files; // still works import core.stdc.stdio; void main() { printf("%s\n", files.ptr[2]); // ptr bypasses the range check } That worked but now I have a more convoluted case: a C array of pointers to int pointers, e.g., int **xs[] = {x1, x2, 0}; int *x1[] = {x1a, 0}; int *x2[] = {x2a, x2b, 0}; ... int x2a[] = { 1, 3, 5, 0}; Only the first line is exposed (and without the initialization). So I tried: extern(C) __gshared extern int**[1] xs; The D compiler accepts that, but just about any manipulation gets screamed at, usually with Error: only one index allowed to index int. Note that I'm trying to access the ints, i.e., in C something like xs[1][0][2] to access the 5 in x2a. Do I have to mimic the intermediate C arrays? But if you can match the length, that's ideal. Unfortunately, although the C array lengths are known at C compile time, they're not made available otherwise so I'm afraid the [1] trick will have to do for now.
Re: unittests, dub and libraries
On Wednesday, 28 March 2018 at 03:07:23 UTC, Jonathan M Davis wrote: Run dub test The problem is that an executable needs a main, and a library doesn't have one, whereas when you're testing a library, you need an executable. So, a main must be inserted - e.g. with the -main flag to dmd. Just building the unittest build doesn't insert one. However, dub test _does_ deal with that for you. Thanks.
unittests, dub and libraries
I'm trying to build a very simple library. For now it just has a single class, constructor, destructor and one method. I added a unit test right after the method, declared the targetType to be "library" and a buildType of "unittest" (with options "unittests", "debugMode", "debugInfo"). When I run dub run -b unittest it builds the library, but then says: Target is a library. Skipping execution. If I compile with ldc2 -unittest the linker throws the error: (.text+0x20): undefined reference to `main' If I add an empty main function to the source file, ldc2/gcc manage to create an executable that can be invoked manually and it runs through the unit test. Is this the best that can be done?
Re: core.stdc.stdlib._compare_fp_t and qsort
On Sunday, 18 March 2018 at 19:01:11 UTC, Joe wrote: I managed to get it working by declaring a D dynamic array, appending n_recs pointers to it and using it as argument to sort. Unfortunately, I then had to copy from the dynamic array to the fixed array in order to continue using the latter. Any shortcuts around this? I just saw your other reply. Passing recs[0..n_recs] does the trick. Thanks.
Re: core.stdc.stdlib._compare_fp_t and qsort
On Sunday, 18 March 2018 at 18:11:02 UTC, Dmitry Olshansky wrote: Well since recs is array of pointers this looks like a null pointer in your data. The usual ways to fix that is either print stuff or poke around in debugger to see if a Record* is null or .name is null. The problem is that although the "recs" array is declared as having 10 pointers, not all pointers are used. In the qsort case, the second argument limited the sort to just the first n_recs (valid) pointers. There doesn't seem to be a way to tell std.algorithm.sorting.sort to only look at part of the fixed array. I managed to get it working by declaring a D dynamic array, appending n_recs pointers to it and using it as argument to sort. Unfortunately, I then had to copy from the dynamic array to the fixed array in order to continue using the latter. Any shortcuts around this?
Re: core.stdc.stdlib._compare_fp_t and qsort
On Sunday, 18 March 2018 at 13:10:08 UTC, Dmitry Olshansky wrote: Do this to get the usual ptr + length: sort!((a, b) => to!string((*a).name) < to!string((*b).name))(recs[]); Also to!string would be computed on each compare anew. May want to use schwartzSort to avoid that, on 10 elements there is no real difference though. The 10 elements are just because it's a small test program. What does changing "recs" to "recs[]" as the argument actually do? Does it duplicate the fixed array on the fly? [I guess I have to study more!] The change does pass the compiler, but at runtime it causes a segfault. The next to last frame in the backtrace shows (having changed from to!string to fromStringz and using a string template instead of a lambda): #6 0x55565760 in std.algorithm.sorting.sort!("fromStringz((*a).name.ptr) < fromStringz((*b).name.ptr)", 0, testd.Record*[]).sort(testd.Record*[]) (r=...) Then it goes through quickSortImpl, shortSort and sort5, moving on to either std.functional.binaryFun or processing of the lambda, with a and b equal to 0, ending with a segfault in a ?? call from fromStringz or in memcpy called from object._dup (in the to!string case).
Re: core.stdc.stdlib._compare_fp_t and qsort
On Monday, 12 March 2018 at 03:50:42 UTC, Joe wrote: On Monday, 12 March 2018 at 03:13:08 UTC, Seb wrote: Out of interest: I wonder what's your usecase for using qsort. Or in other words: why you can't use the high-level std.algorithm.sorting.sort? This is only temporary. I will be using std.algorithm.sorting.sort. I was converting a C program and it annoyed me that I couldn't get the qsort invocation past the D compiler. Now that I'm trying to use std.algorithm.sorting, I'm again puzzled by what I need to use for the "less" predicate. My first try was: sort!((a, b) => to!string((*a).name) < to!string((*b).name))(recs); This results in the error: Error: template std.algorithm.sorting.sort cannot deduce function from argument types !((a, b) => to!string((*a).name) < to!string((*b).name))(Record*[10]), candidates are: /usr/lib/ldc/x86_64-linux-gnu/include/d/std/algorithm/sorting.d(1851):std.algorithm.sorting.sort(alias less = "a < b", SwapStrategy ss = SwapStrategy.unstable, Range)(Range r) if ((ss == SwapStrategy.unstable && (hasSwappableElements!Range || hasAssignableElements!Range) || ss != SwapStrategy.unstable && hasAssignableElements!Range) && isRandomAccessRange!Range && hasSlicing!Range && hasLength!Range) which is not very helpful. It's a bit different from the "no function match" that Adam complains about, but I presume it's because now we're dealing with templates, and although the compiler finds a single candidate, it's not satisfactory but it can't tell the user anything further. I've tried using fromStringz((*x).name.ptr) instead of to!string (I'm still unclear to what extent can templates be used within templates). I also tried using an explicit cast(Record *)x because I'm also unsure that type information is passed down. Neither change helped.
Access to structures defined in C
What is the correct way to declare and access storage managed by C? For example, say a C module defines an array of filenames, e.g., char *files[] = { "one", "two", "three", 0}; The C header of course declares this as: extern char *files[]; The way to declare it in a D module appears to be: extern (C) { __gshared extern char *[] files; } However, trying to index into this, e.g., char *filename = files[1]; does not work. In gdb if I try to examine 'filename' I see something like a reverse text string but treated as an (inaccessible) address and calling fromStringz causes a segfault.
Re: core.stdc.stdlib._compare_fp_t and qsort
On Monday, 12 March 2018 at 03:13:08 UTC, Seb wrote: Out of interest: I wonder what's your usecase for using qsort. Or in other words: why you can't use the high-level std.algorithm.sorting.sort? This is only temporary. I will be using std.algorithm.sorting.sort. I was converting a C program and it annoyed me that I couldn't get the qsort invocation past the D compiler.
Re: core.stdc.stdlib._compare_fp_t and qsort
On Monday, 12 March 2018 at 01:45:54 UTC, Adam D. Ruppe wrote: I just reformatted it but now the difference should be visible: `extern(C)` is missing on your callback. The scope things might make a difference too, but I know for sure extern(C) is necessary on your callback function. I saw the extern(C) and I believe I tried it before my previous post, but dismissed it because I saw no difference in compiler behavior. In any case declaring extern (C) int compar(const (void *) p1, const (void *) p2) { ... // as before } still gives the error: function core.stdc.stdlib.qsort (scope void* base, ulong nmemb, ulong size, extern (C) int function(scope const(void*), scope const(void*)) @system compar) is not callable using argument types (void*, ulong, ulong, extern (C) int function(const(void*) p1, const(void*) p2)) It only went away with extern (C) int compar(scope const (void *) p1, scope const (void *) p2) Could you explain or direct me to something that elucidates why the "scope" qualifiers are needed? And is there something else that is necessary inside compar() or are the "scope"s just for decorative purposes? Joe
Re: core.stdc.stdlib._compare_fp_t and qsort
On Sunday, 11 March 2018 at 23:26:04 UTC, Stefan Koch wrote: You have to pass a pointer to the function. Otherwise it'll be a parenthsis-less call. use : qsort(recs, num_recs, (Record *).sizeof, ); After passing a pointer, getting some other error messages, I changed the call to qsort(cast(void *)recs, num_recs, cast(size_t)(Record *).sizeof, ); and the latest error is: Error: function core.stdc.stdlib.qsort (scope void* base, ulong nmemb, ulong size, extern (C) int function(scope const(void*), scope const(void*)) @system compar) is not callable using argument types (void*, ulong, ulong, int function(const(void*) p1, const(void*) p2)) I fail to see which argument is causing the problem now. Joe
core.stdc.stdlib._compare_fp_t and qsort
I'm getting a compiler error in a qsort() call as follows: qsort(recs, num_recs, (Record *).sizeof, compar); Record is a struct, recs is a fixed array of pointers to Record's and num_recs is a size_t that holds the number of valid records. compar is this: int compar(const void *p1, const void *p2) { import core.stdc.string : strcmp; const Record **rp1 = cast(Record **)p1; const Record **rp2 = cast(Record **)p2; return strcmp((*rp1).name.ptr, (*rp2).name.ptr); } The error is: Error: function testd.compar (const(void*) p1, const(void*) p2) is not callable using argument types () I don't quite understand what those parentheses mean: is it implying "no arguments" and if so, where would one provide arguments? Joe
Re: Article: Why Const Sucks
On Thursday, 8 March 2018 at 15:13:09 UTC, Steven Schveighoffer wrote: On 3/8/18 9:58 AM, joe wrote: On Monday, 5 March 2018 at 10:57:35 UTC, Jonathan M Davis wrote: Here's something I wrote up on const: /snip May be not entirely related, but a little gotcha. given: interface XY {} class Foo: XY {} class Bar: XY {} void doSomething(in XY sth) { auto foo = cast(Foo)sth; // error in @safe code } But the compiler doesn't emit a warning that const got cast away in @system code. Since @system is the default setting and casting const away is by definition undefined behavior, there should be some feedback. If you cast away const in C++ you need to be explicit about it by using const_cast, documents the intention rather than D's swiss-army-cast. This is a problem with the cast system, not const. ... I figured. Sorry for off topic. ... D has one cast mechanism, and it's used to: a) dynamic type cast b) hook user-supplied opCast c) enter undefined behavior by casting away const/immutable. I really think we should have a separate mechanism for a and b, as they are far less dangerous than c. std.conv.to does this as well, ... Thanks for pointing this out. ...but the language should have some safeguards against using "safe casting" incorrectly. -Steve yes, that's what I was aiming for :) This one got me because I assumed, with all the improved security in place, D would never silently do such a thing... Assumptions, right? ...the root of all evil. I should know better. haha. Anyways, thanks for the insight and have a nice Sunday.
Re: Article: Why Const Sucks
On Monday, 5 March 2018 at 10:57:35 UTC, Jonathan M Davis wrote: Here's something I wrote up on const: /snip May be not entirely related, but a little gotcha. given: interface XY {} class Foo: XY {} class Bar: XY {} void doSomething(in XY sth) { auto foo = cast(Foo)sth; // error in @safe code } But the compiler doesn't emit a warning that const got cast away in @system code. Since @system is the default setting and casting const away is by definition undefined behavior, there should be some feedback. If you cast away const in C++ you need to be explicit about it by using const_cast, documents the intention rather than D's swiss-army-cast.
Re: Article: Why Const Sucks
On Monday, 5 March 2018 at 10:57:35 UTC, Jonathan M Davis wrote: Here's something I wrote up on const: http://jmdavisprog.com/articles/why-const-sucks.html I suppose that it's not exactly the most positive article, but I feel that it's accurate. - Jonathan M Davis Interesting read and it explains some of the incomprehensible error messages. Basically you're saying we've got a strong const in D but essentially the situation is much like in Java because the general approach is don't use it ? Kind of ironic. I made a Logger module a couple years back and I used const very generously. Then came the moment where the LogWriters actually had to do the writing. One would assume that a function called 'stdio.write' mutates some state, but it sure as hell doesn't affect the state of my logger. DMD said: So swy! writeln not const-y, no can do, swy! So I remove const. One function at a time and end up with a module where basically no const remains. But it doesn't stop there. Since logger isn't const anymore, it now can't be used in any const functions. So more de-const-ification happened... Also, it's frustrating to have to get rid of const because Object.toString uses some Outputwriter thing which is not const and therefore transitively you can't have const anywhere else. Kind of a disappointing experience. As far as I'm concerned, it's not so much "don't use const; or, it's not worth it" but more like "I would if I could but I can't so I shan't". I bother with const as much as possible but it's incredible frustrating and I would argue the time lost to de-const-ify APIs is easily equivalent to a code breaking change that would make const more applicable. Something possimpible. Like a compiler that doesn't purely make decisions upon a const keyword, but one that can understand that interacting with some random function with only "in" parameters or objects, which aren't even interacting with,or are part of, the object's internal state and are just passed through, can't violate the const-ness of the object. But there's likely much more to consider than that and I couldn't ever dream of implementing such a thing..unfortunately
Re: Postgres and other database interfaces
On Sunday, 25 February 2018 at 16:31:45 UTC, Denis F wrote: What are these sacrifices for? I do not like idea that I should drop some functionality of Postgres for conpatibility with another RDBMSes. Personally, I like specific databases because it have nuances of functionality that they provide to me. Nobody said or implied anything about dropping Postgres functionality. If you can, please take a look at the contents of the Psycopg docs (http://initd.org/psycopg/docs/), the most popular Python Postgres adapter: after the entries for the connection and cursor classes, you'll see pyscopg2.extensions, pyscopg2.extras, psycopg2.pool and more. It even includes support for the recently added logical replication feature. The "sacrifice" benefits the library implementors because it expands their potential market. Even though you decry the differences between RDBMS, when it comes to the basics, I would argue that 80% of the code written by a SQL DB developer is very similar: connect to the DBMS (or to a file for sqlite), issue a query, retrieve results, update some rows. If they're writing in C, D, Python, Ruby or whatever, most of their time is spent writing or tweaking SQL statements. We're not talking about MongoDB, Cassandra, OODBMS or other creatures that *do* have significant differences (it's telling that several have implemented SQL layers on top). And those who wish to "crossbreeding a hedgehog with a snake", at first, can try to write a wrapper around existing SQL libraries. This is faster and more humane in relation to the developers. Please take a look at the list of RDBMSs supported by SQLAlchemy (http://docs.sqlalchemy.org/en/latest/core/engines.html#supported-databases). I'm *not* in favor of ORMs, but do you think that Mike Bayer (the implementor) would have had any incentive to write an ORM if a DB-API hadn't existed about 10 years before (v1 I believe is circa 1996, v2 from 1999, SQLA came out in 2006 and it already supported four DBMSs). Even if he had had the incentive it definitely would've been much slower and error-prone, because he would have had to not only design and implement an ORM but also have to deal with idiosyncracies of each library, whereas now he can support multiple libraries for each DBMS, e.g., psycopg2 and pg8000 for Postgres, because they both conform to the same API. You're obviously reluctant of having to rewrite dpq2 just to "play along with the other kids". I can understand that, but who knows, maybe your design is 90% of what the others want/have/like and you may only have to make some trivial changes here and there. In any case, as I mentioned earlier, I will delve into dpq2 (and vibe-d-postgresql) for my own project. So I'll keep in touch. Joe
Re: Postgres and other database interfaces
On Sunday, 25 February 2018 at 09:23:18 UTC, rikki cattermole wrote: On IRC earlier today we discussed database libs a bit, we agreed that both "?" and "@name" needed to be supported. No other suggestions came up. We don't really need a third or fourth form I think. "@name"? Which DBMS uses that? Although Denis said Postgres uses $1, PG also supports :name, in either the command line tool (psql) and the procedural language PL/pgSQL. Personally, I prefer what Python's PEP 249 calls 'format' (%s) or 'pyformat' (%(name)s). Note that although the %s appears in the SQL string, the actual arguments are passed separately at invocation time, as either tuples or dicts (associative arrays). Joe
Re: Postgres and other database interfaces
On Sunday, 25 February 2018 at 09:07:34 UTC, Denis F wrote: Libraries already are similar automatically because RDBMSes uses similar principles. But it is impossible to make libraries absolutely identical due to the presence of important significant nuances in each engine. I don't see how making libraries compatible with each other, i.e., complying with some standard, will result in losing functionality. What I'm talking about is standardizing the names of certain classes and their methods, exceptions, and perhaps some datatypes and the expected order of certain arguments. For example, instead of having classes named Database (sqlite3), PGConnection (ddb) and Connection (dpq2, mysql-native) there's just one name, and that class is expected to have certain methods. The library implementor can add others. For example, I see mysql-native has a connection pool and that doesn't need to be part of the standard. Also, some methods can be specified as being optional by the standard, e.g., something like callproc, because not all DBMSs support procedures. Ok, I propose to be consistent and ask for compliance with the Standard from the RDBMSes. For example, arguments substitution: MySQL uses the '?' PostgreSQL uses $1 SQLite accepts both Oracle uses a :name (Really, it is very important to come to an agreement here because, for sure, the next obvious step is writing an ORM generator on top of the idea what you are proposing.) In PEP 249, this was left up to the implementors (see the paramstyle global), but a footnote suggests that some styles are supposed to be preferred over others. For example, I would not change unobvious at first sight system of classes "Result" and inherited from it class "Answer" for the sake of similarity to other libraries. Simply because this is the way what Postgres client library works, and if this classes will be removed some significant functionality will be lost. I'm sure that in other libraries there is something similar. I'm not sure I understand your example, but perhaps this is getting too specific and close to the discussions that would need to take place betweeen existing implementors. There has to be a negotiation process for a standard to be developed. In my example above, suppose the sqlite3 library developer says "I don't want to have a class named Connection, because my users don't connect to a server". It will be up to the other developers to convince him or to arrive at some compromise (in the case of D an alias may be all that is needed to resolve such an impasse). Joe
Re: Postgres and other database interfaces
On Saturday, 24 February 2018 at 23:04:39 UTC, Denis F wrote: If anyone really want to impliment your idea, at my first glance at the PEP 249 I had a feeling that this is work for time less than a 1-2 weeks. It can be a simple wrapper over dpq2, mysql-native, sqlite3, etc. I'm not saying that anyone should implement something like PEP 249, particularly *not* as a wrapper over those libraries. My point is that if the implementors of dpq2, mysql-native, etc., could "get together" (at DConf, in forums, by email) and discuss their interfaces, perhaps they could agree on a set of D database interface classes and then each would work towards that common set. That's more or less what the Python DB-SIG did. Then every library would benefit because, for one, the common set can be documented just once. Also, if it's a sufficiently reasonable spec, the libraries that don't move towards it should see less adoption because, unless the divergent library has a significantly enhanced interface, developers will shy away from having to learn a different one. It's the same concept as the SQL (and other) Standards. There may be variances between implementations but no one has implemented a RETRIEVE or PROJECT statement in lieu of a SELECT statement (and note there *were* relational systems that used the former). Joe
Re: Postgres and other database interfaces
On Saturday, 24 February 2018 at 18:56:23 UTC, Denis F wrote: It seems to me that this is a common misconception that it is possible to create a good universal tool for accessing the database in place of the many existing ones. I don't think what is needed a "good universal tool" but a good enough common interface. Python PEP 249 is very similar to what I've seen in the D libraries I've looked at: there is a Connection class of some kind, and a Cursor (or command or query) class, each with a set of well-defined (and generally not unexpected) methods: commit/rollback on the connection, execute/fetch/etc. on the cursor). PEP 249 also standardizes the exceptions and some datatypes. It's not perfect, or universal enough, but I believe it allowed db development under Python to proceed at a faster pace than otherwise. There's a similar story on the web side. When Django first came out, IIRC it had its own interface, but then the WSGI standard came out (PEP 333) and every Python web framework including Django moved to it (it also resulted in a proliferation of frameworks, but that's another story). And WSGI even influenced Ruby's Rack and other such interfaces. Joe
Re: Postgres and other database interfaces
On Saturday, 24 February 2018 at 18:41:21 UTC, Denis F wrote: On Saturday, 24 February 2018 at 05:33:56 UTC, Joe wrote: 2. dpq2. Second most downloaded and built on top of derelict-pq. The "documentation" consists of a README listing the features and a single example, which appears to focus on support for JSON/BSON. Hi! Here is dpq2 and vibe-d-postgresql developer. The problem is that I do not know English very well. So,I think it's better not to write any documentation than to write the wrong one. PRs are welcomed! 3. vibe-d-postgresql. Third most downloaded and built on top of dpq2. Docs consist of a single example program. Well Denis, you may get your PR wish (but I don't have a timeframe). My interest is to develop something using Vibe.d, so I'll be delving more into these two first. Joe
Re: Postgres and other database interfaces
On Saturday, 24 February 2018 at 15:44:49 UTC, Paolo Invernizzi wrote: Joe: I think this is also a terrific 'welcome aboard' about how fast and quickly things can be done in D! I'm a bit overwhelmed by all the replies and "goodies" but I'd like to say thanks to Adam while I digest it all. Joe
Re: Postgres and other database interfaces
On Saturday, 24 February 2018 at 10:13:35 UTC, Paolo Invernizzi wrote: Have you tried to generate the documentation? Look for example inside the source files... https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/postgres.d#L2 https://github.com/pszturmaj/ddb/blob/bd55beea0df6e5da86a71535ff6d9800c0711c7c/source/ddb/db.d#L19 I think no... The documentation on how to use the package is present, it's just inside the modules as Ddoc documentation. Joe, you can easily 'extract' it using the proper -D compiler switch As I said, I'm new and still not familiar with those facilities although I saw those things and they looked like embedded HTML and was wondering what they were for. Again, coming from Python, I'm familiar with RTD (https://readthedocs.org, my own open source Python/Postgres project is hosted there) and Sphinx (http://www.sphinx-doc.org/en/master/). Although it started with Python, RTD now hosts many other kinds of projects (Javascript, PHP, Ruby, etc.). There's even a project named 'dlang' (it's not what you expect!), but I did find "D Tips" and "Quick Start with D" which appear to be standalone Markdown. In any case, it would be nice to have such automatically generated docs available in a public site/repo of some kind, like RTD, so a potential user doesn't have to clone the code and run the compiler to see it.
Postgres and other database interfaces
Back on 13 January, I posted in the Learn forum some questions regarding using Postgres and got a reply that stated the following: On Monday, 15 January 2018 at 02:28:29 UTC, Matthias Klumpp wrote: In any case, please don't start another Postgres library and consider contributing to one of the existing ones, so that we maybe have one really awesome, 100% complete library at some point. I'm revisiting this because I now have had the chance to look at the various libraries and IMHO the "really awesome, 100% complete library" depends on the users' goals and their definition of awesome and complete. Aside from ddbc which is like ODBC/JDBC and thus supports multiple db's (not my interest), I've found five Postgres-specific libraries worthy of further exploration: 1. derelict-pq. This is the most downloaded one and is simply a binding over libpq, so AFAICT it's nearly 100% complete. It's also well-documented, in part because the Derelict libraries provide overall guidance, but mainly because a D user can refer directly to the comprehensive libpq documentation and (C) examples. The only strange fact is that it's been stuck in a beta.3 release for the last five months. 2. dpq2. Second most downloaded and built on top of derelict-pq. The "documentation" consists of a README listing the features and a single example, which appears to focus on support for JSON/BSON. 3. vibe-d-postgresql. Third most downloaded and built on top of dpq2. Docs consist of a single example program. 4. ddb. About the same number of downloads as the above. Implemented on top of front/backend protocol. No documentation although repository has a folder with two sample programs. 5. dpq. Implements its own wrapper over libpq and has some ORM functionality. Docs are a README with example program. The main issue is that, other than derelict-pq, using any of these libraries involves reading the library code and understanding the sui generis interfaces implemented by each. I'm new to the D community, but coming from Python, the contrast is significant. First there is the well-documented PEP 249 (https://www.python.org/dev/peps/pep-0249/) promulgated by the Python DB-SIG (and note that 249 is v.2), which led to the implementation of psycopg (also well-documented including various extensions to the API) and many other adapters to Postgres and other databases. Since I'm new, I don't know if there's any interest in moving in such a direction.
Re: Understanding the AST...
On Thursday, 22 February 2018 at 14:53:11 UTC, Seb wrote: On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote: Hello everybody! Last week end I found this post ( https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library/ ) on the Blog and thought to myself awesome. [...] BTW I know it's not as powerful as DMD (and not the real thing), but often the AST XML dump from DScanner helps to deepen the understanding: https://github.com/dlang-community/D-Scanner#ast-dump You can even play with libdparse on the web: https://run.dlang.io/is/qZsGDD Hello Seb, I had a look at the resources you provided and they are quite useful. Thank you. However, while technically a lexer would be enough to solve the problem at hand, I think I'm going to want that full front-end a bit later. The more information, the better. I rather have the option to ignore something I don't need than to need something I don't have :)
Re: Understanding the AST...
On Thursday, 22 February 2018 at 13:44:51 UTC, RazvanN wrote: On Thursday, 22 February 2018 at 13:21:04 UTC, joe wrote: [...] Indeed, @Stefan is right. The ParseTimeVisitor only contains information available at parse time. If you are interested in the parent you have 2 options: either (1) use the ParseTimeVisitor and implement the AST traversal logic yourself or (2) you can use the SemanticTimeTransitiveVisitor in which case the parent is not going to be null. In the case of (2) you need to also do some semantic analysis (so you need the whole dmd library, not just the parsing one). Here's an example on using the dmd library (including semantic) [1]. You can copy paste that example and add a few lines of code where you instantiate your visitor (which will inherit SemanticTimeTransitiveVisitor). [...] awesome, that helps a lot! Thanks both of you :)
Re: Negative index range violation
On Thursday, 22 February 2018 at 02:41:30 UTC, Steven Schveighoffer wrote: On 2/21/18 7:30 PM, SrMordred wrote: But with a slice negative indexes are never allowed, even on a pointer. youd have to do (c-1)[0 .. 1]; Nice! Thank you both! In D Slice article it says "You can even use negative indexes!" so I thought that the [-1..x] should work too :) Hah! I never thought of doing a slice with negative indexes ;) /SNIP -Steve At night I dream about doing something like this auto pos = haystack.find(needle); auto something = haystack[pos..+3]; // meaning [pos..pos+3] auto somethingElse = haystack[pos..-3]; // and [pos..pos-3] respectively :)
Re: Understanding the AST...
On Monday, 12 February 2018 at 08:47:58 UTC, RazvanN wrote: Hi Joe, /SNIP On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote: [...] The FuncDeclaration node contains all the information for that. For example, you can access fd.parent to see if the function is declared at top-level (in which case, the parent is going to be a module declaration ) or if it is a nested function (in a class, in a struct, in a function). Every AST node contains information about the position in the AST, all you have to do is find how to get that information: which field to access or which member function to call. /SNIP Cheers, RazvanN Follow up question... Why is *.parent always null? e.g.: extern(C++) class MyVisitor(AST): ParseTimeTransitiveVisitor!AST { override void visit(AST.Import i) { assert(i.parent is null); // always true } override void visitFuncBody(AST.FuncDeclaration f) { assert(f.parent is null); // always true } }
Re: Understanding the AST...
On Monday, 12 February 2018 at 08:47:58 UTC, RazvanN wrote: Hi Joe, I suggest you watch this video which explains how the parse time visitors work: https://www.youtube.com/watch?v=tK072jcoWv4 . On Tuesday, 6 February 2018 at 12:03:06 UTC, joe wrote: [...] The FuncDeclaration node contains all the information for that. For example, you can access fd.parent to see if the function is declared at top-level (in which case, the parent is going to be a module declaration ) or if it is a nested function (in a class, in a struct, in a function). Every AST node contains information about the position in the AST, all you have to do is find how to get that information: which field to access or which member function to call. [...] The function average length visitor inherits a transitive visitor which means that the AST traversal logic is already implemented for you. All you have to do is override the visiting methods of interest and do whatever suits you : print stuff, alter the ast, stop the visitation or continue the visitation (by calling super.visit(ASTnode)). [...] I know that my explanations might not be very explicit, but if you have an example please post it and we can work on it. Cheers, RazvanN Hello RazvanN, thank you very much for taking the time to reply and also your effort in making this happen. I watched the video you linked and read your reply over and over, yet I still have a hard time to wrap my head around this idea. Like for example DHTML DOM is very easy for me to grasp. It's like riding the car down the country road and I know where I am and which town I'm going to be next, etc. This AST thing is more like a teleporter room on the Enterprise. Scotty activates the teleporter and a canister appears an a spot labeled imports. He repeats and a canister appears on a spot labeled functions, etc. I will try again...
Re: dub segfault
On Sunday, 11 February 2018 at 00:59:10 UTC, Seb wrote: Ideally you could verify whether this also happens with the official release. Installing the official (ldc) release doesn't cause the problem, so presumably this is a problem with the Debian buster release. However, I notice the official dub reports as version 1.6.0 built on 6 Jan, whereas the official ldc is 1.7.0 and the Debian dub is, as mentioned before, 1.7.0-2, and the Debian ldc2 says it's version 1.5.0, based on DMD 2.075.1, i.e., the version numbers are confusing to say the least.
dub segfault
I'm getting a segmentation fault when I run 'dub build' and I specify a dependency. Specifically, the dependency is 'derelict-pq' but it could be anything (I tried 'dpq2' and even 'mysql-native', just for kicks). The segfault occurs when it's supposedly searching for the dependency. Here's the partial output of 'dub --vverbose build' Collecting dependencies for select Version selection for dependency derelict-pq (derelict-pq) of select is missing. Missing dependency derelict-pq >=3.0.0-beta.3 <3.1.0-0 of select Checking for missing dependencies. Search for versions of derelict-pq (1 package suppliers) Segmentation fault The dub.json is trivial and works to properly build a program with 'libpq' directly. And two months ago I had built the derelict-pq sample program and AFAIR everything went fine. When this first happened last night, code.dlang.org was temporarily not accessible but that's been resolved now. Before I report this as an issue, I thought I'd get some feedback. I'm running dub 1.7.0-2, built on Jan 3 2018, on Debian buster.
Understanding the AST...
Hello everybody! Last week end I found this post ( https://dlang.org/blog/2017/08/01/a-dub-case-study-compiling-dmd-as-a-library/ ) on the Blog and thought to myself awesome. So I built the library and everything went smooth. Thanks for the effort of all the involved people who made that possible! I've had a look at the 2 examples, too, the avg. function lines ( https://github.com/dlang/dmd/blob/master/src/examples/avg.d ) and the import ( https://github.com/dlang/dmd/blob/master/src/examples/impvisitor.d ) ones and for a start I decided to make a program that prints the outline of a module. Turns out I don't really understand how to access the data in the AST. For everything there's a visitor method and overriding a few of them to print return statements and some such works as advertised. However, I have no idea where I am in the tree when any of those methods are called. Like for example in FunctionLengthVisitor(AST).visitFuncBody(AST.FuncDeclaration fd). I have a function declaration object which tells me everything about what's inside the function, but how do I know what or where this function belongs to, where can I get that information ? I don't see anything about UDAs either, nor the doc comment. I understand when visitor.getAvgLen is called with the parsed module, the accept function calls a visitor overload for each member. But this sounds to me like I'd have to do a lot of book keeping in my visitor to keep track of things which are already present in the AST. Any insight to this would be much appreciated :)
Further questions on interfacing to Postgres libpq
An example test program that I'm using to learn D to C interfacing (specifically calling the libpq library) has a call to a C function declared as follows: void PQprint(FILE *fout, /* output stream */ const PGresult *res, const PQprintOpt *po); PQprintOpt is a struct whose first six members are declared as 'pqbool' which is in turn declared as "typedef char pqbool;" in the distributed Postgres header file. I've defined an "alias pqbool = char;" in the D file, which is pretty straightforward. The second of the six members has the name "align", which is a D keyword. So I renamed it "align_" and I presume that won't cause any problems. To deal with the first argument to PQprint, I added "import core.stdc.stdio : FILE;". The question is how to pass the D "stdout" as that argument. The D compiler tells me I can't pass it as is (as was done in C), because in D it's of type "File".
Re: Using Postgres connection functions
On Saturday, 20 January 2018 at 04:54:47 UTC, Adam D. Ruppe wrote: Same as above. The general pattern is: C_Type[] name = new C_Type[](requested_size); // pass as `name.ptr`. This becomes a C_Type* Thanks, Adam. Perhaps something like this ought to make its way into the "D for C Programmers" page.
Re: Using Postgres connection functions
On Saturday, 13 January 2018 at 05:28:17 UTC, Joe wrote: Going beyond the connection, there are various other libpq functions that use a similar pattern of values passed using multiple parallel C arrays, e.g., PGresult *PQexecParams(PGconn *conn, const char *command, int nParams, const Oid *paramTypes, const char * const *paramValues, const int *paramLengths, const int *paramFormats, int resultFormat); Each of the `paramXxxxs' arguments are arrays (Oid is an alias for uint). [...] Focusing on the above function, suppose the first two parameter arrays are defined in a C program as: Oid paramTypes[] = {23, 25}; char *paramValues[] = {"1", "abcde"}; which could be expressed in D as: Oid [] paramTypes = [23, 25]; string [] paramValues = ["1", "abcde"]; I know the paramTypes could be passed as null, letting PG deduce the data types but suppose we have some reason to accumulate types, etc., in D slices. I know the paramValues can be manipulated in a manner similar to the one shown in my first post, namely something like this: extern(C) char * [2] pvs; foreach (i, val; paramValues) pvs[i] = cast(char *)toStringz(val); And then use "cast(const char **)pvs" for the paramValues argument. The only feedback that I received that was explicit to this approach was in response to my initial post, in which Adam D. Ruppe said that what I did was "decent". So I have two lingering doubts: 1. Is malloc() the only way to allocate the arrays, either of Oid's, int's or char *'s, for passing to the libpq functions? IOW, isn't there a way to take advantage of D's 'new' (and thus the GC)? 2. How to convert a slice of Oid's or int's to an array of pointers suitable by processing by the libpq C function? A technique similar to the previous one seems to work, e.g., extern(C) Oid [2] pts; foreach (i, typ; paramTypes) pts[i] = typ; But I'm not sure if it's really working (when I mistakenly had a * in the pts declaration, at one point it also seemed to work).
Re: Using Postgres connection functions
On Monday, 15 January 2018 at 02:28:29 UTC, Matthias Klumpp wrote: In any case, please don't start another Postgres library and consider contributing to one of the existing ones, so that we maybe have one really awesome, 100% complete library at some point. If, on the other hand, your goal is to learn about the low-level Postgres interface and not just to have a Postgres interface for an application you develop, by all means, play with it :-) At this point, I am indeed learning about low-level Postgres interfaces (but not so low-level as the client-server protocol) as a way to understand the challenges of interfacing D to C. However, as part of the Pyrseas open source project, which I maintain, I had started to create a Postgres interface in Python inspired by The Third Manifesto, as opposed to ORMs like SQLAlchemy (see https://pyrseas.wordpress.com/2013/03/07/a-pythonic-ttm-inspired-interface-to-postgresql-requirements/). I got criticized for "reinventing the wheel" but I believe TTM, if properly done, is quite different from an ORM. I understand your concern about not starting another PG library. From the cursory investigation of the existing libraries, I think they span a spectrum, with ddb at one end (low-level protocol), then derelict-pq (low-level binding over libpq), ddbc at the opposite end (multi-DBMS support) and several others in between. So I guess the real problem is with the proliferation in the latter group.
Re: Using Postgres connection functions
On Saturday, 13 January 2018 at 10:10:41 UTC, Jacob Carlborg wrote: There's a native D library, ddb [1], for connecting to Postgres. Then you don't have to worry about null-terminated strings. There are several D libraries that I would consider "native": derelict-pq, dpq, dpq2 and ddb. The latter perhaps has the distinction that it doesn't use libpq, but rather implements the Postgres FE/BE protocol. That's a bit *too* native for my taste. It means the library maintainer has to keep up with changes to the internal protocol, which although published, the Postgres group doesn't have to maintain compatibility from version to version. For example, they haven't dropped the PQsetdbLogin function even though the PQconnectdb and PQconnectdbParams functions are obviously preferred. OTOH, there used to be an AsciiRow message format in the protocol, that was dropped, unceremoniously (not even mentioned in the release notes).
Re: Using Postgres connection functions
Going beyond the connection, there are various other libpq functions that use a similar pattern of values passed using multiple parallel C arrays, e.g., PGresult *PQexecParams(PGconn *conn, const char *command, int nParams, const Oid *paramTypes, const char * const *paramValues, const int *paramLengths, const int *paramFormats, int resultFormat); Each of the `paramXxxxs' arguments are arrays (Oid is an alias for uint). PGresult *PQprepare(PGconn *conn, const char *stmtName, const char *query, int nParams, const Oid *paramTypes); PGresult *PQexecPrepared(PGconn *conn, const char *stmtName, int nParams, const char * const *paramValues, const int *paramLengths, const int *paramFormats, int resultFormat); My point is that there seems to be a need to have a generic or generalized mechanism for passing these argument arrays from D to C.
Re: Using Postgres connection functions
On Saturday, 13 January 2018 at 04:26:06 UTC, Adam D. Ruppe wrote: If and only if the values are known at compile time, you can do: const char** keywords = ["hostaddr".ptr, "port".ptr, "dbname".ptr, null].ptr; or even do it inline: PQconnectdbParams(["hostaddr".ptr, "port".ptr, "dbname".ptr, null].ptr, ditto_for_values, 1); The keywords are (or could be) known at compile time, but almost by definition, the associated values are only known at runtime.
Using Postgres connection functions
I'm trying to learn how to use D to connect (and send queries) to Postgres, i.e., libpq in C. Postgres has three families of connection functions: PQsetdbLogin which takes multiple individual arguments (all as const char *), PQconnectdb which takes a single connection string (which Postgres then has to parse into keywords/values) and PQconnectdbParams (introduced with PG 9.0 in 2010) which takes two arrays of char pointers, one for the keywords and another for the values, i.e., it's pre-parsed, and I believe, more convenient for callers since it's most likely that they get the parameter values from command line arguments, environment values or UI dialogs, so it saves the connection string formatting on the caller side and the parsing/decomposing on the library side. In spite of the latter considerations, it appears most D libraries for Postgres only support connection string usage (only derelict-pq has a declaration for PQconnectdbParams--but I haven't tested it). In any case, I tried to use PQconnectdbParams by first declaring it, as per the Postgres manual, with extern(C) { struct PGconn; PGconn *PQconnectdbParams(const char * const * keywords, const char * const * values, int expand_dbname); } This caused ldc2, on Linux, to complain as follows: connection.d(30): Error: found 'const' when expecting ')' connection.d(30): Error: semicolon expected following function declaration connection.d(30): Error: declaration expected, not '*' It only compiled after I removed the second 'const' in the first and second arguments. The hard thing was figuring out how to pass the keyword/values arrays, defined as: string[] keywords = ["hostaddr", "port", "dbname"]; string[] values = ["127.0.0.1", "5432", "testdb"]; to the D invocation of PQconnectdbParams. I ended up with the following, which looks like covering a patient with lots of bandaids and looking the other way (and I had to modify the arrays above with a fixed length). extern(C) char * [keywords.length + 1] kws; extern(C) char * [keywords.length + 1] vals; foreach (i, k; keywords) { kws[i] = cast(char *)toStringz(k); vals[i] = cast(char *)toStringz(values[i]); } kws[keywords.length] = null; // Postgres wants the NULL termination vals[keywords.length] = null; conn = PQconnectdbParams(cast(const char **)kws, cast(const char **)vals, 1); I assume that in a real program I'd have to malloc the C arrays, since I wouldn't know beforehand how many parameters there would be (or I could define them with a static size of about 30 since that's how many connection parameters are recognized currently). So my question is: is there an easier or better way of passing two arrays of C null-terminated strings to an extern(C) function?