Warning The package will no longer be detected starting from v1.42.0
Most of my homegrown libraries are private and are used locally for the most part. They are in ~/Projects/D/libs . Until now I've always used dub add-path and things worked fine. Updated my install am now getting this message: ``` Warning Package at path '/home/soulsbane/Projects/D/libs/textrecords/' should be under '/home/soulsbane/Projects/D/libs/textrecords/$VERSION/textrecords' Warning The package will no longer be detected starting from v1.42.0 ``` I'm guessing it's caused by this https://github.com/dlang/dub/pull/2610. What's the fix for this exactly? Thanks!
Re: Mixin and compile-time functions for code generation
On Saturday, 24 June 2023 at 17:43:52 UTC, Adam D Ruppe wrote: On Saturday, 24 June 2023 at 17:31:31 UTC, Cecil Ward wrote: Can I get mixin whatever to do this for me? Mixin with a function that runs at compile-time and creates the required source ? have you tried it? No, not so far. Adam, if you think that is a workable approach then I certainly will do so. It’s going to be very awkward generating literal strings in D code using literal strings in the meta-mixin-code or whatever I should call it.
Re: How to setup dub project for contributing to a dub package?
On Friday, 23 June 2023 at 15:22:33 UTC, Ki Rill wrote:
Recently, I tried to set up `dcv` with `dub` to improve a few
things in the library, but I faced some strange issues.
[...]
I recommend adding DCV sub packages separately. Don't add the
entire thing to your dub dependencies. Just only add what you
need. Ffmpeg-d in only needed for DCV videoing. You can add
subpacks like:
"dependencies": {
"dcv:core": ...,
"dcv:plot": {"path": "../../"},
"dcv:imageio": {"path": "../../"},
"mir-random": "*"
}
Re: How to setup dub project for contributing to a dub package?
On Friday, 23 June 2023 at 15:52:44 UTC, Richard (Rikki) Andrew
Cattermole wrote:
First things first, dcv is added to the dub-registry, so use
this.
https://code.dlang.org/packages/dcv
```json
"dependencies": {
"dcv": "~>0.3.0"
}
```
For ffmpeg the binding tells you what to add for search paths
in the lflags directive.
https://github.com/ljubobratovicrelja/ffmpeg-d#adding-to-dub
All I can suggest is make sure you have the right dev packages
for ffmpeg installed and findable by the linker.
Just to make things clear,
I don't recommend to use DCV from dub registry, it's outdated. I
have an opinion that the docs should be updated before creating a
new version updating the dub repo. There are so many API changes
between the last dub repo version and the master repo. I don't
know if I will find time and motivation anytime soon. I doubt
someone else will do it too (it is a huge welcome though).
Re: Mixin and compile-time functions for code generation
On Saturday, 24 June 2023 at 17:31:31 UTC, Cecil Ward wrote: Can I get mixin whatever to do this for me? Mixin with a function that runs at compile-time and creates the required source ? have you tried it?
Mixin and compile-time functions for code generation
I have a function that can be run at compile-time and which will be able to output code to be injected into the D source code stream. Can I get mixin whatever to do this for me? Mixin with a function that runs at compile-time and creates the required source ? Like D’s solution for a replacement for function-style C preprocessor macros ? - but far more advanced and capable ? I need to re-read Ali Çehreli’s excellent book for the third time.
Re: How does D’s ‘import’ work?
On Tuesday, 20 June 2023 at 17:56:27 UTC, Ali Çehreli wrote: On 6/20/23 08:09, Cecil Ward wrote: > I’m used to slow compilers on fast machines and compiling > gives me an excuse for more coffee and possibly fruity buns. Yes, all of us in past projects accepted C++'s slowness. We did get coffee, etc. One of my current colleagues regularly plays solitaire when waiting for C++ compilations. Not only it's not a professional sight, but C++ is proving to be a professional mistake. Nobody should suffer from such context switches. I have a hunch, without any backing research data, that C++'s contribution to humanity may be net negative. D is nothing like that: My turnaround is a few seconds: Write, compile, run, see the effect... I use only dmd partly because of laziness: it just works. Although I take full advantage D's low level powers, my programs have mostly been I/O bound with huge files, so dmd's less-than ideal optimization powers are hidden because most threads are waiting for file system I/O. Aside: std.parallelism and std.concurrency have been very helpful. Ali In the 1980s on our VAX 11/750, compile jobs were batch jobs placed in a queue. Half hour waits were not unknown. A build of the new o/s we were working on took around 40 mins on a 33 MHz 386 Dell PC (later a 486!) iirc. So time for patisserie even. But in oractice you simply got on with other jobs, like writing new code that was not yet integrated, code reviews, all sorts of things.
Compiling the runtime library
Is it possible to recompile the LDC and GDC runtimes yourself so you can do so with the switches you desire? (eg regarding optimisation, release vs debug build modes.) I think I saw a mention of something to help with this in the LDC docs, GDC would be a different story though. I’d have to get hold of the code first somehow, of course.
Unused routines and inlining
(Apologies if I have talked about this before, but my memory is shot because of strong pain drugs that I’m on, and it never was any good before either, so I may be repeating myself.) I’m using GDC and LDC, comparing the two, and in a medium sized routine that I have written pretty much every routine is inlined all the time. No-one takes the addresses of the routines, and they are not called externally to the module, marked private. (Is that the same as ‘static’ in C and D ?) So there’s no reason for the compiled copies of the function bodies to still exist. This makes the module unnecessarily huge with all this unused code just sitting there. Is there anything I can be doing about this, to make them go away, with appropriate configuration? It’s a wishlist item for the compilers, to check for zero-usage-count in functions that are always inlined, private and where there’s no pointer-taking so no chance of indirect calls. Is that sufficient?
Re: Unused routines and inlining
On Saturday, 24 June 2023 at 16:55:05 UTC, Cecil Ward wrote: (Apologies if I have talked about this before, but my memory is shot because of strong pain drugs that I’m on, and it never was any good before either, so I may be repeating myself.) [...] s/medium sized routine/medium-sized module/
Re: A couple of questions about arrays and slices
On Saturday, 24 June 2023 at 16:42:45 UTC, Cecil Ward wrote: On Saturday, 24 June 2023 at 15:12:14 UTC, Jonathan M Davis wrote: [...] Yeah, it would take me forever to get my head around that, and I only want a crude toy partial parser for certain portions of the grammar, and the parsing code is done now. A hand-written recursive descent type thing mainly dealing with things like comments and literal string that have to be taken account of as they prevent hazards to naive straight string searching for what you want to find, as comments and eg double-quoted strings could have things in them that are red-herrings or the things that you want to find items in, depending on circumstances. [...] I read an article about just that good strings trick many many years back, and the author called it ‘a string universe’, which I really liked.
Re: A couple of questions about arrays and slices
On Saturday, 24 June 2023 at 15:12:14 UTC, Jonathan M Davis wrote: On Saturday, June 24, 2023 8:43:00 AM MDT Cecil Ward via Digitalmars-d-learn wrote: I started out looking into a number of runtime library routines, but in the end it seemed quicker to roll my own code for a crude recursive descent parser/lexer that parses part of D’s grammar for expressions, and (again partial grammar) parser for string literal expressions and so on. I find certain special elements and execute actions which involve doing the AA lookup and replacing variable names with ordinal numbers in decimal in the output stream. Admission: The parsing is the thing that has to be fast, even though again the size of the D language text is not likely to be huge at all. But 40 years ago, I came from a world with 2k RAM and 0.9 MHz clock rates so I have developed a habit of always thinking about speed before I do anything, needful or not, to be honest. I once wrote a program that took 35 mins to evaluate 2+2 and print out the answer, so I’m now ashamed of writing slow code. Those were bad days, to be honest. 4 GHz+ and ILP is nicer. Well, dmd is open source (and Boost-licensed, so it doesn't really have any restrictions), so depending on what you're doing, it might make sense to just take code from that (and it's very fast). IIRC, it pulls some fun tricks like replacing identical strings with pointers to the same string so that it can just compare pointers. - Jonathan M Davis Yeah, it would take me forever to get my head around that, and I only want a crude toy partial parser for certain portions of the grammar, and the parsing code is done now. A hand-written recursive descent type thing mainly dealing with things like comments and literal string that have to be taken account of as they prevent hazards to naive straight string searching for what you want to find, as comments and eg double-quoted strings could have things in them that are red-herrings or the things that you want to find items in, depending on circumstances. I’m trying to get my head round the differences between OSX tools and those for Linux relating to LDC and GDC which seems slightly inferior in some situations. I’m a serious professional asm programmer of old, before compilers were of usable output quality for git-hard applications. (‘a git’ a disreputable person, colloquial English English. ‘git hard’ - brain-meltingly hard, like quantum gravity.)
Re: Toolchain with ldc and AArch64 OSX
On Saturday, 24 June 2023 at 15:16:37 UTC, Cecil Ward wrote: I have LDC running on an ARM Mac. If anyone else out there is an LDC or GDC user, could you knock up a quick shell program to compile and link a .d file to produce an executable ? found the linker but these tools are all new to me and a bit of help would save me a lot of trial and error and frustration as I try to find docs. GDC would be great too. I have managed to achieve this before on a Raspberry Pi AArch64 Linux Debian where the compiler can link and generate an executable just in integrated fashion in the one command. The OSX tools seem rather different however. I’m going to try installing GDC on the Mac next, have got that running on the Pi too successfully. I have ldc installed (from `brew`) on my (also arm) Mac, it works fine, or do you specifically want to work out which linker to invoke manually and so on? I'm not sure if gdc is currently easy to obtain on arm macs. I think it should work fine but some packages hadn't enabled arm support on macos yet, last time *I* checked at least.
Toolchain with ldc and AArch64 OSX
I have LDC running on an ARM Mac. If anyone else out there is an LDC or GDC user, could you knock up a quick shell program to compile and link a .d file to produce an executable ? found the linker but these tools are all new to me and a bit of help would save me a lot of trial and error and frustration as I try to find docs. GDC would be great too. I have managed to achieve this before on a Raspberry Pi AArch64 Linux Debian where the compiler can link and generate an executable just in integrated fashion in the one command. The OSX tools seem rather different however. I’m going to try installing GDC on the Mac next, have got that running on the Pi too successfully.
Re: A couple of questions about arrays and slices
On Saturday, June 24, 2023 8:43:00 AM MDT Cecil Ward via Digitalmars-d-learn wrote: > I started out looking into a number of runtime library routines, > but in the end it seemed quicker to roll my own code for a crude > recursive descent parser/lexer that parses part of D’s grammar > for expressions, and (again partial grammar) parser for string > literal expressions and so on. I find certain special elements > and execute actions which involve doing the AA lookup and > replacing variable names with ordinal numbers in decimal in the > output stream. Admission: The parsing is the thing that has to be > fast, even though again the size of the D language text is not > likely to be huge at all. But 40 years ago, I came from a world > with 2k RAM and 0.9 MHz clock rates so I have developed a habit > of always thinking about speed before I do anything, needful or > not, to be honest. I once wrote a program that took 35 mins to > evaluate 2+2 and print out the answer, so I’m now ashamed of > writing slow code. Those were bad days, to be honest. 4 GHz+ and > ILP is nicer. Well, dmd is open source (and Boost-licensed, so it doesn't really have any restrictions), so depending on what you're doing, it might make sense to just take code from that (and it's very fast). IIRC, it pulls some fun tricks like replacing identical strings with pointers to the same string so that it can just compare pointers. - Jonathan M Davis
Re: A couple of questions about arrays and slices
On Saturday, 24 June 2023 at 12:05:26 UTC, Jonathan M Davis wrote: On Saturday, June 24, 2023 1:43:53 AM MDT Cecil Ward via Digitalmars-d-learn wrote: On Saturday, 24 June 2023 at 07:36:26 UTC, Cecil Ward wrote: > [...] I just realised something, your point about altering the table and having to rehash, is well taken. I hadn’t considered that. The reason for my foolishness in failing to realise that I’m asking the impractical is my pattern of usage. I add all the entries into the mapping table and have no interest in any lookups until it is fully built. Then a second function starts to do lookups while the data remains unchanging and that usage pattern can be guaranteed. I could even idup it if that would help, as copying < 32 uints wouldn’t take forever. A typical value would be a mere 5 or less. I only picked 32 to be completely safely ott. Well, if the key were a struct or a class, the hashing function would be opHash. For built-in types, the runtime has hashing functions that it uses. Either way, with AAs, you really don't worry about managing the memory, because it's completely outside of your control. You just put the elements in there using their associated keys, and if you want to try to speed it up after you've populated it, you use rehash so that the runtime can try to move the elements around within the container so that lookup speeds will be closer to optimal. As such, for the most part, when dealing with AAs and worrying about efficiency, the question really becomes whether AAs are the correct solution rather than much of anything having to do with how you manage their memory. With so few elements, it's also possible that using std.algorithm.searching.find would be faster - e.g. having a dynamic array of strings where the matching int is at the same index in a dynamic array of ints - or you could use std.typecons.Tuple!(string, int)[] with something like arr.find!(a => a[0] == key)() to find the tuple with the int you want. Simply comparing a small number of strings like that might be faster than what goes on with hashing the string and then finding the corresponding element within the AA - or it might not be. You'd have to test that to know. The AA would definitely be faster with a large number of elements, but with a small number of elements, the algorithmic complexity doesn't really matter, and the extra overhad with the AA lookups could actually mean that the search through the dynamic array is faster even though it's O(n). But you can only know which is faster by testing it out with the actual data that you're dealing with. Regardless, you need to remember that associative arrays are not arrays in the C sense. Rather, they're hash tables, so they function very differently from dynamic arrays, and the rehash function is the closest that you're going to get to affecting how the elements are laid out internally or how much memory the AA is using. - Jonathan M Davis I started out looking into a number of runtime library routines, but in the end it seemed quicker to roll my own code for a crude recursive descent parser/lexer that parses part of D’s grammar for expressions, and (again partial grammar) parser for string literal expressions and so on. I find certain special elements and execute actions which involve doing the AA lookup and replacing variable names with ordinal numbers in decimal in the output stream. Admission: The parsing is the thing that has to be fast, even though again the size of the D language text is not likely to be huge at all. But 40 years ago, I came from a world with 2k RAM and 0.9 MHz clock rates so I have developed a habit of always thinking about speed before I do anything, needful or not, to be honest. I once wrote a program that took 35 mins to evaluate 2+2 and print out the answer, so I’m now ashamed of writing slow code. Those were bad days, to be honest. 4 GHz+ and ILP is nicer.
Re: A couple of questions about arrays and slices
On Saturday, June 24, 2023 1:43:53 AM MDT Cecil Ward via Digitalmars-d-learn wrote: > On Saturday, 24 June 2023 at 07:36:26 UTC, Cecil Ward wrote: > > Jonathan, is it possible that I wanted one thing and got > > another? My description in the earlier post was of the _aim_ of > > the program. What I ended up with might be something else? I > > wanted an array of uints whose values are the results/outputs > > of the mapping function. Since it is keyed by strings I assumed > > that the runtime generates some kind of hash for fast lookup > > when I ask it to retrieve an entry by the string (key) > > associated with it. I assumed that in some sense the hashing > > was sort of separate with some degree of independence from the > > underlying array, if that makes sense. The lookup is just > > assumed to be fast but how it is done we don’t really care. I > > just wanted to expand the array as I did successfully elsewhere > > with reserve, as I built this structure by successive additions > > of data. I have a number of strings and the map is meant to > > output the ordinal number in which I first saw them, > > zero-based. Then I want to come back and randomly look up one > > ordinal given a string preferably with a very fast lookup. The > > number of entries can not practically be more than 30, and even > > that would be highly unusual, maybe ten is the practical limit > > in my particular case, so it’s hardly MySQL. > > I just realised something, your point about altering the table > and having to rehash, is well taken. I hadn’t considered that. > The reason for my foolishness in failing to realise that I’m > asking the impractical is my pattern of usage. I add all the > entries into the mapping table and have no interest in any > lookups until it is fully built. Then a second function starts to > do lookups while the data remains unchanging and that usage > pattern can be guaranteed. I could even idup it if that would > help, as copying < 32 uints wouldn’t take forever. A typical > value would be a mere 5 or less. I only picked 32 to be > completely safely ott. Well, if the key were a struct or a class, the hashing function would be opHash. For built-in types, the runtime has hashing functions that it uses. Either way, with AAs, you really don't worry about managing the memory, because it's completely outside of your control. You just put the elements in there using their associated keys, and if you want to try to speed it up after you've populated it, you use rehash so that the runtime can try to move the elements around within the container so that lookup speeds will be closer to optimal. As such, for the most part, when dealing with AAs and worrying about efficiency, the question really becomes whether AAs are the correct solution rather than much of anything having to do with how you manage their memory. With so few elements, it's also possible that using std.algorithm.searching.find would be faster - e.g. having a dynamic array of strings where the matching int is at the same index in a dynamic array of ints - or you could use std.typecons.Tuple!(string, int)[] with something like arr.find!(a => a[0] == key)() to find the tuple with the int you want. Simply comparing a small number of strings like that might be faster than what goes on with hashing the string and then finding the corresponding element within the AA - or it might not be. You'd have to test that to know. The AA would definitely be faster with a large number of elements, but with a small number of elements, the algorithmic complexity doesn't really matter, and the extra overhad with the AA lookups could actually mean that the search through the dynamic array is faster even though it's O(n). But you can only know which is faster by testing it out with the actual data that you're dealing with. Regardless, you need to remember that associative arrays are not arrays in the C sense. Rather, they're hash tables, so they function very differently from dynamic arrays, and the rehash function is the closest that you're going to get to affecting how the elements are laid out internally or how much memory the AA is using. - Jonathan M Davis
Issues in the gprof output report
Hello, I have followed these steps to generate a profiling data from gprof: 1) -pg flag added for both compiler and linker while compiling. Also added compiler flags like -fno-inline-functions, -fno-builtin, -no-pie as suggested in many online forums. 2) Executed the program and got the gmon.out file generated. 3) Using gprof to read the profiling data: ```gprof app_name gmon.out > analysis.txt``` Note: My embedded application is multithreaded running on Linux platform. It has infinite 'for' loop to keep the main thread alive. But I am limiting this 'for' loop to few thousands of iterations and returning from the main() function to get the gmon.out file generated. The below points made me doubt the correctness of the profiling data: *) main() function and its details is not shown in the gprof output file. *) There is a function that gets called inside the 'for' loop in my application which I know is taking a lot of time(as it is using ioctl() calls everytime and confirmed that it takes too much time with testing). But gprof output file shows that it is taking very less time to get executed. 1) Please let me know where I'm going wrong or should I do anything more to get correct profiling data from gprof. 2) The default sampling rate of gprof is 0.01 seconds. Is there a way to increase this sampling rate of gprof? I want to try by increasing the sampling rate because 0.01 seconds seems to be very less.
Re: A couple of questions about arrays and slices
On Saturday, 24 June 2023 at 07:36:26 UTC, Cecil Ward wrote: Jonathan, is it possible that I wanted one thing and got another? My description in the earlier post was of the _aim_ of the program. What I ended up with might be something else? I wanted an array of uints whose values are the results/outputs of the mapping function. Since it is keyed by strings I assumed that the runtime generates some kind of hash for fast lookup when I ask it to retrieve an entry by the string (key) associated with it. I assumed that in some sense the hashing was sort of separate with some degree of independence from the underlying array, if that makes sense. The lookup is just assumed to be fast but how it is done we don’t really care. I just wanted to expand the array as I did successfully elsewhere with reserve, as I built this structure by successive additions of data. I have a number of strings and the map is meant to output the ordinal number in which I first saw them, zero-based. Then I want to come back and randomly look up one ordinal given a string preferably with a very fast lookup. The number of entries can not practically be more than 30, and even that would be highly unusual, maybe ten is the practical limit in my particular case, so it’s hardly MySQL. I just realised something, your point about altering the table and having to rehash, is well taken. I hadn’t considered that. The reason for my foolishness in failing to realise that I’m asking the impractical is my pattern of usage. I add all the entries into the mapping table and have no interest in any lookups until it is fully built. Then a second function starts to do lookups while the data remains unchanging and that usage pattern can be guaranteed. I could even idup it if that would help, as copying < 32 uints wouldn’t take forever. A typical value would be a mere 5 or less. I only picked 32 to be completely safely ott.
Re: A couple of questions about arrays and slices
On Saturday, 24 June 2023 at 01:28:03 UTC, Jonathan M Davis wrote:
On Friday, June 23, 2023 7:02:12 PM MDT Cecil Ward via
Digitalmars-d-learn wrote:
I just had a fight with LDC over the following code when I
tried out reserve. I have an associative array that maps
strings to ‘ordinals’ ie uints that are unique, and the
compiler hates the call to reserve.
==
struct decls_t
{
uintn_entries = 0;
uint[ dstring ] ordinals; // Associative array maps
variable
names to ordinals
}
static decls_t Decls;
enum NPreAllocEntries = 32;
Decls.ordinals.reserve( NPreAllocEntries );
source>(82): Error: none of the overloads of template
`object.reserve` are callable using argument types
`!()(uint[dstring], ulong)`
/opt/compiler-explorer/ldc1.32.1/ldc2-1.32.1-linux-x86_64/bin/../import/obje
ct.d(3983):Candidate is: `reserve(T)(ref T[] arr,
size_t
newcapacity)` Compiler returned: 1
Associative arrays and dynamic arrays are completely different
things. Associative arrays are hash tables, and reserve really
doesn't make sense for them. reserve is for telling the GC to
make sure that a dynamic array has at least a specific amount
of room to grow into before the GC needs to do a reallocation
so that the dynamic array refers to a different memory block
with enough memory to hold the data, whereas if and when
associative arrays have to reallocate any of their internals is
largely implementation-defined.
Any time that you add or remove elements from an AA, it might
reallocate some of its internals depending on its current state
and what the key of the element is - and that could be
different between different compiler releases (though it's
unlikely to change very often, since I don't think that the AA
implementation gets messed with much).
You can use the rehash function on AAs to tell the GC to try to
reorder how it's structured all of its buckets so that lookups
are more efficient with the data that's currently in there, and
you can call clear to remove all its elements, but in general,
you don't do much to manage an AA's memory. It's a much more
complicated data structure than an array.
https://dlang.org/spec/hash-map.html
- Jonathan M Davis
Jonathan, is it possible that I wanted one thing and got another?
My description in the earlier post was of the _aim_ of the
program. What I ended up with might be something else? I wanted
an array of uints whose values are the results/outputs of the
mapping function. Since it is keyed by strings I assumed that the
runtime generates some kind of hash for fast lookup when I ask it
to retrieve an entry by the string (key) associated with it. I
assumed that in some sense the hashing was sort of separate with
some degree of independence from the underlying array, if that
makes sense. The lookup is just assumed to be fast but how it is
done we don’t really care. I just wanted to expand the array as I
did successfully elsewhere with reserve, as I built this
structure by successive additions of data. I have a number of
strings and the map is meant to output the ordinal number in
which I first saw them, zero-based. Then I want to come back and
randomly look up one ordinal given a string preferably with a
very fast lookup. The number of entries can not practically be
more than 30, and even that would be highly unusual, maybe ten is
the practical limit in my particular case, so it’s hardly MySQL.
Re: A couple of questions about arrays and slices
On 6/20/23 19:09, Cecil Ward wrote:
> 2.) I have a dynamic array and I wish to preinitialise its alloc cell to
> be a certain large size so that I don’t need to reallocate often
To be complete, 'assumeSafeAppend' must be mentioned here as well.
Without it, there will be cases where the GC cannot guarantee that there
are no slices to this particular one; so it has to reallocate:
import std;
void main() {
// An array with room for 100 elements
int[] arr;
arr.reserve(100);
// Take note of current address of the elements
auto ptr = arr.ptr;
foreach (i; 0 .. 80) {
// Add elements
arr ~= i;
// Was there a reallocation?
if (arr.ptr != ptr) {
writeln("relocated to ", arr.ptr, " at ", i);
ptr = arr.ptr;
}
// Let's say our algorithm shrinks the array
if (i == 50) {
arr.length = 0;
// assumeSafeAppend(arr);
}
}
}
Although the array has room for 100 elements, the program will print
something similar to the following:
relocated to 7F058B02B000 at 51
relocated to 7F058B02C000 at 54
relocated to 7F058B02D000 at 58
relocated to 7F058B02E000 at 62
relocated to 7F058B02F000 at 66
relocated to 7F058B03 at 74
When it's known that there is no other slice to the old elements, the
programmer calls assumeSafeAppend() by uncommenting that line :o). Now
there are no relocations. Sweet!
Ali
