Re: Global thread-local free-list allocator
On Thursday, 16 March 2023 at 20:53:28 UTC, Witold wrote:
It crashes at `new_.value`, as `new_` is `null`
It looks like this:
```d
alias VariableAllocator = ThreadLocal!(FreeList!(Mallocator,
Variable.sizeof, unbounded));
```
should be this instead:
```d
alias VariableAllocator = ThreadLocal!(FreeList!(Mallocator,
Variable.sizeof, Variable.sizeof));
```
or
```d
alias VariableAllocator = ThreadLocal!(FreeList!(Mallocator, 0,
Variable.sizeof));
```
Otherwise if the list is empty (which it is at the start), the
free list will attempt to allocate from `Mallocator.instance`,
but not at requested size but rather at `maxSize` of the free
list allocator (that makes sense), which is `unbound`, which is
`size_t.max`, or about 18446 PiB. `malloc` will return null for
such crazy amount.
Also using `Mallocator` as a parent might not be a good idea.
The reason is that while it will allocate memory, it will not
tell GC to scan that area for pointers to other structs. So if
some objects are allocated using normal GC allocator, and only
reference is from Mallocator/FreeList allocated struct, GC will
think this object is unreachable, and clean it up.
Using `GCAllocator` works. I made this handy alias template:
```d
alias ListAllocator(Type) = ThreadLocal!(FreeList!(GCAllocator,
Type.sizeof, Type.sizeof));
```
Then just in a struct:
```d
struct MyStruct(T) {
alias allocator = ListAllocator!(MyStruct!T); // or
typeof(this)
...
auto m = allocator.instance.make!(MyStruct!T)();
...
allocator.instance.dispose(m);
}
Any ways to get addr2line working with DMD? works fine with GDC
Hello,
I'm trying to catch segfaults, after reading lot of ressources
online, i came up with this:
```D
import core.stdc.signal: SIGSEGV, SIGFPE, SIGILL, SIGABRT, signal;
import core.stdc.stdlib: free;
import core.stdc.string: strlen;
import core.sys.posix.unistd: STDERR_FILENO, readlink;
import core.sys.posix.signal: SIGUSR1;
import core.sys.posix.stdio: popen, pclose;
import core.sys.linux.execinfo: backtrace, backtrace_symbols;
import core.stdc.stdlib;
import core.stdc.stdio;
import core.stdc.string;
extern (C) void main()
{
signal(SIGSEGV, );
signal(SIGUSR1, );
test();
}
void test()
{
int* a;
*a = 1;
}
extern (C) void handler(int sig) nothrow @nogc
{
enum STACK_HIST = 6;
void*[STACK_HIST] array;
int size;
string signal_string;
switch (sig)
{
case SIGSEGV:
signal_string = "SIGSEGV";
break;
case SIGFPE:
signal_string = "SIGFPE";
break;
case SIGILL:
signal_string = "SIGILL";
break;
case SIGABRT:
signal_string = "SIGABRT";
break;
default:
signal_string = "unknown";
break;
}
import core.stdc.stdio : fprintf, stderr;
fprintf(stderr,
"---+\r\n");
fprintf(stderr, "Received signal %s (%d)\r\n",
signal_string.ptr, sig);
fprintf(stderr,
"---+\r\n");
// get void*'s for all entries on the stack
size = backtrace([0], STACK_HIST);
char** strings = backtrace_symbols([0], size);
enum BUF_SIZE = 1024;
char[BUF_SIZE] syscom;
char[BUF_SIZE] my_exe;
ulong path_size = readlink("/proc/self/exe", _exe[0],
BUF_SIZE);
my_exe[path_size] = 0;
printf("executable: %s\n", _exe[0]);
printf("backtrace: %i\n", size);
for (auto i = 2; i < size; ++i)
{
auto line = strings[i];
auto len = strlen(line);
bool insideParenthesis;
int startParenthesis;
int endParenthesis;
bool insideShit;
int startShit;
int endShit;
for(int j = 0; j < len; j++)
{
// ()
if (!insideParenthesis && line[j] == '(')
{
insideParenthesis = true;
startParenthesis = j+1;
}
else if (insideParenthesis && line[j] == ')')
{
insideParenthesis = false;
endParenthesis = j;
}
// []
if (!insideShit && line[j] == '[')
{
insideShit = true;
startShit = j+1;
}
else if (insideShit && line[j] == ']')
{
insideShit = false;
endShit = j;
}
}
// printf("%.*s :: %.*s\n", (endParenthesis -
startParenthesis), [startParenthesis], (endShit -
startShit), [startShit]);
// char[256] addr = 0;
// memcpy(addr.ptr, [startShit],
(endShit-startShit));
sprintf([0], "addr2line %s -e %s | ddemangle",
array[i], _exe[0]);
system([0]);
printf(" >%s\n", strings[i]);
//sprintf([0], "addr2line %p -f -e %s |
ddemangle", strings[i], _exe[0]);
//system([0]);
}
exit(-1);
}
```
The problem is when i compile with DMD i get:
```
??:0
```
When i compile it with GDC i get proper file + line
What does GDC do different than DMD?
Is there a way to get the same result as GDC with DMD?
Thanks
Re: read/peek and automatically advance index in buffer
On Thursday, 16 March 2023 at 18:58:18 UTC, ag0aep6g wrote: On Thursday, 16 March 2023 at 18:39:00 UTC, jwatson-CO-edu wrote: ```d int rtnVal = buffer.peek(int,Endian.bigEndian)(); // Error: found `,` when expecting `.` following int ``` You just forgot the exclamation mark there. "there" was not descriptive in this context, but I was able to infer what you meant by trial and error. The following gives me the behavior I needed: ```d int rtnVal = buffer.peek!(int, Endian.bigEndian)(); ```
Global thread-local free-list allocator
I ported a classic DeltaBlue benchmark to D, and for allocations
I use a mix of malloc/free (for structs with some trailing inline
arrays), and new for some other structs and arrays.
After running it, it felt kind of slow (I did not compare it to
any other language yet, so no idea if it is actually slower, just
feels slow), so I did some profiling (perf / hotspot mostly), and
noticed that substantial amount of time is spent in gc (both
allocation and collection). This is on Linux, and benchmark
indeed creates a lot of garbage, as I rerun and rerun it in a
loop.
In the past I used custom per-type allocator using D1 style
allocator methods, mostly together with hand coded free lists
(was working fine and was short).
Now I tried to use new `std.experimental.allocator`, and have
trouble understanding how to use them.
```d
import std.experimental.allocator : allocatorObject, ThreadLocal;
import std.experimental.allocator.building_blocks.free_list :
FreeList;
import std.experimental.allocator.gc_allocator : GCAllocator;
import std.experimental.allocator.mallocator : Mallocator;
import std.experimental.allocator.common : unbounded;
// static auto variableAllocator =
allocatorObject(FreeList!(GCAllocator, Variable.sizeof)());
// static auto variableAllocator =
allocatorObject(FreeList!(GCAllocator, Variable.sizeof,
unbounded)());
// static auto variableAllocator =
allocatorObject(FreeList!(Mallocator, Variable.sizeof,
unbounded)());
alias VariableAllocator = ThreadLocal!(FreeList!(Mallocator,
Variable.sizeof, unbounded));
// VariableAllocator variableAllocator;
// this() {
// variableAllocator = VariableAllocator.instance;
//}
struct Variable {
long value;
List!Constraint* constraints;
Constraint* determinedBy;
long mark;
Strength walkStrength;
bool stay;
// char[10] name;
string name;
static Variable* Create(string name, long initialValue) {
// auto new_ = new Variable();
auto new_ = VariableAllocator.instance.make!Variable();
// auto new_ = variableAllocator.make!Variable();
new_.value = initialValue; // CRASH
...
return new_;
}
static void Destroy(Variable* v) {
assert(v.constraints !is null, "bad Variable struct; already
freed?");
List!Constraint.Destroy(v.constraints);
v.constraints = null;
// free(v);
// destroy(v);
VariableAllocator.instance.dispose(v);
}
...
...
}
```
It crashes at `new_.value`, as `new_` is `null`
```
Reading symbols from ./t1_gdc_debug...
(gdb) r
Starting program: /home/user/benchy/t1_gdc_debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Program received signal SIGSEGV, Segmentation fault.
0x5557c86a in
deltablue.Variable.Create(immutable(char)[], long) (name=...,
initialValue=0) at /home/user/benchy/deltablue.d:248
248 new_.value = initialValue;
(gdb) bt
#0 0x5557c86a in
deltablue.Variable.Create(immutable(char)[], long) (name=...,
initialValue=0) at /home/user/benchy/deltablue.d:248
#1 0x5557f14e in deltablue.ChainTest(int) (n=100) at
/home/user/benchy/deltablue.d:901
#2 0x5557f88c in deltablue.deltablue(out ulong, out
ulong) (iters=@0x7fffd9b8: 0, flops=@0x7fffd9c0: 0)
at /home/user/benchy/deltablue.d:999
#3 0x555e2b3e in b (__capture=0x7fffdba0, name=...,
expectedResult=0) at /home/user/benchy/main.d:60
#4 0x555e2417 in D main (args=...) at
/home/user/benchy/main.d:91
(gdb) q
```
I want to use FreeList with parent allocator being Mallocator.
Never return allocations back to Mallocator, and for each
instance to be thread-local, AND for accessing thread-local
instance to not require ANY locks.
Documentation for `std.experimental.allocator` is kind of vague,
complicated and not very prescriptive.
Help would be appreciated.
Re: read/peek and automatically advance index in buffer
On Thursday, 16 March 2023 at 18:39:00 UTC, jwatson-CO-edu wrote: ```d int rtnVal = buffer.peek(int,Endian.bigEndian)(); // Error: found `,` when expecting `.` following int ``` You just forgot the exclamation mark there.
read/peek and automatically advance index in buffer
I read a file into a `ubyte` buffer as shown: ```d \\ ... buffer = cast(ubyte[]) read( fName ); // Read the entire file as bytestring marker = 0; // Current index to read from, managed manually, :( \\ ... ``` The data file contains numbers of varying size, and I want to read them sequentially. I gleaned the following from forum posts: ```d // Start read at marker, cast as int int rtnVal = peek!(int, Endian.bigEndian)(buffer[marker..$]); marker += 4; // I just peeked 32 bits ``` [The docs imply](https://dlang.org/library/std/bitmanip/peek.html) that I can peek and also advance my index appropriately, but I'm unsure how to specify the return type, endianness, and index pointer all at once. The following does not work: ```d int rtnVal = buffer.peek(int,Endian.bigEndian)(); // Error: found `,` when expecting `.` following int ``` What is the idiom / function call that will automatically advance my `marker` index variable?
Re: Is comparison of shared data thread-safe?
On Thursday, 16 March 2023 at 12:32:34 UTC, Nick Treleaven wrote: With -preview=nosharedaccess, I get: int y = 2; shared int x = y; // OK assert(x == 2); // no error y = x; // error This also does not error: ```d bool b = x == 3; ``` Filed: https://issues.dlang.org/show_bug.cgi?id=23783
Is comparison of shared data thread-safe?
With -preview=nosharedaccess, I get: int y = 2; shared int x = y; // OK assert(x == 2); // no error y = x; // error So for the assignment to y, reading x is an error and atomicLoad should be used instead. But is it an oversight that reading x in the assert is not an error? I have also found this in a unittest in core.atomic: shared(size_t) i; atomicOp!"+="(i, cast(size_t) 1); assert(i == 1); Is the assert somehow thread-safe?
Re: evenChunks on a string - hasLength constraint fails?
On Tuesday, 14 March 2023 at 18:41:50 UTC, Paul Backus wrote: On Tuesday, 14 March 2023 at 08:21:00 UTC, amarillion wrote: I'm trying to understand why this doesn't work. I don't really understand the error. If I interpret this correctly, it's missing a length attribute on a string, but shouldn't length be there? By default, D's standard library treats a `string` as a range of Unicode code points (i.e., a range of `dchar`s), encoded in UTF-8. Because UTF-8 is a variable-length encoding, it's impossible to know how many code points there are in a `string` without iterating it--which means that, as far as the standard library is concerned, `string` does not have a valid `.length` property. Thanks for the clear explanation! I was already aware that you could iterate by codepoint with foreach(dchar c; s), but it just didn't cross my mind that the same concept was playing a role here. I guess it's just one of those things that you just have to know. regards, Amarillion
