Re: Are Fibers just broken in D?
On Tuesday, 24 April 2018 at 13:36:48 UTC, Steven Schveighoffer
wrote:
On 4/24/18 5:11 AM, bauss wrote:
On Tuesday, 24 April 2018 at 07:58:01 UTC, Radu wrote:
On Tuesday, 24 April 2018 at 00:46:39 UTC, Byron Heads wrote:
Fibers on Win32 have a memory leak for sure:
import core.thread : Fiber;
void main() {
foreach(ulong i; 0..99_999) {
auto foo = new Foo();
foo.call();
foo.call();
}
}
class Foo : Fiber {
this() {
super();
}
void run() {
Fiber.yield();
}
}
Running this with -m64 on windows runs without a problem,
but with -m32 it failes aith a Memory Allocation failed
error.
This is not a fiber issue but a more memory management issue.
Your run out of address space on win32, the GC will not
always collect all those 9 fibers that you allocate in
that loop. As an exercise replace `auto` with `scope` like
`scope foo = new Foo();` in that loop - you should see
different results.
This shouldn't be a requirement, the 32-bit GC is generally not
this bad.
The issue boils down to the call to VirtualAlloc that the
fiber constructor makes, which fails as Windows will not
allocate any more virtual pages for the process. If you
really want that many fibers you should reconsider 32 bit,
and definitely use a different allocation strategy.
And in the end of the day it makes no sense to have that many
fibers as they would probably perform terrible.
Let's not forget though, that he's executing the fiber
completely within the loop. It should be done and collected.
This is not the case of executing 100,000 concurrent fibers,
but executing 100,000 *sequential* fibers. It should work just
fine.
-Steve
Correct, in a normal run of my system there maybe 10-20 fibers
max alive. I was using threads only before, but found the system
to execute jobs in a balanced way. But using a few threads to
process Fibers keep in a queue balances out the work evenly. It
is also easier to track down bugs by just using 1 thread to
process the fiber pool.
I would love to use dip1000, Allocators, and shared. But none of
that stuff really works beyond trivial examples. (Allocators
probably works fine, but there are forum post about it changing
and I dont want to refactor it twice..)
I will start ignoring win32 when win64 doesn't require dealing
with visual studio installs.
Also I have a feeling a client will ask for it.
Re: Are Fibers just broken in D?
On Friday, 20 April 2018 at 20:52:17 UTC, Byron Moxie wrote:
On Friday, 20 April 2018 at 20:46:20 UTC, Steven Schveighoffer
wrote:
On 4/20/18 2:58 PM, Byron Moxie wrote:
[...]
It sounds like the problems may be due to Win32 and not the
other pieces. Have you tried on a Win64 build? Even if that's
not your target, at least it can help you discover whether
that is the problem or not. The DMC runtime is the default on
Win32, and it's not especially thread-safe in all places.
FWIW, I'm using vibe.d on Linux 64 bit with no problems (and
I've NEVER heard of atomicLoad and atomicStore not working on
any arch).
-Steve
I had move the data I wanted to sync with atomicLoad/Store into
a shared struct and pass a pointer to this struct to the other
fibers. Not sure if this was an issue with TLS messing with
class object I was passing around.
Fibers on Win32 have a memory leak for sure:
import core.thread : Fiber;
void main() {
foreach(ulong i; 0..99_999) {
auto foo = new Foo();
foo.call();
foo.call();
}
}
class Foo : Fiber {
this() {
super();
}
void run() {
Fiber.yield();
}
}
Running this with -m64 on windows runs without a problem, but
with -m32 it failes aith a Memory Allocation failed error.
Re: Error 42: Symbol Undefined __lseeki64
On Thursday, 17 December 2015 at 04:11:56 UTC, tcak wrote: I searched the function "__lseek64" under /usr/include/dmd" with "grep -R __lseek64", but nothing is found. I work on Linux 64-bit. So, I guess it is either Windows related, or 32bit dmd related. "lseek64" is found in "unistd.d", but this doesn't solve any problem. After some testing it seems that -m32mscoff and -m64 both build correctly so its related to 32bit dmd on windows
Re: Error 42: Symbol Undefined __lseeki64
On Wednesday, 16 December 2015 at 18:21:33 UTC, Byron Heads wrote:
On Wednesday, 16 December 2015 at 18:14:35 UTC, Byron Heads
wrote:
On Wednesday, 16 December 2015 at 17:23:15 UTC, Byron Heads
wrote:
Seeing this issue on 2.069.2 using etc.c.zlib.
C:\d\dmd2\windows\bin\..\lib\phobos.lib(gzlib)
Error 42: Symbol Undefined __lseeki64
The code was compiling in 2.067. Not clear on where to look
to fix this issue.
I can reproduce with this code...
Windows dmd 2.069.2 32bit
import std.stream;
import std.exception;
// todo: add bzip support..
class GZipBufferedFile : BufferedFile {
private:
GZipFile gZipFile;
/**
* GZipFile stream, safly opens and closes a gzip file,
also will correctly read from the zip file
*/
class GZipFile : File {
import std.c.stdio;
import etc.c.zlib;
FILE* fp;
gzFile fpGZip;
/**
* Use gzopen to open the zip file
*/
this(string filename) {
fp = fopen(filename.toStringz, "rb");
enforce(fp !is null, "Failed to open file
'%s'".format(filename));
version(Windows) {
fpGZip = gzdopen(fp._file, "rb");
super(fpGZip, FileMode.In);
} else {
fpGZip = gzdopen(fileno(fp), "rb");
super(cast(int)fpGZip, FileMode.In);
}
seekable = true;
// Still not supported... sigh
//gzbuffer(fpGZip, 1024 * 1024);
}
ulong tell() {
fflush(fp);
return ftell(fp);
}
/**
* read data block with gzread
*/
override size_t readBlock(void* buffer, size_t size) {
assertReadable();
size = gzread(fpGZip, buffer, cast(uint)size);
if (size == -1) {
size = 0;
}
// use gzeof to test for end of file
readEOF = fpGZip.gzeof != 0;
return size;
}
/**
* make sure we close with gzclose
*/
override void close() {
gzclose(fpGZip);
fpGZip = null;
fp = null;
}
}
public:
this(string filename) {
gZipFile = new GZipFile(filename);
super(gZipFile);
}
ulong tell() {
return gZipFile.tell;
}
}
void main() {
}
Commenting out
gzclose(fpGZip);
allows it to compile..
Submitted reduced case as a bug:
https://issues.dlang.org/show_bug.cgi?id=15457
import etc.c.zlib;
void main() {
gzclose(null);
}
Error 42: Symbol Undefined __lseeki64
Seeing this issue on 2.069.2 using etc.c.zlib. C:\d\dmd2\windows\bin\..\lib\phobos.lib(gzlib) Error 42: Symbol Undefined __lseeki64 The code was compiling in 2.067. Not clear on where to look to fix this issue.
Re: Error 42: Symbol Undefined __lseeki64
On Wednesday, 16 December 2015 at 17:23:15 UTC, Byron Heads wrote:
Seeing this issue on 2.069.2 using etc.c.zlib.
C:\d\dmd2\windows\bin\..\lib\phobos.lib(gzlib)
Error 42: Symbol Undefined __lseeki64
The code was compiling in 2.067. Not clear on where to look to
fix this issue.
I can reproduce with this code...
Windows dmd 2.069.2 32bit
import std.stream;
import std.exception;
// todo: add bzip support..
class GZipBufferedFile : BufferedFile {
private:
GZipFile gZipFile;
/**
* GZipFile stream, safly opens and closes a gzip file, also
will correctly read from the zip file
*/
class GZipFile : File {
import std.c.stdio;
import etc.c.zlib;
FILE* fp;
gzFile fpGZip;
/**
* Use gzopen to open the zip file
*/
this(string filename) {
fp = fopen(filename.toStringz, "rb");
enforce(fp !is null, "Failed to open file
'%s'".format(filename));
version(Windows) {
fpGZip = gzdopen(fp._file, "rb");
super(fpGZip, FileMode.In);
} else {
fpGZip = gzdopen(fileno(fp), "rb");
super(cast(int)fpGZip, FileMode.In);
}
seekable = true;
// Still not supported... sigh
//gzbuffer(fpGZip, 1024 * 1024);
}
ulong tell() {
fflush(fp);
return ftell(fp);
}
/**
* read data block with gzread
*/
override size_t readBlock(void* buffer, size_t size) {
assertReadable();
size = gzread(fpGZip, buffer, cast(uint)size);
if (size == -1) {
size = 0;
}
// use gzeof to test for end of file
readEOF = fpGZip.gzeof != 0;
return size;
}
/**
* make sure we close with gzclose
*/
override void close() {
gzclose(fpGZip);
fpGZip = null;
fp = null;
}
}
public:
this(string filename) {
gZipFile = new GZipFile(filename);
super(gZipFile);
}
ulong tell() {
return gZipFile.tell;
}
}
void main() {
}
Re: Error 42: Symbol Undefined __lseeki64
On Wednesday, 16 December 2015 at 18:14:35 UTC, Byron Heads wrote:
On Wednesday, 16 December 2015 at 17:23:15 UTC, Byron Heads
wrote:
Seeing this issue on 2.069.2 using etc.c.zlib.
C:\d\dmd2\windows\bin\..\lib\phobos.lib(gzlib)
Error 42: Symbol Undefined __lseeki64
The code was compiling in 2.067. Not clear on where to look
to fix this issue.
I can reproduce with this code...
Windows dmd 2.069.2 32bit
import std.stream;
import std.exception;
// todo: add bzip support..
class GZipBufferedFile : BufferedFile {
private:
GZipFile gZipFile;
/**
* GZipFile stream, safly opens and closes a gzip file,
also will correctly read from the zip file
*/
class GZipFile : File {
import std.c.stdio;
import etc.c.zlib;
FILE* fp;
gzFile fpGZip;
/**
* Use gzopen to open the zip file
*/
this(string filename) {
fp = fopen(filename.toStringz, "rb");
enforce(fp !is null, "Failed to open file
'%s'".format(filename));
version(Windows) {
fpGZip = gzdopen(fp._file, "rb");
super(fpGZip, FileMode.In);
} else {
fpGZip = gzdopen(fileno(fp), "rb");
super(cast(int)fpGZip, FileMode.In);
}
seekable = true;
// Still not supported... sigh
//gzbuffer(fpGZip, 1024 * 1024);
}
ulong tell() {
fflush(fp);
return ftell(fp);
}
/**
* read data block with gzread
*/
override size_t readBlock(void* buffer, size_t size) {
assertReadable();
size = gzread(fpGZip, buffer, cast(uint)size);
if (size == -1) {
size = 0;
}
// use gzeof to test for end of file
readEOF = fpGZip.gzeof != 0;
return size;
}
/**
* make sure we close with gzclose
*/
override void close() {
gzclose(fpGZip);
fpGZip = null;
fp = null;
}
}
public:
this(string filename) {
gZipFile = new GZipFile(filename);
super(gZipFile);
}
ulong tell() {
return gZipFile.tell;
}
}
void main() {
}
Commenting out
gzclose(fpGZip);
allows it to compile..
Re: Interfacing Chromium & SQLite
On Saturday, 5 September 2015 at 13:32:04 UTC, Mike McKee wrote: On Saturday, 5 September 2015 at 11:43:16 UTC, Mike McKee wrote: On a Mac (Yosemite version), how would I create a window in D, embed Chromium, use D to show a local SQLite test database (id, firstname, lastname) inside Chromium, and let someone have a small search form to do like a "full name" search that goes back to D to query the database again? Note: I'm not asking for source code examples, just wondering what technologies I would have to use to connect these pieces together? These libraries might help.. http://code.dlang.org/packages/derelict-cef http://code.dlang.org/packages/d2sqlite3 Other option might be to write a vibe.d (http://vibed.org/) web server as a front end for your DB
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 21:21:11 UTC, Byron Heads wrote:
On Wednesday, 18 February 2015 at 21:05:10 UTC, Byron Heads
wrote:
On Wednesday, 18 February 2015 at 20:55:56 UTC, ketmar wrote:
On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote:
On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote:
On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote:
are you forking? ;-)
I am in the daemonize library
https://github.com/NCrashed/daemonize
can you drop that and just let the program run on foreground?
i suspect
that it may solve your problem. you can detach your program
with setsid
and to avoid forking.
`fork()` is a very fragile thing, and it may be the source of
heisenbugs.
stop `fork()`ing may be the easiest solution (if it solves
something, of
course ;-).
By passing daemonize with the GC enabled is working.. going to
dig into the signals and see if thats it.
My guess is this is going to be related to forking, going to
see if I can make a test case.
Now I am not sure. This code runs correctly:
import std.stdio;
import std.concurrency;
import core.sys.posix.unistd;
import core.sys.posix.sys.stat;
import core.memory;
import std.c.linux.linux;
extern(C) nothrow {
int __libc_current_sigrtmin();
int close(int rd);
}
void foo(Tid tid) {
writeln(1);
foreach(i; 0..1024) {
ubyte[] buffer = new ubyte[](8_192);
auto f = new float[1024];
GC.collect;
}
writeln(2);
send(tid, true);
}
extern(C) void sigsig(int sig) nothrow pure @system @nogc {
}
void main() {
auto pid = fork();
if(pid 0) {
writeln(fork failed);
} if (pid 0) {
writeln(Spawned , pid);
return;
}
umask(0);
auto sid = setsid();
if(sid 0) {
writeln(failed to set sid);
}
assert(signal(SIGABRT, sigsig) != SIG_ERR);
assert(signal(SIGTERM, sigsig) != SIG_ERR);
assert(signal(SIGQUIT, sigsig) != SIG_ERR);
assert(signal(SIGINT, sigsig) != SIG_ERR);
assert(signal(SIGQUIT, sigsig) != SIG_ERR);
assert(signal(SIGHUP, sigsig) != SIG_ERR);
assert(signal(__libc_current_sigrtmin+1, sigsig) != SIG_ERR);
writeln(spawning);
spawn(foo, thisTid);
spawn(foo, thisTid);
spawn(foo, thisTid);
foreach(i; 0..1024) {
auto x = new long[1024];
GC.collect;
}
receiveOnly!bool;
receiveOnly!bool;
receiveOnly!bool;
GC.collect;
writeln(done);
}
GC deadlocks on linux
I have a medium size daemon application that uses several threads, libasync, and daemonize. On windows it runs correctly with GC enabled, but on linux the GC causes a deadlock while allocating memory. Adding core.memory.GC.disable; to main causes the application to work correctly (and quickly till it runs out of memory :D ) I am running in a vagrant VM for test, but this should not be the issue. I am developing this for a future product so I wont have control over the environment that it runs in. I am not sure how to debug or work around this issue. My boss is a little concerned that I am using D but does think its cool. I really want to make this stable and get D in to one of our products, if not I will have to switch to C++ or even worse JAVA! Any help would be appreciated. -Byron System: DMD 2.066.1 Linux vagrant-ubuntu-precise-64 3.2.0-76-virtual #111-Ubuntu SMP Tue Jan 13 22:3 3:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux DISTRIB_ID=Ubuntu DISTRIB_RELEASE=12.04 DISTRIB_CODENAME=precise DISTRIB_DESCRIPTION=Ubuntu 12.04.5 LTS NAME=Ubuntu VERSION=12.04.5 LTS, Precise Pangolin ID=ubuntu ID_LIKE=debian PRETTY_NAME=Ubuntu precise (12.04.5 LTS) VERSION_ID=12.04
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 21:05:10 UTC, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:55:56 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize can you drop that and just let the program run on foreground? i suspect that it may solve your problem. you can detach your program with setsid and to avoid forking. `fork()` is a very fragile thing, and it may be the source of heisenbugs. stop `fork()`ing may be the easiest solution (if it solves something, of course ;-). By passing daemonize with the GC enabled is working.. going to dig into the signals and see if thats it. My guess is this is going to be related to forking, going to see if I can make a test case.
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 21:21:11 UTC, Byron Heads wrote: On Wednesday, 18 February 2015 at 21:05:10 UTC, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:55:56 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize can you drop that and just let the program run on foreground? i suspect that it may solve your problem. you can detach your program with setsid and to avoid forking. `fork()` is a very fragile thing, and it may be the source of heisenbugs. stop `fork()`ing may be the easiest solution (if it solves something, of course ;-). By passing daemonize with the GC enabled is working.. going to dig into the signals and see if thats it. My guess is this is going to be related to forking, going to see if I can make a test case. Might be related: https://issues.dlang.org/show_bug.cgi?id=6846
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:41:12 UTC, Dicebot wrote: Any chance you are using gdm-3.12.x? I was so mad when I have encountered this: https://issues.dlang.org/show_bug.cgi?id=4890 Dont think so $dpkg --get-selections | grep gdm doesn't return anything also running via ssh
Re: GC deadlocks on linux
On Wednesday, 18 February 2015 at 20:55:56 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:35:44 +, Byron Heads wrote: On Wednesday, 18 February 2015 at 20:33:40 UTC, ketmar wrote: On Wed, 18 Feb 2015 20:27:07 +, Byron Heads wrote: are you forking? ;-) I am in the daemonize library https://github.com/NCrashed/daemonize can you drop that and just let the program run on foreground? i suspect that it may solve your problem. you can detach your program with setsid and to avoid forking. `fork()` is a very fragile thing, and it may be the source of heisenbugs. stop `fork()`ing may be the easiest solution (if it solves something, of course ;-). By passing daemonize with the GC enabled is working.. going to dig into the signals and see if thats it.
DMD assert error
Anyone seeing this error? Assertion failure: '0' on line 423 in file 'backend\aa.c' DMD 2.065 win8 -Byron
Re: enum template shorthand and short circuit evaluation
On Tue, 10 Jun 2014 17:06:08 +0200, Artur Skawina via Digitalmars-d-learn wrote: On 06/10/14 02:28, Byron via Digitalmars-d-learn wrote: Should this work? It seems like the short circuit booleans are not working: enum isPrimitive(T) = isBasicType!T || (isArray!T isBasicType! (ForeachType!T)); No. Allowing invalid code just because it will be skipped when the code executes (either at RT or CT) would create other problems, while only solving a minor and relatively rare one (the verbosity of static-if). artur [1] and a few other contexts, like in template constraints, is-expressions etc. Still feels like a bug to me (ie turtles all the way down is not being applied here)
