Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 05:14:22 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: I have another question: it seems I can spawn hundreds of threads (Heck, even 10_000 is accepted), even when I have 4-8 cores. Is there: is there a limit to the number of threads? I tried a threadpool because in my application I feared having to spawn ~100-200 threads but if that's not the case, I can drastically simplify my code. Is spawning a thread a slow operation in general? Without going into much detail: Threads are heavy, and creating a thread is an expensive operation (which is partially why virtually every standard library includes a ThreadPool). Along with the overhead of creating the thread, you also get the overhead of additional context switches for each thread you have actively running. Context switches are expensive and a significant waste of time where your CPU gets to sit there doing effectively nothing while the OS manages scheduling which thread will go and restoring its context to run again. If you have 10,000 threads even if you won't run into limits of how many threads you can have, this will provide very significant overhead. I haven't looked into detail your code, but consider using the TaskPool if you just want to schedule some tasks to run amongst a few threads, or potentially using Fibers (which are fairly light-weight) instead of Threads.
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 05:14:22 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: This is correct – the LLVM optimizer indeed gets rid of the loop completely. OK,that's clever. But I get this even when put a writeln(some msg) inside the task. I thought a write couldn't be optimized away that way and that it's a slow operation? You need the _result_ of the computation for the writeln. LLVM's optimizer recognizes what the loop tries to compute, though, and replaces it with an equivalent expression for the sum of the series, as Trass3r alluded to. Cheers, David
Re: Can't build phobos
On Sunday, 3 August 2014 at 23:41:27 UTC, Freddy wrote: I am currently working on a phobos fork to include associative ranges, however the unittest fail when i try to build. How am a supposed test any unittests that i add. link:https://github.com/Superstar64/phobos/tree/associative_ranges $ ../dmd/src/dmd |head -1 DMD32 D Compiler v2.066-devel-9d6cef9 0.030s PASS std.algorithm 0.023s PASS std.array 0.034s PASS std.ascii 0.001s PASS std.base64 0.006s PASS std.bigint 0.003s PASS std.bitmanip 0.000s PASS std.complex 0.002s PASS std.concurrency 0.023s PASS std.conv 0.000s PASS std.cstream 0.001s PASS std.csv ** FAIL std.datetime core.exception.AssertError@std/datetime.d(1398): [2012-Apr-30 01:00:00] [2012-Apr-30 00:00:00] generated/linux/debug/32/unittest/libphobos2-ut.so(void std.datetime.SysTime.__unittestL1363_26()+0xd08) [0x41b35ea0] generated/linux/debug/32/unittest/libphobos2-ut.so(void std.datetime.__modtest()+0xdd) [0x41d6e6ed] generated/linux/debug/32/unittest/test_runner(bool test_runner.tester()+0x104) [0x8049524] generated/linux/debug/32/unittest/libphobos2-ut.so(runModuleUnitTests+0xd1) [0x42cce4c1] generated/linux/debug/32/unittest/libphobos2-ut.so(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()+0x2d) [0x42cedd45] generated/linux/debug/32/unittest/libphobos2-ut.so(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x42cedcea] generated/linux/debug/32/unittest/libphobos2-ut.so(_d_run_main+0x152) [0x42cedc5a] generated/linux/debug/32/unittest/test_runner(main+0x14) [0x804968c] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x430a2a83] That's curious. I cannot reproduce this... Unfortunately I don't have time to look into it right now. Is it possible that the test uses the system's timezone definitions, which might somehow be buggy on your computer?
Re: Threadpools, difference between DMD and LDC
Without going into much detail: Threads are heavy, and creating a thread is an expensive operation (which is partially why virtually every standard library includes a ThreadPool). I haven't looked into detail your code, but consider using the TaskPool if you just want to schedule some tasks to run amongst a few threads, or potentially using Fibers (which are fairly light-weight) instead of Threads. OK, I get it. Just to be sure, there is no ThreadPool in Phobos or in core, right? IIRC, there are fibers somewhere in core, I'll have a look. I also heard the vibe.d has them.
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 12:05:31 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: OK, I get it. Just to be sure, there is no ThreadPool in Phobos or in core, right? IIRC, there are fibers somewhere in core, I'll have a look. I also heard the vibe.d has them. There is. It's called taskPool, though: http://dlang.org/phobos/std_parallelism.html#.taskPool
Re: Threadpools, difference between DMD and LDC
On Mon, Aug 4, 2014 at 2:13 PM, Chris Cain via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: OK, I get it. Just to be sure, there is no ThreadPool in Phobos or in core, right? There is. It's called taskPool, though: http://dlang.org/phobos/std_parallelism.html#.taskPool Ah, std.parallelism. I stoopidly searched in std.concurrency and core.* Thanks!
Re: Emacs d-mode cannot handle backquoted backslashe
I took a look and I don't really know if it's possible without using the Emacs 24 only suggestion in the Stack Overflow comment to your question. As far as I can see, before that Emacs syntax tables have a notion of what a string is and what an escape character is. The d-mode code adds the backtick as an alternative to double and single quotes but can't do anything about the backslash. Unless it disables backslash as an escape character everywhere. I tried setting backtick to a function that returns \ instead of the value to see if I could then tailor the function to depend on whether not the point was in a backtick string but that didn't work since `modify-syntax-entry` checks to see if the argument is a string. BTW, python-mode has the same problem, I checked with r\. Atila On Thursday, 31 July 2014 at 08:40:12 UTC, Nordlöw wrote: Currently Emacs d-mode cannot correctly highlight `\` because it doesn't understand that single backslashes are self-contained in back-quoted strings. I believe this extract from d-mode.el (defvar d-mode-syntax-table nil Syntax table used in d-mode buffers.) (or d-mode-syntax-table (setq d-mode-syntax-table (let ((table (funcall (c-lang-const c-make-mode-syntax-table d ;; Make it recognize D `backquote strings` (modify-syntax-entry ?` \ table) ;; Make it recognize D's nested /+ +/ comments (modify-syntax-entry ?+ . 23n table) table))) is highly related to this problem but I don't understand the comment ;; Make it recognize D `backquote strings` Ideas on how to fix this anyone?
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 05:14:22 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: I have another question: it seems I can spawn hundreds of threads (Heck, even 10_000 is accepted), even when I have 4-8 cores. Is there: is there a limit to the number of threads? I tried a threadpool because in my application I feared having to spawn ~100-200 threads but if that's not the case, I can drastically simplify my code. Is spawning a thread a slow operation in general? Most likely those threads either do nothing or are short living so you don't get actually 10 000 threads running simultaneously. In general you should expect your operating system to start stalling at few thousands of concurrent threads competing for context switches and system resources. Creating new thread is rather costly operation though you may not spot it in synthetic snippets, only under actual load. Modern default approach is to have amount of worker threads identical or close to amount of CPU cores and handle internal scheduling manually via fibers or some similar solution. If you are totally new to the topic of concurrent services, getting familiar with http://en.wikipedia.org/wiki/C10k_problem may be useful :)
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 12:05:31 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: IIRC, there are fibers somewhere in core, I'll have a look. I also heard the vibe.d has them. http://dlang.org/phobos/core_thread.html#.Fiber vibe.d adds some own abstraction on top, for example Task concept and notion of Isolated types for message passing but basic are from Phobos.
Re: Threadpools, difference between DMD and LDC
On Mon, Aug 4, 2014 at 3:36 PM, Dicebot via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: Most likely those threads either do nothing or are short living so you don't get actually 10 000 threads running simultaneously. In general you should expect your operating system to start stalling at few thousands of concurrent threads competing for context switches and system resources. Creating new thread is rather costly operation though you may not spot it in synthetic snippets, only under actual load. Modern default approach is to have amount of worker threads identical or close to amount of CPU cores and handle internal scheduling manually via fibers or some similar solution. That's what I guessed. It's juste that I have task that will generate other (linked) tasks, in a DAG. I can use a thread pool of 2-8 threads, but that means storing tasks and their relationships (which is waiting on which, etc). I rather liked the idea of spawning new threads when I needed them ;) If you are totally new to the topic of concurrent services, getting familiar with http://en.wikipedia.org/wiki/C10k_problem may be useful :) I'll have a look. I'm quite new, my only knowledge comes from reading the concurrency threads here, std.concurrency, std.parallelism and TDPL :)
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 14:56:36 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: On Mon, Aug 4, 2014 at 3:36 PM, Dicebot via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: Modern default approach is to have amount of worker threads identical or close to amount of CPU cores and handle internal scheduling manually via fibers or some similar solution. That's what I guessed. It's juste that I have task that will generate other (linked) tasks, in a DAG. I can use a thread pool of 2-8 threads, but that means storing tasks and their relationships (which is waiting on which, etc). I rather liked the idea of spawning new threads when I needed them ;) If you can live with the fact that your tasks might not be truly parallel (i.e. don't use busy waiting or other things that assume that other tasks make progress while a specific task is running), and you only use them for computing (no synchronous I/O), you can still use the fibers in core.thread: http://dlang.org/phobos/core_thread.html#.Fiber
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 14:56:36 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: On Mon, Aug 4, 2014 at 3:36 PM, Dicebot via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: Most likely those threads either do nothing or are short living so you don't get actually 10 000 threads running simultaneously. In general you should expect your operating system to start stalling at few thousands of concurrent threads competing for context switches and system resources. Creating new thread is rather costly operation though you may not spot it in synthetic snippets, only under actual load. Modern default approach is to have amount of worker threads identical or close to amount of CPU cores and handle internal scheduling manually via fibers or some similar solution. That's what I guessed. It's juste that I have task that will generate other (linked) tasks, in a DAG. I can use a thread pool of 2-8 threads, but that means storing tasks and their relationships (which is waiting on which, etc). I rather liked the idea of spawning new threads when I needed them ;) vibe.d additions may help here: http://vibed.org/api/vibe.core.core/runTask http://vibed.org/api/vibe.core.core/runWorkerTask http://vibed.org/api/vibe.core.core/workerThreadCount task abstraction allows exactly that - spawning new execution context and have it scheduled automatically via underlying fiber/thread pool. However, I am not aware of any good tutorials about using those so jump in at your own risk. If you are totally new to the topic of concurrent services, getting familiar with http://en.wikipedia.org/wiki/C10k_problem may be useful :) I'll have a look. I'm quite new, my only knowledge comes from reading the concurrency threads here, std.concurrency, std.parallelism and TDPL :) Have fun :P It is rapidly changing topic though, best practices may be out of date by the time you have read them :)
Re: Threadpools, difference between DMD and LDC
Sorry, I missed this thread (!) till now. On Mon, 2014-08-04 at 13:36 +, Dicebot via Digitalmars-d-learn wrote: On Monday, 4 August 2014 at 05:14:22 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: I have another question: it seems I can spawn hundreds of threads (Heck, even 10_000 is accepted), even when I have 4-8 cores. Is there: is there a limit to the number of threads? I tried a threadpool because in my application I feared having to spawn ~100-200 threads but if that's not the case, I can drastically simplify my code. Is spawning a thread a slow operation in general? Are these std.concurrent threads or std.parallelism tasks? A std.parallelism task is not a thread. Like Erlang or Java Fork/Join framework, the program specifies units of work and then there is a thread pool underneath that works on tasks as required. So you can have zillions of tasks but there will only be a few actual threads working on them. Most likely those threads either do nothing or are short living so you don't get actually 10 000 threads running simultaneously. I suspect it is actually impossible to start this number of kernel threads on any current kernel. In general you should expect your operating system to start stalling at few thousands of concurrent threads competing for context switches and system resources. Creating new thread is rather costly operation though you may not spot it in synthetic snippets, only under actual load. Modern default approach is to have amount of worker threads identical or close to amount of CPU cores and handle internal scheduling manually via fibers or some similar solution. I have no current data, but it used to be that for a single system it was best to have one or two more threads than the number of cores. Processor architectures and caching changes so new data is required. I am sure someone somewhere has it though. If you are totally new to the topic of concurrent services, getting familiar with http://en.wikipedia.org/wiki/C10k_problem may be useful :) I thought they'd moved on the the 100k problem. There is an issue here that I/O bound concurrency and CPU bound concurrency/parallelism are very different beasties. Clearly tools and techniques can apply to either or both. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 16:38:24 UTC, Russel Winder via Digitalmars-d-learn wrote: Modern default approach is to have amount of worker threads identical or close to amount of CPU cores and handle internal scheduling manually via fibers or some similar solution. I have no current data, but it used to be that for a single system it was best to have one or two more threads than the number of cores. Processor architectures and caching changes so new data is required. I am sure someone somewhere has it though. This is why I had or close remark :) Exact number almost always depends on exact deployment layout - i.e. what other processes are running in the system, how hardware interrupts are handled and so on. It is something to decide for each specific application. Sometimes it is even best to have amount of worker threads _less_ than amount of CPU cores if affinity is to be used for some other background service for example. If you are totally new to the topic of concurrent services, getting familiar with http://en.wikipedia.org/wiki/C10k_problem may be useful :) I thought they'd moved on the the 100k problem. True, C10K is a solved problem but it is best thing to start with to understand why people even bother with all the concurrency complexity - all details can be a bit overwhelming if one starts completely from scratch. There is an issue here that I/O bound concurrency and CPU bound concurrency/parallelism are very different beasties. Clearly tools and techniques can apply to either or both. Actually with CSP / actor model one can simply consider long-running CPU computation as form of I/O an apply same asynchronous design techniques. For example, have separate dedicated thread running the computation and send input there via message passing - respond message will act similar to I/O notification from the OS. Choosing optimal concurrency architecture for application is probably even harder problem than naming identifiers.
Re: spawnProcess command-line arguments help
On Sunday, 3 August 2014 at 23:48:09 UTC, Martin wrote: When I use the spawnProcess function in std.process, the command line arguments that I provide to the function seem to get quoted. I can't reproduce this on OS X with 2.066rc1 (args are unquoted). Can someone else check Windows? Sounds like a bug to me.
Re: Threadpools, difference between DMD and LDC
On Mon, 2014-08-04 at 16:57 +, Dicebot via Digitalmars-d-learn wrote: […] This is why I had or close remark :) Exact number almost always depends on exact deployment layout - i.e. what other processes are running in the system, how hardware interrupts are handled and so on. It is something to decide for each specific application. Sometimes it is even best to have amount of worker threads _less_ than amount of CPU cores if affinity is to be used for some other background service for example. David chose to have the pool thread default be (number-of-cores - 1) if I remember correctly. I am not sure he manipulated affinity. This ought to be on the list of things for a review of std.parallelism. […] Actually with CSP / actor model one can simply consider long-running CPU computation as form of I/O an apply same asynchronous design techniques. For example, have separate dedicated thread running the computation and send input there via message passing - respond message will act similar to I/O notification from the OS. Now you are on my territory :-) I have been banging on about message passing parallelism architectures for 25 years, but sadly shared memory multi-threading became the standard model for some totally bizarre reason. Probably everyone was taught they had to use all the wonderful OS implementation concurrency techniques in all their applications codes. CSP is great, cf. Go, Python-CSP, GPars, actors are great, cf. Erlang, Akka, GPars, but do not forget dataflow, cf. GPars, Actian DataRush. There have been a number of PhDs trying to provide tools for deciding which parallelism architecture is best suited to a given problem. Sadly most of them have been ignored by the programming language community at large. Choosing optimal concurrency architecture for application is probably even harder problem than naming identifiers. 'Fraid not, it's actually a lot easier. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 18:22:47 UTC, Russel Winder via Digitalmars-d-learn wrote: Actually with CSP / actor model one can simply consider long-running CPU computation as form of I/O an apply same asynchronous design techniques. For example, have separate dedicated thread running the computation and send input there via message passing - respond message will act similar to I/O notification from the OS. Now you are on my territory :-) I have been banging on about message passing parallelism architectures for 25 years, but sadly shared memory multi-threading became the standard model for some totally bizarre reason. Probably everyone was taught they had to use all the wonderful OS implementation concurrency techniques in all their applications codes. Well it is a territory not completely alien to me either ;) I am less aware of academia research on topic though, just happen to work in industry where it matters. I think initial spread of multi-threading approach has happened because it was so temptingly easy - no need to worry about actually modelling the concurrency execution flow, blocking I/O or scheduling; just write the code as usual and OS will take care of it. But there is no place for magic in programming world in it has fallen hard once network services started to scale. Right now is the glorious moment when engineers are finally starting to appreciate how previous academia research can help them solve practical issues and all this good stuff goes mainstream :) There have been a number of PhDs trying to provide tools for deciding which parallelism architecture is best suited to a given problem. Sadly most of them have been ignored by the programming language community at large. Doubt programming / engineering community will ever accept research that states that choosing architecture can be done on pure theoretical basis :) It simply contradicts too much all daily experience which says that every concurrent application has some unique traits to consider and only profiling can rule them all.
Haskell calling D code through the FFI
TLDR -- Calling D code from Haskell through the FFI works with
DMD but not with GDC or LDC2.
The time consuming version:
Since D allows C to directly call it, if the D code is wrapped in
extern (C){ ... }, I thought it should be possible to call such D
code from Haskell using the FFI.
The FFI in Haskell allows directly calling C code given a header
file, the name of a function, and its C-type. So I made a D file
with the functions I want to call, FunctionsInD.d
extern (C){
int d_f(int x){
return x+2;
}
int d_g(int x, int y){
return x+y;
}
}
Then made a header file, FunctionsInD.h
int d_f(int x);
int d_g(int x, int y);
Then the Haskell wrapper ToD.hs:
module ToD where
import Foreign.C
foreign import ccall FunctionsInD.h d_f
c_f :: CInt - CInt
foreign import ccall FunctionsInD.h d_g
c_g :: CInt - CInt - CInt
h_f :: Int - Int -- the pure haskell version
h_f x = fromIntegral (c_f (fromIntegral x))
h_g :: Int - Int - Int
h_g x y = fromIntegral (c_g (fromIntegral x) (fromIntegral y))
For reasons I don't completely understand, you also need a fake
main function, dummy.d:
void main(){}
And a driver for test, Main.hs:
module Main where
import ToD
main :: IO ()
main = do
let (a,b) = (h_f 3, h_g 3 4)
sequence_ [putStrLn (show a), putStrLn (show b)]
To put this all together, here is the makefile.
DC = dmd
# DC = ldc2
# DC = ldmd2
# DC = gdc
main: FunctionsInD.o FunctionsInD.h dummy.o
ghc Main.hs --make -o main_ffi_test FunctionsInD.o
dummy.o -lphobos2
clean:
rm FunctionsInD.o dummy.o main_ffi_test
FunctionsInD.o FunctionsInD.d:
$(DC) -c FunctionsInD.d
dummy.o: dummy.d
$(DC) -c dummy.d
The question: if I comment out DC=dmd, and uncomment any of the
other options, the build fails. The problem is that ghc
complains of multiple definitions of main (not with DMD only GDC
and LDC2). If I remove dummy.o from the ghc line, I get,
undefined reference to _DMain.
But wait! It gets weirder. I can compile FunctionsInD.d with
GDC or LDC2, as long as I compile dummy.d with DMD, everything
works fine.
Once I get through this hump, I will try to get on with more
interesting interfaces, like strings and structs.
Re: Haskell calling D code through the FFI
Don't forget to call rt_init: http://dlang.org/phobos/core_runtime.html#.rt_init
Re: Haskell calling D code through the FFI
On Monday, 4 August 2014 at 21:10:46 UTC, safety0ff wrote: Don't forget to call rt_init: http://dlang.org/phobos/core_runtime.html#.rt_init Where/when should I call this?
Re: Threadpools, difference between DMD and LDC
On Mon, Aug 4, 2014 at 6:21 PM, Dicebot via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: vibe.d additions may help here: http://vibed.org/api/vibe.core.core/runTask http://vibed.org/api/vibe.core.core/runWorkerTask http://vibed.org/api/vibe.core.core/workerThreadCount task abstraction allows exactly that - spawning new execution context and have it scheduled automatically via underlying fiber/thread pool. However, I am not aware of any good tutorials about using those so jump in at your own risk. Has anyone used (the fiber/taks of) vibe.d for something other than powering websites?
Command Line Application in D
I am trying to build some simple command line applications that I have written in python as a way to learn D. Can you give some examples for me? For instance, I think I remember once seeing somewhere in the documentation an example that took several D files and compiled them all by running some kind of system command. I much appreciate your help! TJB
Re: Haskell calling D code through the FFI
On Monday, 4 August 2014 at 21:14:17 UTC, Jon wrote: On Monday, 4 August 2014 at 21:10:46 UTC, safety0ff wrote: Don't forget to call rt_init: http://dlang.org/phobos/core_runtime.html#.rt_init Where/when should I call this? Before calling any D functions, but usually it's simplest to call it early in main. It initializes the GC and notifies the D runtime of its existence. For simple D functions you might get away without calling it.
Re: Threadpools, difference between DMD and LDC
On Mon, Aug 4, 2014 at 6:38 PM, Russel Winder via Digitalmars-d-learn
digitalmars-d-learn@puremagic.com wrote:
Are these std.concurrent threads or std.parallelism tasks?
A std.parallelism task is not a thread. Like Erlang or Java Fork/Join
framework, the program specifies units of work and then there is a
thread pool underneath that works on tasks as required. So you can have
zillions of tasks but there will only be a few actual threads working on
them.
That's it. Many tasks, a few working threads. That's what I'm
converging to. They are not particularly 'concurrent', but they can
depend on one another.
My only gripes with std.parallelism is that I cannot understand
whether it's interesting to use the module if tasks can create other
tasks and depend on them in a deeply interconnected graph. I mean, if
I have to write lots of scaffolding just to manage dependencies
between task, I might as well built it on core.thread and message
passing directly. I'm becoming quite enamored of message passing,
maybe because it's a new shiny toy for me :)
That's for parsing, btw. I'm trying to write a n-core engine for my
Pegged parser generator project.
Most likely those threads either do nothing or are short living
so you don't get actually 10 000 threads running simultaneously.
I suspect it is actually impossible to start this number of kernel
threads on any current kernel
So, what happens when I do
void doWork() { ... }
Tid[] children;
foreach(_; 0 .. 10_000)
children ~= spawn(doWork);
?
I mean, it compiles and runs happily.
In my current tests, I end the application by sending all thread a
CloseDown message and waiting for an answer from each of them. That
takes about 1s on my machine.
I have no current data, but it used to be that for a single system it
was best to have one or two more threads than the number of cores.
Processor architectures and caching changes so new data is required. I
am sure someone somewhere has it though.
I can add that, depending on the tasks I'm using, it's sometime better
to use 4, 6, 8 or 10 threads, repeatedly for a given task. I'm using a
Core i7, Linux sees it as an 8-core.
So, well, I'll try and see.
Re: Haskell calling D code through the FFI
I get Error: core.runtime.rt_init is private. And Error: core.runtime.init is not accessible. On Monday, 4 August 2014 at 21:22:37 UTC, safety0ff wrote: On Monday, 4 August 2014 at 21:14:17 UTC, Jon wrote: On Monday, 4 August 2014 at 21:10:46 UTC, safety0ff wrote: Don't forget to call rt_init: http://dlang.org/phobos/core_runtime.html#.rt_init Where/when should I call this? Before calling any D functions, but usually it's simplest to call it early in main. It initializes the GC and notifies the D runtime of its existence. For simple D functions you might get away without calling it.
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 21:19:14 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: Has anyone used (the fiber/taks of) vibe.d for something other than powering websites? Atila has implemented MQRR broker with it : https://github.com/atilaneves/mqtt It it still networking application though - I don't know of any pure offline usage.
Re: Haskell calling D code through the FFI
On Monday, 4 August 2014 at 21:35:21 UTC, Jon wrote: I get Error: core.runtime.rt_init is private. And Error: core.runtime.init is not accessible. I would add them to the header and Haskell wrapper (FunctionsInD.h and ToD.hs.) The signatures are: int rt_init(); int rt_term(); When it is linked it will find the symbols in druntime.
Re: Command Line Application in D
I am a little bit confused as to what you want. There is a command line example at dlang.org, and there exists a program (rdmd) that compiles several D files and runs them. http://dlang.org/rdmd.html 2014-08-04 23:20 GMT+02:00 TJB via Digitalmars-d-learn digitalmars-d-learn@puremagic.com: I am trying to build some simple command line applications that I have written in python as a way to learn D. Can you give some examples for me? For instance, I think I remember once seeing somewhere in the documentation an example that took several D files and compiled them all by running some kind of system command. I much appreciate your help! TJB
Declaring run time variables
Hello everyone,
I'm trying to write a command-line application that can detect
whether the input file is gzipped or not. Sounds simple enough,
but I can't seem to do it properly in D (this is my first foray
to the language).
After checking whether the file is gzipped or not, I try to
create a new instance of a class that is basically an InputRange
returning each line of the input file:
```
File f = File(input_file)
// detect gzip ...
if (isGzip)
auto fileIter = new MyFileReader!GzipIterator(f)
else
auto fileIter = new MyFileReader!NormalIterator(f)
foreach(string line; fileIter) {
// do things
}
```
However I always get the compiler error `undefined identifier
lineIter`.
Now, my questions are:
1. What is causing the error? Is this caused by the compiler not
being able to figure out what fileIter is at compile time?
2. I realize the current way of creating file handles may not be
the best way to handle gzipped or non-gzipped file. Is there a
better way to detect and create such file handles based on run
time values?
Thanks in advance :).
Re: Command Line Application in D
On Monday, 4 August 2014 at 21:58:09 UTC, maarten van damme via Digitalmars-d-learn wrote: I am a little bit confused as to what you want. There is a command line example at dlang.org, and there exists a program (rdmd) that compiles several D files and runs them. http://dlang.org/rdmd.html Sorry. I wasn't very clear. Say I want to find all of the files that have a certain extension within a directory and process them somehow at the command line. How could I do that?
Re: Declaring run time variables
On Monday, 4 August 2014 at 22:00:18 UTC, splatterdash wrote:
```
File f = File(input_file)
// detect gzip ...
if (isGzip)
{
auto fileIter = new MyFileReader!GzipIterator(f)
}
else
{
auto fileIter = new MyFileReader!NormalIterator(f)
}
foreach(string line; fileIter) {
// do things
}
```
(Added braces for clarity.)
However I always get the compiler error `undefined identifier
lineIter`.
You have two independent variables called fileIter. Both are only
available in their respective scopes. Declare the variable before
the `if` to get one variable with a wider scope:
SomeCommonBaseTypeOfThoseMyFileReaderVariants fileIter;
if (isGzip)
fileIter = new MyFileReader!GzipIterator(f);
else
fileIter = new MyFileReader!NormalIterator(f);
Re: Haskell calling D code through the FFI
Yes, thank you. That is exactly what I did. On Monday, 4 August 2014 at 21:48:40 UTC, safety0ff wrote: On Monday, 4 August 2014 at 21:35:21 UTC, Jon wrote: I get Error: core.runtime.rt_init is private. And Error: core.runtime.init is not accessible. I would add them to the header and Haskell wrapper (FunctionsInD.h and ToD.hs.) The signatures are: int rt_init(); int rt_term(); When it is linked it will find the symbols in druntime.
Re: Declaring run time variables
On Monday, 4 August 2014 at 22:09:49 UTC, anonymous wrote:
On Monday, 4 August 2014 at 22:00:18 UTC, splatterdash wrote:
```
File f = File(input_file)
// detect gzip ...
if (isGzip)
{
auto fileIter = new MyFileReader!GzipIterator(f)
}
else
{
auto fileIter = new MyFileReader!NormalIterator(f)
}
foreach(string line; fileIter) {
// do things
}
```
(Added braces for clarity.)
However I always get the compiler error `undefined identifier
lineIter`.
You have two independent variables called fileIter. Both are
only
available in their respective scopes. Declare the variable
before
the `if` to get one variable with a wider scope:
SomeCommonBaseTypeOfThoseMyFileReaderVariants fileIter;
if (isGzip)
fileIter = new MyFileReader!GzipIterator(f);
else
fileIter = new MyFileReader!NormalIterator(f);
Indeed I do. I'm not sure which type I should use for the common
base type, though. MyFileReader is a templated class, so using it
plainly did not work. I also tried `InputRange!string` to no
avail despite `MyFileReader` implementing the three InputRange
requirement (popFront(), front, and empty).
Any ideas on what I should as the class?
Re: Declaring run time variables
On Monday, 4 August 2014 at 22:18:24 UTC, splatterdash wrote:
Indeed I do. I'm not sure which type I should use for the
common base type, though. MyFileReader is a templated class, so
using it plainly did not work. I also tried `InputRange!string`
to no avail despite `MyFileReader` implementing the three
InputRange requirement (popFront(), front, and empty).
Any ideas on what I should as the class?
Let MyFileReader implement an interface that has the operations
you need. That interface can be std.range.InputRange!string, or
you can define your own.
Note that a type is an input range when it has the input range
primitives (front, popFront, empty), but it's only a
std.range.InputRange!T when it implements the interface in the
OOP sense: class C : InputRange!E {...}.
Phobos generally doesn't use InputRange, but templatizes
everything. You can go that way, too, and move the foreach loop
to a templated function:
void main()
{
File f = File(input_file)
// detect gzip ...
if (isGzip)
doThings(new MyFileReader!GzipIterator(f));
else
doThings(new MyFileReader!NormalIterator(f));
}
void doThings(I)(I fileIter)
{
foreach(string line; fileIter) {
// do things
}
}
Re: Can't build phobos
On Monday, 4 August 2014 at 10:30:40 UTC, Marc Schütz wrote: On Sunday, 3 August 2014 at 23:41:27 UTC, Freddy wrote: I am currently working on a phobos fork to include associative ranges, however the unittest fail when i try to build. How am a supposed test any unittests that i add. link:https://github.com/Superstar64/phobos/tree/associative_ranges $ ../dmd/src/dmd |head -1 DMD32 D Compiler v2.066-devel-9d6cef9 0.030s PASS std.algorithm 0.023s PASS std.array 0.034s PASS std.ascii 0.001s PASS std.base64 0.006s PASS std.bigint 0.003s PASS std.bitmanip 0.000s PASS std.complex 0.002s PASS std.concurrency 0.023s PASS std.conv 0.000s PASS std.cstream 0.001s PASS std.csv ** FAIL std.datetime core.exception.AssertError@std/datetime.d(1398): [2012-Apr-30 01:00:00] [2012-Apr-30 00:00:00] generated/linux/debug/32/unittest/libphobos2-ut.so(void std.datetime.SysTime.__unittestL1363_26()+0xd08) [0x41b35ea0] generated/linux/debug/32/unittest/libphobos2-ut.so(void std.datetime.__modtest()+0xdd) [0x41d6e6ed] generated/linux/debug/32/unittest/test_runner(bool test_runner.tester()+0x104) [0x8049524] generated/linux/debug/32/unittest/libphobos2-ut.so(runModuleUnitTests+0xd1) [0x42cce4c1] generated/linux/debug/32/unittest/libphobos2-ut.so(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()+0x2d) [0x42cedd45] generated/linux/debug/32/unittest/libphobos2-ut.so(void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())+0x2a) [0x42cedcea] generated/linux/debug/32/unittest/libphobos2-ut.so(_d_run_main+0x152) [0x42cedc5a] generated/linux/debug/32/unittest/test_runner(main+0x14) [0x804968c] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x430a2a83] That's curious. I cannot reproduce this... Unfortunately I don't have time to look into it right now. Is it possible that the test uses the system's timezone definitions, which might somehow be buggy on your computer? That is probably it, I am going to commentout that line not stage it.
Re: Threadpools, difference between DMD and LDC
On Monday, 4 August 2014 at 21:19:14 UTC, Philippe Sigaud via Digitalmars-d-learn wrote: Has anyone used (the fiber/taks of) vibe.d for something other than powering websites? https://github.com/D-Programming-Language/phobos/pull/1910
Re: Haskell calling D code through the FFI
As a note, I can interact with strings as expected, but working with structs looks like it will take a little bit of work. On Monday, 4 August 2014 at 22:17:36 UTC, Jon wrote: Yes, thank you. That is exactly what I did. On Monday, 4 August 2014 at 21:48:40 UTC, safety0ff wrote: On Monday, 4 August 2014 at 21:35:21 UTC, Jon wrote: I get Error: core.runtime.rt_init is private. And Error: core.runtime.init is not accessible. I would add them to the header and Haskell wrapper (FunctionsInD.h and ToD.hs.) The signatures are: int rt_init(); int rt_term(); When it is linked it will find the symbols in druntime.
Taking from infinite forward ranges
Is there a way to take a bounded rage from a infinite forward range?
Given the Fibonacci sequence:
auto fib = recurrence!(a[n-1] + a[n-2])(1, 1);
I can take the first n elements:
take(fib, 10);
But say I want all positive elements below 5 in value (there are
eight such values [2, 8, 34, 144, 610, 2584, 10946, 46368]), how would I
take them? Of course I could filter the range, leaving only positive
values, and then take(fib, 8). But what if I didn't know there were 8,
how could I take them from there filtered range?
Currently I do this:
foreach(e; fib)
{
if (e = val) break;
// so something with e
}
or
while((e = fib.front()) n)
{
// do something with e
fib.popFront();
}
Is there a better way?
Re: Taking from infinite forward ranges
On Tuesday, 5 August 2014 at 01:23:19 UTC, Andrew Edwards wrote:
Is there a way to take a bounded rage from a infinite forward
range?
Given the Fibonacci sequence:
auto fib = recurrence!(a[n-1] + a[n-2])(1, 1);
I can take the first n elements:
take(fib, 10);
But say I want all positive elements below 5 in value
(there are eight such values [2, 8, 34, 144, 610, 2584, 10946,
46368]), how would I take them? Of course I could filter the
range, leaving only positive values, and then take(fib, 8). But
what if I didn't know there were 8, how could I take them from
there filtered range?
Currently I do this:
foreach(e; fib)
{
if (e = val) break;
// so something with e
}
or
while((e = fib.front()) n)
{
// do something with e
fib.popFront();
}
Is there a better way?
I'd use std.algorithm.until:
void main()
{
import std.algorithm, std.range, std.stdio;
auto fib_until_50k = recurrence!(a[n-1] + a[n-2])(1, 1)
.until!(a = a 50_000);
writeln(fib_until_50k);
}
Re: Taking from infinite forward ranges
On 8/5/14, 10:28 AM, Brad Anderson wrote: On Tuesday, 5 August 2014 at 01:23:19 UTC, Andrew Edwards wrote: Is there a way to take a bounded rage from a infinite forward range? I'd use std.algorithm.until: Precisely what I was looking for. Thanks.
Re: Command Line Application in D
On 5/08/2014 10:03 a.m., TJB wrote:
On Monday, 4 August 2014 at 21:58:09 UTC, maarten van damme via
Digitalmars-d-learn wrote:
I am a little bit confused as to what you want.
There is a command line example at dlang.org, and there exists a program
(rdmd) that compiles several D files and runs them.
http://dlang.org/rdmd.html
Sorry. I wasn't very clear. Say I want to find all of the files that
have a certain extension within a directory and process them somehow at
the command line. How could I do that?
Just a little something I made for you. Untested of course. But takes an
argument from cli, which is a glob. Foreach file under current working
directory, if its a file write out processing.
(I gave std.stdio an alias because std.file and std.stdio conflict for
some symbols)
import std.file;
import stdio = std.stdio;
void main(string[] args) {
if (args.length == 2) {
foreach(entry; dirEntries(., args[1], SpanMode.Depth)) {
if (isDir(entry.name)) {
} else if (isFile(entry.name)) {
stdio.writeln(Processing ~ entry.name);
}
}
} else {
stdio.writeln(Arguments: glob);
}
}
Re: Threadpools, difference between DMD and LDC
https://github.com/D-Programming-Language/phobos/pull/1910 Very interesting discussion, thanks. I'm impressed by the amount of work you guys do on github.
