Re: Is D a good choice for embedding python/octave/julia
On Sunday, 13 March 2016 at 13:02:16 UTC, Bastien wrote: Hi, apologies for what may be a fairly obvious question to some. ## The background: I have been tasked with building software to process data output by scientific instruments for non-experts - basically with GUI, menus, easy config files (JSON or similar) - and the ability to do some serious number crunching. [...] If the other language has some C api that can be called to interpret code then you can do so from D as well. See also e.g. https://github.com/ariovistus/pyd to make this easier for python.
Re: Is D a good choice for embedding python/octave/julia
On Tuesday, 15 March 2016 at 05:56:36 UTC, Ellery Newcomer wrote:
On 03/13/2016 02:36 PM, Laeeth Isharc wrote:
InterpContext context = new InterpContext();
context.py_stmts(outdent("
import numpy
a = numpy.eye(2, dtype='complex128')
"));
context.a.to_d!(Complex!double[][] )();
nitpicking, but the outdent is unnecessary, py_stmts calls it.
hm. maybe that should be documented..
I was in a hurry so just copy and pasted from the unit test... I
never do that myself.
Re: Is D a good choice for embedding python/octave/julia
On 03/13/2016 02:36 PM, Laeeth Isharc wrote:
InterpContext context = new InterpContext();
context.py_stmts(outdent("
import numpy
a = numpy.eye(2, dtype='complex128')
"));
context.a.to_d!(Complex!double[][] )();
nitpicking, but the outdent is unnecessary, py_stmts calls it.
hm. maybe that should be documented..
Re: Is D a good choice for embedding python/octave/julia
On Sunday, 13 March 2016 at 18:42:59 UTC, Bastien wrote:
The sticking point is unless I commit the rest of my life to
maintaining this software, I can't write it all in D. The
algorithms change/are improved yearly; the output format from
the instrument changes once in a while and therefore these need
to be easily scripted/modified by other (non-programming)
scientists and the community that only really know python and
octave.
It's pretty easy to use D from python and python from D. The
documentation for PyD isn't so great, but once you have figured
it out, it's easy. Start with something very small (the examples
and tests in the PyD repo are the best documentation) and work up
from there.
You can even embed D in a Jupyter/iPython notebook, and write
some cells in D and some in python and have them call each other.
See PydMagic by John Colvin.
https://github.com/DlangScience/PydMagic
D can talk to Julia via C linkage already. Just extern(C) when
you declare your D function, and I guess use C style arrays.
I started porting julia.h to D, but didn't get have time to
finish. Ilya Yaroshenko, who created the ndslice library in
std.experimental, will be working on julia integration in coming
months.
By the way, you can also embed R in D and call D from R - see
work by bachmeier on bitbucket. That's very important because of
the huge numbers of R libraries. Although R is slow, there
apparently shouldn't be so much overhead in calling a C library
written for R from D.
I guess my resilience to using D for the algorithms is because
with python, I have access to numpy and matplotlib. There do
seem to be some ongoing developments though:
http://forum.dlang.org/post/mailman.4923.1434903477.7663.digitalmar...@puremagic.com
You already have access to matplotlib from D - see here (and
scroll down or search for matplotlib):
https://d.readthedocs.org/en/latest/examples.html
And you can call numpy from D, but there is some overhead, and I
wouldn't want to do it inside a tight loop.
From PyD unit tests:
InterpContext context = new InterpContext();
context.py_stmts(outdent("
import numpy
a = numpy.eye(2, dtype='complex128')
"));
context.a.to_d!(Complex!double[][] )();
Re: Is D a good choice for embedding python/octave/julia
On Sunday, 13 March 2016 at 18:12:07 UTC, cym13 wrote: On Sunday, 13 March 2016 at 13:02:16 UTC, Bastien wrote: Hi, apologies for what may be a fairly obvious question to some. ## The background: I have been tasked with building software to process data output by scientific instruments for non-experts - basically with GUI, menus, easy config files (JSON or similar) - and the ability to do some serious number crunching. My background is python/octave and would be happy building it in python (or god forbid, even octave), but it would end up clunky and slow once ported to a standalone executable. Hence why I'm looking at other languages. D caught my eye. ## The problem: The sticking point is unless I commit the rest of my life to maintaining this software, I can't write it all in D. The algorithms change/are improved yearly; the output format from the instrument changes once in a while and therefore these need to be easily scripted/modified by other (non-programming) scientists and the community that only really know python and octave. Essentially I'd like a D front end, and a D back-end that does most of the memory and data management but calls and interprets .py, .m and/or .jl scripts (python, matlab, julia) to know how to treat the data. This leaves the py/m/jl scripts visible to be edited by the end user. ## The question: Can it be done? Does this entirely defeat the point of using D and I should just code it in python because of the added overheads? Thanks for your help! B I don't have much experience in mixing python and D but here's my take on it: D is a great language but it's not a great glue language. I know of no binding to Julia but good bindings to python exist (pyd as said above). However, if what you want to keep in python is the algorithms themselves then I don't see the point. If I were to mix the two languages I'd use python to do the user interface, some module interface in order to link the tool to others maybe, but the algorithm would definitely be the one thing I would do in D because that's what D is for. Thanks for all the very useful replies! Overall seems that D on its own may be better. May not be such a bad thing in the end if it moves the scientists away from commerical matlab and the great python 2/3 schism. I guess my resilience to using D for the algorithms is because with python, I have access to numpy and matplotlib. There do seem to be some ongoing developments though: http://forum.dlang.org/post/mailman.4923.1434903477.7663.digitalmar...@puremagic.com So maybe that will all change. I've just ordered a couple books which will hopefully give me a bit more insight into the feasibility of this project. Otherwise, I'll fall back on python...
Re: Is D a good choice for embedding python/octave/julia
On Sunday, 13 March 2016 at 13:02:16 UTC, Bastien wrote: Hi, apologies for what may be a fairly obvious question to some. ## The background: I have been tasked with building software to process data output by scientific instruments for non-experts - basically with GUI, menus, easy config files (JSON or similar) - and the ability to do some serious number crunching. My background is python/octave and would be happy building it in python (or god forbid, even octave), but it would end up clunky and slow once ported to a standalone executable. Hence why I'm looking at other languages. D caught my eye. ## The problem: The sticking point is unless I commit the rest of my life to maintaining this software, I can't write it all in D. The algorithms change/are improved yearly; the output format from the instrument changes once in a while and therefore these need to be easily scripted/modified by other (non-programming) scientists and the community that only really know python and octave. Essentially I'd like a D front end, and a D back-end that does most of the memory and data management but calls and interprets .py, .m and/or .jl scripts (python, matlab, julia) to know how to treat the data. This leaves the py/m/jl scripts visible to be edited by the end user. ## The question: Can it be done? Does this entirely defeat the point of using D and I should just code it in python because of the added overheads? Thanks for your help! B I don't have much experience in mixing python and D but here's my take on it: D is a great language but it's not a great glue language. I know of no binding to Julia but good bindings to python exist (pyd as said above). However, if what you want to keep in python is the algorithms themselves then I don't see the point. If I were to mix the two languages I'd use python to do the user interface, some module interface in order to link the tool to others maybe, but the algorithm would definitely be the one thing I would do in D because that's what D is for.
Re: Is D a good choice for embedding python/octave/julia
On Sun, 13 Mar 2016 13:02:16 +, Bastien wrote: > The sticking point is unless I commit the rest of my life to maintaining > this software, I can't write it all in D. The algorithms change/are > improved yearly; the output format from the instrument changes once in a > while and therefore these need to be easily scripted/modified by other > (non-programming) scientists and the community that only really know > python and octave. http://code.dlang.org/packages/pyd has you covered. The degree to which it has you covered is based on how much you can wrap expensive operations up opaquely. If it turns out next year that someone has a new algorithm they need for something you implemented in D, it's going to be more work overall.
Re: Is D a good choice for embedding python/octave/julia
On Sunday, 13 March 2016 at 13:02:16 UTC, Bastien wrote: Hi, apologies for what may be a fairly obvious question to some. ## The background: I have been tasked with building software to process data output by scientific instruments for non-experts - basically with GUI, menus, easy config files (JSON or similar) - and the ability to do some serious number crunching. My background is python/octave and would be happy building it in python (or god forbid, even octave), but it would end up clunky and slow once ported to a standalone executable. Hence why I'm looking at other languages. D caught my eye. ## The problem: The sticking point is unless I commit the rest of my life to maintaining this software, I can't write it all in D. The algorithms change/are improved yearly; the output format from the instrument changes once in a while and therefore these need to be easily scripted/modified by other (non-programming) scientists and the community that only really know python and octave. Essentially I'd like a D front end, and a D back-end that does most of the memory and data management but calls and interprets .py, .m and/or .jl scripts (python, matlab, julia) to know how to treat the data. This leaves the py/m/jl scripts visible to be edited by the end user. ## The question: Can it be done? Does this entirely defeat the point of using D and I should just code it in python because of the added overheads? Thanks for your help! B I REALLY don't think you should use _any_ scripted language, if what you're looking for is speed. Now for your main question: It can be done. An incomplete list of libraries and bindings for D: http://wiki.dlang.org/List_of_Libraries_and_Frameworks. It includes tools such as GTK. And I'm very sure that it will be faster than writing it completely in python. Another thing: I myself find D *much* easier to program in than python (having experience in both). The many meta-programming tools in D and the nice syntactic features of D really make-up for the increased complexity of the language compared to Python. Somethings I'd like to recommend: OpenCL. For algorithms and such, using the GPU is much much faster than using the CPU.
