beginner's pyd question - exporting structs to python
Hi there.
Brief introduction, and a beginner's question.
I just started playing with D a couple of weeks ago. I have been
programming in C on and off since the late 80s, but I do finance
for a living and my programming skills grew rusty. I have a bit
more time now to catch up with developments, and D looks a very
intriguing robust alternative to C when python won't cut the
mustard.
I am trying to implement the Demark technical analysis indicators
in D, and want to be able to call them from python.
So I have set up pyd okay, and the examples seem to work fine.
Running on Fedora 20. I am struggling a bit with the syntax on
wrap_struct to export a struct and function returning a struct to
python. Here is what I have:
module hello2;
import pyd.pyd;
import std.stdio;
import std.conv;
struct t_mystruct {
int i;
string s;
};
t_mystruct hello(int[] inp) {
int i;
t_mystruct mystruct;
mystruct.i=inp.length;
mystruct.s=hello there;
return mystruct;
}
extern(C) void PydMain() {
def!(hello)();
module_init();
wrap_struct!(
t_mystruct,
t_mystruct
);
}
Error:
def: hello
/root/anaconda/lib/python2.7/site-packages/celerid/infrastructure/pyd/def.d(151):
Error: static assert string parameters must be wrapped with
Docstring, Mode, etc
/root/anaconda/lib/python2.7/site-packages/celerid/infrastructure/pyd/class_wrap.d(1474):
instantiated from here: Args!(, , t_mystruct, ,
t_mystruct)
hellostruct.d(30):instantiated from here:
wrap_struct!(t_mystruct, t_mystruct)
error: command 'dmd' failed with exit status 1
[root@fedora demark]#
Apologies if it's in one of the examples - I did poke around, and
couldn't see any obvious sample.
Many thanks.
Laeeth.
Re: beginner's pyd question - exporting structs to python
Thank to jwhear on irc who solved it for me despite claiming not
to be a pyd guru.
In case it's of benefit, here is what works:
module hellostruct;
import pyd.pyd;
import std.stdio;
import std.conv;
struct t_mystruct {
int i;
string s;
};
t_mystruct hellostruct(int[] inp) {
int i;
t_mystruct mystruct;
for (i=0;iinp.length;i++)
{
writefln(inp ~ to!string(i) ~ is ~
to!string(inp[i]));
}
writefln( * ok, bye);
mystruct.i=99;
mystruct.s=hello there;
return mystruct;
}
extern(C) void PydMain() {
def!(hellostruct)();
module_init();
wrap_struct!(
t_mystruct,
Member!s,
Member!i
)();
}
[root@fedora demark]# cat test.py
import os.path, sys
import distutils.util
# Append the directory in which the binaries were placed to
Python's sys.path,
# then import the D DLL.
libDir = os.path.join('build', 'lib.%s-%s' % (
distutils.util.get_platform(),
'.'.join(str(v) for v in sys.version_info[:2])
))
sys.path.append(os.path.abspath(libDir))
import hellostruct
inp=[1,2,3,4]
mystruct= hellostruct.hellostruct(inp)
print mystruct.i
print mystruct.s
Re: beginner's pyd question - exporting structs to python
All the cool folk doing data analysis and visualization using Python no longer bother with hand written C (*) for when pure Python won't cut the mustard. If Numba can't do the job, then Cython gets used. I have all my computational pure Python source codes running as fast as C these days thanks to Numba. (And judicious profiling.) I would say that Python folk will now only be looking to C, C++, Fortran, D, for pre-written libraries in those language. Given all the codes are written in C, C++ or Fortran with none in D… Thanks for the colour - I appreciate it. I have played with numba and pypy with numpy and it seems a powerful tool for some kinds of jobs. Perhaps it is my relative unfamiliarity with python, but for the time being I feel more comfortable with C type languages for other kinds of work. As a pragmatic idealist, one may as well use whatever tool seems to be pretty good generally and which one feels confident in wielding to accomplish the task at hand. Out of curiosity, what do you use D for given your views about the redundancy of C type languages for non-system programming? Should use print as a function not as a statement. Use Python 3, or if you have to use Python 2 (which almost no-one does): from __future__ import print_function as the first statement. Thank you.
Re: beginner's pyd question - exporting structs to python
On Monday, 18 August 2014 at 18:08:59 UTC, Russel Winder via Digitalmars-d-learn wrote: […] distutils.util.get_platform(), […] Does os.uname() not provide sufficient information? This was boilerplate generated by pyd.
Re: beginner's pyd question - exporting structs to python
On Monday, 18 August 2014 at 19:28:55 UTC, Russel Winder via Digitalmars-d-learn wrote: On Mon, 2014-08-18 at 19:00 +, Laeeth Isharc via Digitalmars-d-learn wrote: On Monday, 18 August 2014 at 18:08:59 UTC, Russel Winder via Digitalmars-d-learn wrote: […] distutils.util.get_platform(), […] Does os.uname() not provide sufficient information? This was boilerplate generated by pyd. Hummm… if I get time I feel a PR in order. Perhaps I should rephrase. I copied the example as a base, so technically it wasn't generated. But no harm in tweaking the examples.
Re: beginner's pyd question - exporting structs to python
Whilst the hardcore Pythonistas remain Pythonistas, some of the periphery has jumped ship to Go. Sadly D did not capture these folk, it perhaps should have done. It would be easy to blame fadism, but I think the actual reasons are far less superficial. So I gather that you agree that what everyone is doing may not be the best in this case (python vs D) if there are no direct network effects beyond libraries and getting help and you have the freedom to determine your own platform choices? For me, NumPy has some serious problems despite being the accepted norm for computational work. If not too offtopic, do you have a link describing, or would you briefly summarize these problems? I am intrigued. And what would you suggest in its place? Fortran? Out of curiosity, what do you use D for given your views about the redundancy of C type languages for non-system programming? In a sense I could rightly be labelled a D dilettante. I had a possible startup a couple of years ago where we would have used D, but it never started. A side-effect was that I gave some support to David Simcha creating std.parallelism, but for the last couple of years my income has come from Python or Groovy/GPars with no real D activity. Would you consider D stable enough/suitable for general financial market work with development initially by a small underresourced team? Not ultra high frequency execution - at most legging in and managing longer term positions. But I am more interested in sentiment analysis, producing technical analysis indicators that summarize market activity across many different securities, some bond arb stuff. C++ just seems so ugly, and I feel uncomfortable only having python in the toolbox. D seems so far to be quite suitable...
Re: beginner's pyd question - exporting structs to python
Dr Russel Winder 41 Buckmaster Road London SW11 1EN, UK Are there any D users groups/meetups in London? I see you are not far away (I am in Barnes). Laeeth
building shared library from D code to import into cython
Hi.
I am trying to create a shared library in D linked against phobos
so that I may use this in a cython extension module for Python.
Ultimately I would like to be able to use a D class or struct
(via the C++ interface) and call it from within cython, since
cython classes cannot be instantiated without the gil (and this
prevents easy parallelisation).
I feel a bit foolish asking the question as there is a nice
example here for working with plain C using dmd as the linker,
and using dmd and gcc to create a DMD shared library statically
linked to phobos. However, I have not succeeded in creating a D
library statically linked to phobos that works with cython and
python,
http://dlang.org/dll-linux.html#dso7
I tried it first with test C code to make sure I am able to get
the C library/cython/Python interaction working.
pytest.c:
#include stdio.h
long pytest(long a)
{
return a+1;
}
int main()
{
long a =pytest(100);
printf(%ld,a);
return 0;
}
pytestpy.pyx:
cdef extern long pytest(long a)
cpdef pytestpy():
return pytest(109)
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
ext_modules = cythonize([
Extension(pytestpy, [pytestpy.pyx],
libraries=[pytest],
)
]))
command line:
gcc -shared -o libpytest.so pytest.o
python setup.py build_ext -i
copied libpytest.so to /usr/local/lib
python
import pytestpy
pytestpy.pytestpy()
it works
now try pytest.d
import std.stdio;
extern (C) long pytest(long a)
{
return a*2;
}
void main()
{
auto a =pytest(100);
writefln(%d,a);
}
command line:
rm pytestd.o
rm libpytest.so
rm /usr/local/lib/libpytest.so
dmd -c pytest.d -fPIC
gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so
-L-rpath=/usr/local/lib
python
import pytestpy
Traceback (most recent call last):
File stdin, line 1, in module
ImportError: /usr/local/lib/libpytest.so: undefined symbol:
_D3std5stdio12__ModuleInfoZ
I guess it is not linking to the D runtime, but I am not sure
what I should be doing to fix.
Any thoughts appreciated.
(The next step I was going to try when this works was C++
interface vs importing as a Cython class, but I thought best to
start simple).
I am running this on 64 bit Fedora 20.
Thanks.
Laeeth.
Re: building shared library from D code to import into cython
Hi.
Thanks for the quick response.
The -defaultlib was left around from trying all kinds of
combinations of dmd and gcc. I am not used to gcc, and it will
take me some time to become properly acquainted with all the
options.
I simply could not get it to recognize libphobos no matter what
path settings I tried passing.
http://forum.dlang.org/thread/k3vfm9$1tq$1...@digitalmars.com?page=4
I don't know whether this was necessary, but I did manage to get
it to work after looking at a thread from a couple of years back.
I recompiled the D runtime and Phobos with position independent
code (PIC) and pointed everything to that version instead, and it
worked fine. (Just export PIC=True, rename the posix.mak to
Makefile, change the DMD path to /usr/sbin/dmd - in my case, and
it all worked).
Next step is to try calling D interface from Cython.
Laeeth
In case anyone else should struggle with this in future.
Makefile here:
PHOBOS_PATH=/opt/dmd2/src/phobos/generated/linux/release/64
pytestpy.so: pytest.d pytestpy.o
dmd -fPIC -c pytest.d #-shared -defaultlib=libphobos2.so
-L-rpath=.:${PHOBOS_PATH}
gcc -shared pytest.o pytestpy.o -o pytest.so -lphobos2 -lpthread
-lrt -L. -L${PHOBOS_PATH} -Wl,-rpath=.:${PHOBOS_PATH} -o
pytestpy.so -defaultlib=libphobos2.so -L-rpath=.:${PHOBOS_PATH}
pytest.o: %.d
dmd -c $
pytestpy.o: pytestpy.pyx
cython pytestpy.pyx
gcc -fPIC -c pytestpy.c -o pytestpy.o -I/usr/include/python2.7
clean:
rm -f pytestpy.c pytest.o pytestpy.o pytest.so
pytest.d
extern (C) long pytest(long a)
{
return a*2;
}
pytestpy.pyx
cdef extern long pytest(long a)
cpdef pytestpy():
return pytest(109)
print pytest(102)
setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
ext_modules = cythonize([
Extension(pytestpy, [pytestpy.pyx],
libraries=[pytest],
)
]))
On Tuesday, 7 October 2014 at 22:46:09 UTC, Freddy wrote:
On Tuesday, 7 October 2014 at 20:55:59 UTC, Laeeth Isharc wrote:
Hi.
I am trying to create a shared library in D linked against
phobos so that I may use this in a cython extension module for
Python. Ultimately I would like to be able to use a D class
or struct (via the C++ interface) and call it from within
cython, since cython classes cannot be instantiated without
the gil (and this prevents easy parallelisation).
I feel a bit foolish asking the question as there is a nice
example here for working with plain C using dmd as the linker,
and using dmd and gcc to create a DMD shared library
statically linked to phobos. However, I have not succeeded in
creating a D library statically linked to phobos that works
with cython and python,
http://dlang.org/dll-linux.html#dso7
I tried it first with test C code to make sure I am able to
get the C library/cython/Python interaction working.
pytest.c:
#include stdio.h
long pytest(long a)
{
return a+1;
}
int main()
{
long a =pytest(100);
printf(%ld,a);
return 0;
}
pytestpy.pyx:
cdef extern long pytest(long a)
cpdef pytestpy():
return pytest(109)
setup.py:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
ext_modules = cythonize([
Extension(pytestpy, [pytestpy.pyx],
libraries=[pytest],
)
]))
command line:
gcc -shared -o libpytest.so pytest.o
python setup.py build_ext -i
copied libpytest.so to /usr/local/lib
python
import pytestpy
pytestpy.pytestpy()
it works
now try pytest.d
import std.stdio;
extern (C) long pytest(long a)
{
return a*2;
}
void main()
{
auto a =pytest(100);
writefln(%d,a);
}
command line:
rm pytestd.o
rm libpytest.so
rm /usr/local/lib/libpytest.so
dmd -c pytest.d -fPIC
gcc -shared -o libpytest.so pytest.o -defaultlib=libphobos2.so
-L-rpath=/usr/local/lib
python
import pytestpy
Traceback (most recent call last):
File stdin, line 1, in module
ImportError: /usr/local/lib/libpytest.so: undefined symbol:
_D3std5stdio12__ModuleInfoZ
I guess it is not linking to the D runtime, but I am not sure
what I should be doing to fix.
Any thoughts appreciated.
(The next step I was going to try when this works was C++
interface vs importing as a Cython class, but I thought best
to start simple).
I am running this on 64 bit Fedora 20.
Thanks.
Laeeth.
Since when does gcc have a -defaultlib option?
---
$ man gcc | grep defaultlib
object-file-name -llibrary -nostartfiles -nodefaultlibs
This is useful when you use -nostdlib or
-nodefaultlibs but you do
-nodefaultlibs is used.
-nodefaultlibs
-nodefaultlibs is libgcc.a, a library of
Re: building shared library from D code to import into cython
Thanks for this. I am aware of pyd and will take a look at source/build process. Any thoughts on speed in 2014 of pyd vs using cython to talk to D directly via C/C++ interface? I saw this old coment here: prabhuramachandran.blogspot.co.uk/2008/09/python-vs-cython-vs-d-pyd-vs-c-swig predict that the D version's relative slowness might have something to do with Pyd's somewhat awful handling of arrays (at least in part). More low-level and verbose, but possibly faster code could be written to compensate for this if this is indeed the problem. However, this is not a very attractive solution. (Optimally, Pyd should be capable of directly pointing D arrays at numpy arrays, but this is not actually implemented.) On Friday, 10 October 2014 at 02:19:17 UTC, Ellery Newcomer wrote: On Wednesday, 8 October 2014 at 00:25:57 UTC, Laeeth Isharc wrote: Hi. Thanks for the quick response. The -defaultlib was left around from trying all kinds of combinations of dmd and gcc. I am not used to gcc, and it will take me some time to become properly acquainted with all the options. I managed to get it to compile with default dmd rpm: https://github.com/ariovistus/cythonic_d fedora 20, x86_64 Are you aware of pyd? (https://bitbucket.org/ariovistus/pyd) It knows how to build shared libraries, so I recommend you play with it, if only to watch how it does that. Biggest gotcha is starting up druntime.
Returning multiple arrays from function - struct or byref the only option?
Hi.
I have to write a bunch of functions that operate on input arrays
to return multiple output arrays.
In case helpful the inputs are price bars or economic data points
(datetime, ohlc) and the outputs are nx1 arrays (I won't say
vectors) of doubles or structs.
What is the best way to return multiple arrays in this kind of
situation. In Python I returned a tuple of numpy arrays, and the
C way would be to pass a pointer to the return destinations, and
I guess I could do the same in a D way by passing by ref not
value).
I see that I can return a struct containing dynamic arrays, and
return it (by value of course). I have read the forum discussion
a while back over how to return multiple values, and tried using
tuples. I don't see that I can return a tuple of arrays - am I
missing something?
Here is a simple case. Can I do better ?
Thanks.
import std.typecons;
import std.stdio;
struct RetStruct
{
double[] a;
double[] b;
}
RetStruct myfunction(double x)
{
RetStruct s;
double[] a,b;
a~=x+1.0;
a~=x+9.0;
b~=x+2.0;
b~=x+11.0;
s.a=a;
s.b=b;
return s;
}
void main()
{
writefln(%s,myfunction(99.0));
}
Re: Returning multiple arrays from function - struct or byref the only option?
Here is byref:
import std.typecons;
import std.stdio;
void myfunction(double x, ref double[] a, ref double[] b)
{
a~=x+1.0;
a~=x+9.0;
b~=x+2.0;
b~=x+11.0;
return;
}
void main()
{
double[] a;
double[] b;
myfunction(99.0,a,b);
writefln(%s %s,a,b);
}
Re: Returning multiple arrays from function - struct or byref the only option?
Thanks for the thoughts Meta and Ali.
Laeeth.
On Wednesday, 15 October 2014 at 17:56:06 UTC, Ali Çehreli wrote:
On 10/15/2014 09:48 AM, Laeeth Isharc wrote:
struct RetStruct
{
double[] a;
double[] b;
}
RetStruct myfunction(double x)
That's my preference. Tuples would work as well but they have
two minor issues for me:
- Unlike a struct, the members are anonymous. (Yes, tuples
members can have names as well but not when returning or
creating conveniently by 'return tuple(a, b)'.)
- Unlike Python, there is no automatic tuple expansion so one
has to refer to the members as result[0] and result[1], which
is less readable than struct members. (Yes, there is some
support for tuple expansion e.g. in foreach but it has some
issues with the automatic foreach loop counter.)
Summary: I return by struct. :)
Ali
Re: Issue with WIKI example Win32_DLLs_in_D
Funnily enough I was just playing with this last night trying to get Excel to talk to dlang DLL. I borrowed a C example elsewhere on web and used a different .def file. Something like this: LIBRARY dprop DESCRIPTION 'My DLL written in D' EXETYPE NT CODEPRELOAD DISCARDABLE DATAWRITE EXPORTS useArray = useArray usemyTest = usemyTest Where the two functions exported from D to excel were as above. Try that and let me know. If that doesn't work I will send you a link to from repository. Tuesday, 21 October 2014 at 19:59:51 UTC, Andre wrote: Hi, by copy and paste the example from http://wiki.dlang.org/Win32_DLLs_in_D exactly as described, the following errors is thrown: J:\Projects\Tests\Exampledmd test mydll.lib -g OPTLINK (R) for Win32 Release 8.00.15 test.obj(test) Error 42: Symbol Undefined _D5mydll12__ModuleInfoZ I created a batch program which executes the commands as described on the wiki page: dmd -c mydll -g dmd mydll.obj mydll.def -g -L/map C:\D\dmd2\windows\bin\implib /noi /system mydll.lib mydll.dll dmd test mydll.lib -g Could you check whats wrong with the example and maybe update the WIKI page? Thanks. Kind regards André
Dart bindings for D?
Hi. I have had a look around for these, but was not able to see them. It looks perhaps like dart_api.h is the main file to convert - I will have a crack at starting this unless anyone knows of any already in existence. Rationale for using Dart in combination with D is that I am not thrilled about learning or writing in Javascript, yet one has to do processing on the client in some language, and there seem very few viable alternatives for that. It would be nice to run D from front to back, but at least Dart has C-like syntax and is reasonably well thought out. Am I missing any existing bindings somewhere? Thanks. Laeeth.
Re: Dart bindings for D?
Hi. Thanks for all the thoughts, and sorry it has taken me a little while to reply. Adam - I liked your book very much: it really complemented the other resources out there, especially in communicating a refreshing spirit of enthusiasm and fearless exploration. ketmar - I took a look at script.d very briefly. I would love if I could use some kind of D-like and D-friendly scripting language on the client (Suliman has a point!), but my understanding is that I cannot if I want to use standard browsers without a plugin - Chrome, Firefox, etc. BTW what was the story behind dscript? It seems to have changed its name and no longer have so much connection to D: http://forum.dlang.org/thread/422be824.6030...@nospam.org It would be awesome to write front-end tools in D. However, there won't be much browser support unless you're backed by Google or Microsoft. Etienne - yes, indeed. That is exactly the problem. In theory your suggestion of compiling D to Javascript sounds intriguing, but I wonder if in practice it will be worth it on the client (I confess that you likely know much more about this than me). Even if interoperability with the ecosystem is possible, I suppose it will be clunkier to write in D compiled to JS than in Dart because there are fewer people pouring energy into the project to make it easy. I don't mind reimplementing some things on the back end in order to make it fast, beautiful, and efficient but I have no interest in re-writing anything in user interface domain. (Others may be different). As a second-best, but overall pretty appealing choice (superficially, since I haven't written anything substantial in it), Dart at least has a C-like syntax, seems to avoid most of the JS infelicities, and has some of the benefits of type-checking. It hasn't taken off yet, but my guess is it will. What kind of client are you doing? If you are writing a web page, you don't need any kind of script language API. JavaScript or dart or whatever talk with your server application through http requests or websockets, whereas script language APIs are meant for extending your application in the same process. For example, a text editor might have a script language to make custom functions for hotkeys. - Adam Ruppe Proprietary trading analytics and charting - still at an early stage so the design is not fully mapped out. I understand that if I run D on server and Dart on client then I don't need to worry about APIs. But 1) if I decide to run Dart on server and write my number-crunching analytics in D then I suppose I either need the Dart headers translated to D (API) or have the Dart web server talk to a D analytics server via a socket. And 2) I cannot say that I definitely do not want to cache things and do some work on client, and so I need to see what's possible to avoid getting trapped in purely local optimum. 3) It's a long way from being relevant, but ultimately realtime data licensing occurs at local user level for Bloomberg etc, so much better to be able to tap in to a data source the user already has. Dart makes most sense for internal web applications.- Ola Yes - exactly my situation. Dart VM is available as a standalone, which can be set up to act as a web server. But you want to integrate it into D? - Ola There are already C headers, and it is simple to wrap any C function manually so you can call it from Dart. Perhaps it would be possible to do something like PyD (I am still learning templates/CTFE so I don't know), but for now I was just thinking of converting the 3000 line odd (but mostly whitespace) header from .h to .d. Most of it's easy, but I struggled a bit with syntax for using alias in D to declare function pointer return types and parameters (I think I get it now). Why bother? D seems just perfect for the particular kind of quant financial work I want to do, but for the time being it is lacking on web server and client. Vibe.d is great, but is not a complete framework from what I have seen (which is fine, since it doesn't pretend to be one, but makes it harder if you don't want to spend your time on this). And we discussed the client above. So I need to have some way for Dart to talk to D. I don't think so, but integrating DartVM into D means you have to deal with two different garbage collectors or put a substantial amount of work into making D use the Dart collector. Thanks for the warning. I will have a low number of users, and whilst on occasion working data sets might be large, they won't stick around for very long so pre-allocating buffers should work fine (I hope). Thanks once again to everyone for all the suggestions and observations. Laeeth/ On Wednesday, 29 October 2014 at 22:12:32 UTC, Laeeth Isharc wrote: Hi. I have had a look around for these, but was not able to see them. It looks perhaps like dart_api.h is the main file to convert - I will have a crack at starting this
alias and extern(C)
Hi.
I am trying to translate the following from the Dart header:
typedef void (*Dart_MessageNotifyCallback)(Dart_Isolate
dest_isolate);
So I made a little simple test to understand callbacks in D. The
code below works fine if you remove the extern(C). But I get the
error functionpointertest.d(6): Error: basic type expected, not
extern with the code as it is.
How do I use alias with extern ?
Thanks.
Laeeth.
import std.stdio;
// test_typedef.d
//typedef void (*Dart_MessageNotifyCallback)(Dart_Isolate
dest_isolate);
alias Callback= extern(C) void function(int);
extern(C) void testfn(int i)
{
writefln(%s,i+1);
return;
}
extern(C) void testfn2(int i)
{
writefln(%s,i*i);
return;
}
void foo(Callback callback)
//void foo(void function(int) callback)
{
callback(100);
//callback(101);
}
void main()
{
foo(testfn);
foo(testfn2);
}
Re: alias and extern(C)
The code below works fine if you remove the extern(C). But I get the error functionpointertest.d(6): Error: basic type expected, not extern with the code as it is. How do I use alias with extern ? [...] alias Callback= extern(C) void function(int); Compiles as is with dmd 2.066. For 2.065 and older you have to put extern(C) in front of alias: extern(C) alias Callback= void function(int); Many thanks, anon. You are right - I was running older dmd on this machine. It works fine with your modification, as you suggest. Laeeth.
Re: Dart bindings for D?
Ah - makes sense. It is satisfyingly fast... On Thursday, 30 October 2014 at 21:33:59 UTC, ketmar via Digitalmars-d-learn wrote: On Thu, 30 Oct 2014 17:39:13 + Laeeth Isharc via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: BTW what was the story behind dscript? It seems to have changed its name and no longer have so much connection to D: http://forum.dlang.org/thread/422be824.6030...@nospam.org we have rdmd now, which can be used in shebangs and allows to run D programs as shell scrips. it is able to track dependencies and caching compiled script, so 2nd and other invocations will be very fast.
simd and dmd compiler v 2.066 and 2.067.0-b1
Hi.
Not sure if my code is correct - I wanted to build the simplest
working example of simd use. The following compiles and works
under ldc (I have not disassessembled the result to see if it is
using simd instructions), but generates a compiler error under
dmd (2.066 and 2.067.0-b1 running under Fedora 20).
The only message is:
Internal error: backend/el.c 2874
Let me know if you think the code is okay, and if so I will see
if I can figure out how to generate a bug report (if it's not
something already known).
Thanks.
Laeeth.
import std.stdio;
import core.simd;
void main()
{
short8 vec;
foreach(i;0..8)
{
vec.ptr[i]=cast(short)i;
}
vec=3*vec;
foreach(i;0..8)
{
writefln(%s,vec.ptr[i]);
}
}
Re: simd and dmd compiler v 2.066 and 2.067.0-b1
On Monday, 3 November 2014 at 21:23:50 UTC, Marc Schütz wrote:
Reduced testcase:
import core.simd;
void main()
{
short8 vec;
vec=vec*3;
}
I've filed a bug report:
https://issues.dlang.org/show_bug.cgi?id=13674
Thanks - appreciate it. Laeeth.
rndtonl
what am I doing wrong here?
import std.math;
import std.stdio;
void main()
{
real fac;
fac=1.2;
fac=rndtonl(fac);
}
[root@fedorabox util]# dmd bug.d
bug.o: In function `_Dmain':
bug.d:(.text._Dmain+0x3b): undefined reference to `rndtonl'
collect2: error: ld returned 1 exit status
--- errorlevel 1
Re: rndtonl
Thanks, Adam. Should we perhaps make a pull to suggest updating the docs/wiki? As the point below is not what one would infer from the dlang.org library reference page. (If I say we, it's because I don't know what the protocol is, or whether my perception is right). On Tuesday, 4 November 2014 at 18:39:29 UTC, Adam D. Ruppe wrote: I think rndtonl is a C library function that isn't always present in the system. It doesn't work on my computer either and I can't find any documentation about it. It is probably not meant to be called by end users.
status of D optimizers benefiting from contracts ?
https://www.youtube.com/watch?v=e2F2pqeMLuwlist=PL4EvMyUrlAJmEfs8l6oW2BlnALiDu7kGy 31 minutes in, Walter Bright suggests that a supplementary benefit of using contrats is helping the compiler make optimisations. He uses the example of being able to do faster 32 bit arithmetic when the variables are longs but per contract within the bounds for 32 bits. I wondered to what extent D compilers are doing this already (peeking at the contract), and what plans if any there are to incorporate these in generating fast code. Thanks. Laeeth
Re: status of D optimizers benefiting from contracts ?
Thanks. Laeeth.
PyD-like wrapping for Excel/VBA and Julia?
I have a bunch of D functions I would like to make available to
Excel (and possibly Julia) without having to write wrappers for
each function individually.
For Excel, I think one needs two levels of wrapper - one is to
create a C style interface [using extern(Windows) calling
convention, and pointers to doubles or structs rather than
dynamic arrays], and the second is to write the VBA wrapper that
calls the C interface. (There may be more efficient purer ways
of doing this, but I don't wish to spend time learning Excel
internals/object models, and I know my route will work reasonably
well).
So a very simple D function:
double test(double[] inp, ref double[] oup)
{
double sum=0.0;
oup.length=inp.length;
foreach(i;0..inp.length)
{
oup[i]=inp[i]*inp[i];
sum+=oup[i];
}
return sum;
}
and my first attempt at a wrapper:
extern(Windows) double vbwrap_test(double* inp,size_t
num_inp,double* oup,size_t num_oup)
{
double[] arg_inp;
arg_inp.length=num_inp;
double[] arg_oup;
arg_oup.length=num_oup;
foreach(arg;0..num_inp)
{
arg_inp[arg]=inp[arg];
}
foreach(arg;0..num_oup)
{
arg_oup[arg]=oup[arg];
}
return test(arg_inp,arg_oup);
}
I didn't yet write the bit that copies the result from test back
to the calling double*.
Slowly learning metaprogramming/CTFE in D, and the code above was
generated from the function definition by some horrible looking D
code, ready to place into a string mixin. I need to make it more
general (to accept structs etc), and write the VBA wrapper
generation too.
But if anyone has any useful pointers or suggestions or would
like to help, do let me know. I guess this project could be of
broader application since in the financial and other sectors
people still are stuck with Excel as a front end in many cases,
for better or for worse.
I will look at LuaD and PyD and Adam's web.d for inspiration..
Julia was just something to think about further down the line. I
haven't used it much yet.
Thanks.
Laeeth.
Re: PyD-like wrapping for Excel/VBA and Julia?
On Friday, 19 December 2014 at 01:59:05 UTC, Ellery Newcomer
wrote:
On 12/18/2014 12:41 PM, Laeeth Isharc wrote:
I have a bunch of D functions I would like to make available
to Excel
(and possibly Julia) without having to write wrappers for each
function
individually.
I've thought about refactoring the reflection parts of pyd into
a reusable library for e.g. resurrecting RuD. Come to think of
it, that would probably be necessary for supporting pypy.
It'd be a heck of a lot of work, though.
For your wrapper, you can probably do something like
extern(Windows) double vbwrap_test(double* inp,size_t
num_inp,double* oup,size_t num_oup)
{
return test(inp[0 .. num_inp], arg_oup[0 .. num_oup]);
}
with .dup sprinkled in as you see fit. And you don't need to
explicitly copy the results back! Might need to take the ref
off oup in test..
Thanks for the pointers, Ellery.
What was RuD? RubyD?
There may be more efficient purer ways of doing this, but I
don't wish to spend time learning Excel internals/object
models, and I know my route will work reasonably well).
ActiveX is not internal to Excel. Being a generic component
interface, it's available over almost all microsoft technologies.
True, but how does that help me right a function I can call from
a spreadsheet? I had understood ActiveX was for scripting the
app, but not so useful if you want to write a function that can
be entered in a cell formula. So it seems to me that ActiveX
doesn't create an obvious way to avoid writing a C API wrapper
and then VBA wrapper around that.
optimization / benchmark tips a good topic for wiki ?
Replacing import core.stdc.math with import std.math in the D example increases the avg runtime from 19.64 to 23.87 seconds (~20% slower) which is consistent with OP's statement. + GDC/LDC vs DMD + nobounds, release Do you think we should start a topic on D wiki front page for benchmarking/performance tips to organize peoples' experience of what works? I took a quick look and couldn't see anything already. And it seems to be a topic that comes up quite frequently (less on forum than people doing their own benchmarks and it getting picked up on reddit etc). I am not so experienced in this area otherwise I would write a first draft myself. Laeeth
Data frames in D?
I thought about it once but quickly abandoned the idea. The primary reason was that D doesn't have REPL and is thus not suitable for interactive data exploration. The quick compile times could allow interactive data exploration I agree with other posters that a D REPL and interactive/visualization data environment would be very cool, but unfortunately doesn't exist. Batch computing is more practical, but REPLs really hook new users. I see statistical computing as a huge opportunity for D adoption. (R is just super-ugly and slow, leaving Python + its various native-code cyborg appendages as the hot new stats environment). There are tons of ways of accomplishing the same thing in D, but as far as I know there isn't a standard at this point. A statically typed dataframe is, at minimum, just a range of structs -- even more minimally, a bare *array* of structs, or alternatively just a 2-D array in a thin wrapper that provides access via column labels rather than indexes. You can manipulate these ranges with functions from std.range and std.algorithm. Missing or N/A data is a common issue, and can be represented in a variety of ways, with integers being the most annoying since there is no built-in NaN value for ints (check out the Nullable template from std.typecons). Supporting features like having *both* rows and columns are accessible via labels rather than indexes requires a little bit more wrapping. We have a NamedMatrix class at my workplace for that purpose. It's easy to overload the index operator [] for access, * for matrix multiplication, etc. CSV loads can be done with std.csv; unfortunately there's no corresponding support in that module for *writing* CSV (I've rolled my own). At my workplace we also have a MysqlConnection class that provides one-liner loading from a SQL query into minimalist, range-of-structs dataframes. Beyond that, it really depends on how you want to manipulate the dataframes. What specific things do you want to do? If you've got an idea, I could work up some sample code. So yes, there are people doing it in The Real World. Unfortunately my colleagues don't have a nice, tidy, self-contained DataFrame module to share (yet). But having one would be a great thing for D. The bigger problem though is matching the huge 3rd-party stats libraries (like CRAN for R). Since we do have an interactive shell (the pastebin), and now bindings and wrappers for hdf5 (key for large data sets) and basic seeds for a matrix library, should we start to think about what would be needed for a dataframe, and the best way to approach it, starting very simply? One doesn't need to have a comparable library to R for it to start being useful in particular use cases. Pandas and Julia would be obvious potential sources of inspiration (and it may be that one still uses them to call out to D in some cases), but rather than trying to just port pandas to D, it seems to make sense to ask how one should do it from scratch to better suit D. Laeeth.
Re: Data frames in D?
REPLs are over-hyped and have become a fashion touchstone that few dare argue against for fear of being denounced as un-hip. REPLs have their place, but in the main are nowhere near as useful as people claim. IPython Notebooks on the other hand are a balance between editor/execution environment and REPL that really has a lot going for it. Fair argument against an earlier poster but from my perspective, all I meant is that the absence of a shell is not a good reason to write off D for exploring data. Because there is a shell already that could be developed, and because one can call D from python / Julia in a notebook. Stats folks using R, love R and hate Python. Stats folk using Python, love Python and hate R. In the end it's all about what you know and can use to get the job done. To be frank (as in open rather than Jill), D hasn't got the infrastructure to compete with either R or Python and so is a non-starter in the data science arena. About the future you may or may not be right. (Whether it is commercially interesting to run workshops in D for stats people is certainly a interesting question. However given the ways that technology unfolds it may be that it is less relevant for the question I am most interested today in answering). I want to do things in D myself, and I would find a data frame helpful. I understand you don't program much in D these days, and that's a reasonable decision, but for those who want to use it to do quantish things with dataframes, perhaps we could think about how to approach the problem. And having weighed your warnings, if you have any suggestions on how best to implement this, I would be open to these also. Laeeth.
Data Frames in D - let's not wait for linear algebra; useful today in finance and Internet of Things
On Friday, 26 December 2014 at 21:31:00 UTC, aldanor wrote: On Wednesday, 25 September 2013 at 03:41:36 UTC, Jay Norwood wrote: I've been playing with the python pandas app enables interactive manipulation of tables of data in their dataframe structure, which they say is similar to the structures used in R. It appears pandas has laid claim to being a faster version of R, but is doing so basically limited to what they can exploit from moving operations back and forth from underlying cython code. Has anyone written an example app in D that manipulates dataframe type structures? Pandas has numpy as backend which does a lot of heavy lifting, so first things first -- imo D needs a fast and flexible blas/lapack-compatible multi-dimensional rectangular array library that could later serve as backend for pandas-like libraries. I don't believe I agree that we need a perfect multi-dimensional rectangular array library to serve as a backend before thinking and doing much on data frames (although it will certainly be very useful when ready). First, it seems we do have matrices, even if lacking in complete functionality for linear algebra, and the like. There is a chicken and egg aspect in the development of tools - it is rarely the case that one kind of tool necessarily totally precedes another, and often complementarities and dynamic effects between different stages. If one waits till one has everything one needs done for one, one won't get much done. Secondly, much of the kind of thing Pandas is useful for is not exactly rocket science from a quantitative perspective, but it's just the kinds of thing that is very useful if you are thinking about working with data sets of a decent size.The concepts seem to me to fit very well with std.algorithm and std.range, and can be thought of as just as way to bring out the power of the tools we alreaady have when working with data in the world as it is. See here for an example of just how simple. Remember Excel pivottables? http://pandas.pydata.org/pandas-docs/stable/groupby.html Thirdly, one of the reasons Pandas is popular is because it is written in C/Cython and very fast. It's significantly faster than Julia. One might hit roadblocks down the line when it comes to the Global Interpreter Lock and difficulty of processing larger sets quickly in Python, but at least this stage is fast and easy. So people do care about speed, but they also care about the frictions being taken away, so that they can spend their energies on addressing the problem at hand. Ie a dataframe will be helpful, in my view. Processing of log data is a growing domain - partly from internet, but also from the internet of things. See below for one company using D to process logs: http://venturebeat.com/2014/11/12/adroll-hits-gigantic-130-terabytes-of-ad-data-processed-daily-says-size-matters/ http://tech.adroll.com/blog/data/2014/11/17/d-is-for-data-science.html A poster on this forum is already using D as a library to call from R (from Reddit), which brings home the point that it isn't necessary for D to be able to do every part of the process for it to be able to take over some of the heavy work. [–]bachmeier 6 points 1 month ago I call D shared libraries from R. I've put together a library that offers similar functionality to Rcpp. I've got a presentation showing its use on Linux. Both the presentation and library code should be made available within the next couple of days. My library makes available the R API and anything in Gretl. You can allocate and manipulate R objects in D, add R assert statements in your D code, and so on. What I'm working on now is calling into GSL for optimization. These are all mature libraries - my code is just an interface. It's generally easy to call any C library from D, and modern Fortran, which provides C interoperability, is not too much harder. See here, for just one use case in the internet of things. They don't use D, but maybe they should have. And it shows an example where perhaps at least log processing could easily be handled by what we have with a few small additional data structures - even if people use outside libraries for the machine learning part. http://www.forbes.com/sites/danwoods/2014/11/04/how-splunk-caught-wall-streets-eye-by-taming-the-messy-world-of-iot-data/3/ By using Splunk software, Hrebek said that his division’s leader product is able to offer customers a real-time view of operations on a train and to use machine learning to suggest optimal strategies for driving trains along various routes. Just shaving a small percentage off of fuel costs can mean huge savings for a railroad. Why Doesn’t BI Work for the IoT? In both of the use cases just mentioned, for years, existing business intelligence technology had been applied to the problem of making sense of the data with little success. The problem is not that that it is impossible to
Re: Data Frames in D - let's not wait for linear algebra; useful today in finance and Internet of Things
Russell: I think we are agreeing. Very lightweight editor and executor of code fragments is as good, if not better, that the one line REPL. Yes - the key for me is that the absence of a shell is by no means a reason to say that D is not suited to this task. One may wish to refine what exists, but that is another question entirely. Part of the problem here is tribalism. Most data science people want to use the same tools that other data science people use, even though the issue is to differentiate themselves. Yes - we are answering two different questions. I could not care less about persuading anyone en masse in a broad sector, those who think of themselves as being 'data scientists' included. It's silly, in my view, to think of it as an established field very distinct from others, and with a fixed way of doing things. If for no other reason that things are in flux and the sector is growing quickly, which means that there is room for many different approaches, and it is premature to think the popularity of approach X or Y today means that approach 'D' can't be productive tomorrow. But as I said, I am less convinced in persuading anyone, and rather more concerned with getting a basic data frame in D up and running because I could certainly use it, and the hard work has been done already. The basics should be an evening's work for an advanced D hacker, but it will probably take me longer than that. In any case, since nobody else has come forward, I will keep working away at it. A BLAS library is certainly a precusor, as is very good data visualization tools, graphs, diagrams etc. Perhaps a prerequisite to D being seen as a contender, but I don't see how it's a prerequisite just to have a dataframe, which is really a very simple yet incredibly useful thing. Go has masses of people putting a lot of effort into Web. It's not the ideas, it's the number of people getting on board and doing things. Also about the quality of the people. (I have no view about Go, but have a very positive view on D). When things get big there is a danger they get cluttered. That's one blessing for D. To get some traction in any of these areas, finance data analysis and model building, or systems activity, it is all about people doing it, publicizing it and making things available for others to use. Yes - so do you have any thoughts on what a data frame structure should look like? I am trying to do and after that will make available. But it needs to be better than Julia in some way that makes others sit up and take notice. There has to be the ability to create some hype. Don't care ;) This concept of what is your edge is not my cup of tea because I do not see the world in those terms. Something of high quality that's highly productive will over time stand a decent chance of becoming more widely adopted, whereas trying to force it into some kind of marketing framework can prove counterproductive. Right now, the main thing I care about is solving the problem at hand, because if it solves my problem well then I am pretty sure it will be useful to others too, and be so better than if one had adopted a more consciously 'commercial/marketing' mindset. I would post the dataframe skeleton here, but it's too embarassing right now and want to read the std.variant library to see what tricks I can learn. (A data series seems kind of like a variant, but with every cell the same type). Obviously in some cases the data frame type is defined at compile time, like a struct, and that's easy. But if you are loading from a file you need to be able to have dynamic typing for the column. I don't believe I agree that we need a perfect multi-dimensional rectangular array library to serve as a backend before thinking and doing much on data frames (although it will certainly be very useful when ready). Also, if there is a ready made C or C++ library that can be made use of, do it. Well, the hard parts of arrays themselves (and it's not that fiendishly hard, I would think) seem to need to be tightly integrated with the language, so I don't see how a C/C++ library will help so much. For the linear algebra, yes... hyping it up. I recently discovered a number of hedge funds work solely on moving average based algorithmic trading. NumPy, SciPy and Pandas all have variations on this basic algorithm. Well, having worked for more or less quanty hedge funds since 98, I would think it unlikely that anyone depends only on moving averages although basic old-school trend-following certainly does work - it is just a hard sell to herding institutional investors, and does not fit very well with the concept of a 'career'. (You have to be able to see the five years of subdued returns since 2009 as just part of the cycle, which indeed may be the correct view when one sees markets as a natural phenomenon, but is not the view of asset allocators, or talented
Re: Data Frames in D - let's not wait for linear algebra; useful today in finance and Internet of Things
On Saturday, 27 December 2014 at 16:41:04 UTC, Russel Winder via Digitalmars-d-learn wrote: On Sat, 2014-12-27 at 15:33 +, Laeeth Isharc via Digitalmars-d-learn wrote: […lots of agreed uncontentious stuff :-) …] You write as if Christensen's book The Innovator's Dilemma had never been written, and nor had it been a standard textbook in business schools for some years. You may have good arguments as to why he is wrong, or why it doesn't apply to D, but you haven't set them out, as far as I am aware. In the post-production world as I know it (Nuke, etc.) The C++/Python combination has never failed to be adequate to the innovation demanded by film makers. In the image processing world the C++/Lua combination has never failed to adapt to the innovation needed by photograph tinkerers. My point was really that the customers have never found an innovative need that the extant platforms couldn't provide. I felt this was somewhat different to the Christensen argument. On the other hand, I may have missed the point… No matter how plugged in a person may be, it is impossible to be aware of everything that is going on, especially in exactly the kind of domains Christensen talks about - ones that aren't by any standard important in a spot sense to the bigger picture, but that critically provide a quiet relatively uncontested niche for the seeds of something to unfold until it is ready to break out into the broader world. So I think the point is that one shouldn't be bothered one jot by the disinclination of the people you know to want to use D, particularly since you are so plugged in to all these other worlds (and being an insider in a sense that matters today has an opportunity cost because it means one is not spending time and attention speaking to non insiders as much at that instant). New growth will come from the fringes. I think one should be very worried if the Adam Ruppe of the world would start to say D sucks - nice idea, but just not expressive enough for me, and I am switching back to Ruby and Python. Because that would indicate a loss of ground in the home niche. But somehow I don't think so...! And meantime quietly things continue to develop. What matters is not the challenges one faces, but how one deals with them. An outpouring of frustration in recent days, and the result is we are going to get better docs, better examples, and who knows what else. That's a sign of health. Will post code I have in a few days. Laeeth.
Re: Data Frames in D - let's not wait for linear algebra; useful today in finance and Internet of Things
On Monday, 29 December 2014 at 04:08:58 UTC, Vlad Levenfeld wrote: Laeeth - I am not sure exactly what your needs are but I have a fairly complete solution for generic multidimensional interfaces (template-based, bounds checked, RAII-ready, non-integer indices, the whole shebang) that I have been building. Anyway I don't want to spam the forum if I've missed the point of this discussion, but perhaps we could speak about it further over email and you could give me your opinion? I'm at vlevenf...@gmail.com Hi Vlad. Thanks v much for getting in touch. Your work sounds very interesting. I will drop you a line in coming days. Happy new year. Laeeth.
Re: Passing string literals to C
Argh - no way to edit.
What's best practice here?
D strings are not null-terminated.
===
cpling.c
char* cpling(char *s)
{
s[0]='!';
return s;
}
===
dcaller.d
extern(C) char* cpling(char* s);
void callC()
{
writefln(%s,fromStringz(cpling(hello\0)));
}
or
void callC()
{
writefln(%s,fromStringz(cpling(toStringz(hello;
}
===
am I missing a better way to do this?
Passing string literals to C
What's best practice here?
D strings are not null-terminated.
char* cpling(char *s)
{
So toString(This i
Checking C return results against NULL
Am I missing a more agreeable way to check the return value of a C function against NULL. It's fine if it's a char*, but if it returns a pointer to some kind of struct, one has to go through and convert each instance of NULL to a cast of the appropriate return type. Eg cast(funnystruct*)0 if ((tomime_fields.fld_description==cast(char*)0)) throw new Exception(show_part_info: description is NULL);
Re: Passing string literals to C
Thanks for the help. Laeeth
Re: Checking C return results against NULL
Thanks for the help. Laeeth
Re: Can the order in associative array change when keys are not midified?
On Thursday, 1 January 2015 at 18:58:04 UTC, Andrej Mitrovic via Digitalmars-d-learn wrote: On 1/1/15, Tobias Pankrath via Digitalmars-d-learn digitalmars-d-learn@puremagic.com wrote: You could implement an OrderedMap!(Key, Value) via RedBlackTree!(Tuple!(Key, Value), (a,b) = a[0] b[0]). We could add this as an alias into Phobos or perhaps as just a documentation line on the website. V good point. This is one of those small frictions that cumulatively raises the cost of adoption for newcomers who are not by nature hackers but want to get stuff done, and yet could be valuable members of the D community over time. For whatever reason, there is some resistance for many people to ask for help in a forum and it is easy to be overwhelmed and give up. Modern people don't have much tolerance for pain, even though that's an essential part of learning and being a developer. And compare python where you have OrderedDict and a search will find many examples of how to use it. I agree with comments about readability of library functions, but maybe also worth extending hints and tips for common patterns / code fragments. What is the protocol for contributing to the wiki? Can one just edit it ?
Re: simple question about using dub to import CyberShadow's aeutils
On Saturday, 3 January 2015 at 12:08:16 UTC, Vladimir Panteleev wrote: On Saturday, 3 January 2015 at 11:58:48 UTC, Laeeth Isharc wrote: import ae.utils; ae.utils is a package, perhaps you meant to import ae.utils.xml? aha. schoolboy error on my part. thank you for your help, and especially for writing such a useful library and sharing it with the world. Laeeth.
simple question about using dub to import CyberShadow's aeutils
Hi.
I would like to use the XML parser from CyberShadow's ae.utils -
I am building a tool to index RSS feeds in elasticsearch
(something like rssriver but with more complete functionality).
I am using dub to build the code.
So far I just have an empty boilerplate app.d with the line
import ae.utils;
Using the suggestion from code.dlang.org:
{
...
dependencies: {
ae: =1.0.1
}
}
It cannot find ae/utils.d If I try replacing the = with or
just ~master it still does not build. Error text below this
message. I suppose this might be because dub is designed to
import a single package, and not just part of the hierarchy.
But if I change the import to just plain import ae, I still have
problems.
What am I doing wrong? I could just copy the file over, but I
would like to try to do it properly.
Do let me know if you do not see this as the right forum to ask
about dub.
Thanks.
Laeeth.
Selected package openssl 1.0.0+1.0.0e does not match the
dependency specification in package ae (=1.1.3+1.0.1g
(optional)). Need to dub upgrade?
Target vibe-d 0.7.21 is up to date. Use --force to rebuild.
Building rsselastic ~master configuration application, build
type debug.
Compiling using dmd...
source/app.d(1): Error: module utils is in file 'ae/utils.d'
which cannot be read
import path[0] = source/
import path[1] = ../../../root/.dub/packages/ae-1.0.1
import path[2] = ../../../root/.dub/packages/openssl-1.0.0_1.0.0e
import path[3] = ../../../root/.dub/packages/vibe-d-0.7.21/source/
import path[4] = ../../../root/.dub/packages/libevent-2.0.1_2.0.16
import path[5] = /usr/include/dmd/phobos
import path[6] = /usr/include/dmd/druntime/import
FAIL
.dub/build/application-debug-linux.posix-x86_64-dmd_2067-6DD7C73322D93ABC7BBACEAF4B4E0E3D/
rsselastic executable
Error executing command run: dmd failed with exit code 1.
===
Selected package openssl 1.0.0+1.0.0e does not match the
dependency specification in package ae (=1.1.3+1.0.1g
(optional)). Need to dub upgrade?
Target vibe-d 0.7.21 is up to date. Use --force to rebuild.
Building rsselastic ~master configuration application, build
type debug.
Compiling using dmd...
source/app.d(1): Error: module ae is in file 'ae.d' which cannot
be read
import path[0] = source/
import path[1] = ../../../root/.dub/packages/ae-1.0.1
import path[2] = ../../../root/.dub/packages/openssl-1.0.0_1.0.0e
import path[3] = ../../../root/.dub/packages/vibe-d-0.7.21/source/
import path[4] = ../../../root/.dub/packages/libevent-2.0.1_2.0.16
import path[5] = /usr/include/dmd/phobos
import path[6] = /usr/include/dmd/druntime/import
FAIL
.dub/build/application-debug-linux.posix-x86_64-dmd_2067-6DD7C73322D93ABC7BBACEAF4B4E0E3D/
rsselastic executable
Error executing command run: dmd failed with exit code 1.
Other libraries - web site link, and other thoughts
At the moment it goes straight go code.dlang.org, which may be a bit overwhelming if you have just arrived at dlang.org and want to see what it can do. Is it worth changing to the library wiki write up page on libraries? And making sure link to code.dlang.org is prominent, saying code.dlang.org is a registry of D libraries hosted on github. There are currently XYZ libraries, of which ABC are under active development. At the moment, the list of libraries on the wiki is fairly complete, but not oriented towards answering the question of how do I do stuff. It would be nice to have a set of pages on this that mines the gold from the forums and makes it easy to find. Eg how do I navigate html or xml?. Use dom.d or ae, and here are some simple programs. Compare python community doc situation to ours. (We are leaner, but that doesn't mean we can't do a better job), Also say that if you cannot find the library you need, you should know that it is easy to interface to all C libraries (click for details), and that it is possible and fairly easy to interface to many C++ libraries (click here). p0nce and others are doing great work on D idioms, but what is missing is pointers to examples of larger programs. I know it is common sense if you are experienced to read the source of other projects to learn, but is it worth making this easy by having in the section on learning D some pointers to some model D code bases. The FUD crowd on slashdot always brings up the Tango / Phobos split, but this is not prominently addressed on the web site. Should this not be answered in the main FAQ? The compilation speed of dmd (and somewhat ldc) is a major advantage. It is stunning in outright terms, and almost inconceivable if you come from the world of c++. It really matters because it allows you to shift styles to iterative development. So it should be featured on the front page, and there should be quantitative benchmarks (because that's what modern people like) comparing dmd to other language compilers as well as pointing out how quickly phobos compiles. A YouTube video may make it more concrete. And there should be quotes from authoritative users of D talking about how useful they find quick compilation and iterative development. People are lazy. Or more charitably, not wealthy in terms of attention and time in the modern world. The documentation is often written from the point of view of the compiler and library writer, and this is not the point of view of the user, who wants to know what he can do, and how he can do it. In the interfacing to C tutorial, how about starting off by getting to the point. Have the following section at the beginning, and maybe even make it crisper by saying that D can utilise all existing C libraries, and all that needs to be done is to translate the header files to D. Using Existing C Libraries Since D can call C code directly, it can also call any C library functions, giving D access to the smorgasbord of existing C libraries. To do so, however, one needs to write a D interface (.di) file, which is a translation of the C .h header file for the C library into D. For popular C libraries, the first place to look for the corresponding D interface file is the Deimos Project. If it isn't there already, and you write one, please contribute it to the Deimos Project. --- In the spirit of accommodating peoples' attention poverty, why not say at the top that there are translation tools for objective C and C. Click here to download dstep binaries (linux / win etc), and here to go to the github dstep site. I confess that I struggled for ages unsuccessfully to compile dstep on my platform without realising that the binaries were available for download. Similarly for C++, the web site starts off completely on the wrong note. While D is fully capable of interfacing to C, its ability to interface to C++ is much more limited. There are three ways to do it: I hear much more limited, and think I am not going to bother with this (if I have not yet committed to the language). Whereas in actual fact, as Walter pointed out in his talk, nobody else can interface with C++ and it is a marvel to be able to do so at all, let alone in a way that is much more complete than the docs suggest. And it sounds like this phrasing is in any case outdated. Change to something like: Interfacing to C++ is a challenging endeavour for any language, but it is a challenge that must be met in order to take advantage of the large set of existing libraries and code bases in C++. It is therefore a priority for 2015 to implement an ability to link to a set of C++ that will be complete for most use cases. Walter Bright spoke recently at Microsoft on this topic, and the reader may find the following forum threads of interest. As things stand, the following features are complete:..., whilst there are limitations with respect to:
Re: D Beginner Trying Manual Memory Management
I don't think you've read h5py source in enough detail :) You're right - I haven't done more than browsed it. It's based HEAVILY on duck typing. There is a question here about what to do in D. On the one hand, the flexibility of being able to open a foreign HDF5 file where you don't know beforehand the dataset type is very nice. On the other, the adaptations needed to handle this flexibly get in the way when you are dealing with your own data that has a set format and where recompilation is acceptable if it changes. Looking at the 'ease' of processing JSON, even using vibed, I think that one will need to implement both eventually, but perhaps starting with static typing. In addition, it has way MORE classes than the C++ hierarchy does. E.g., the high-level File object actually has these parents: File : Group, Group : HLObject, MutableMappingWithLock, HLObject : CommonStateObject and internally the File also keeps a reference to file id which is an instance of FileID which inherits from GroupID which inherits from ObjectID, do I need to continue? Okay - I guess there is a distinction between the interface to the outside world (where I think the h5py etc way is superior for most uses) and the implementation. Is not the reason h5py has lots of classes primarily because that is how you write good code in python, whereas in many cases this is not true in D (not that you should ban classes, but often structs + free floating functions are more suitable). PyTables, on the contrary is quite badly written (although it works quite well and there are brilliant folks on the dev team like francesc alted) and looks like a dump of C code interweaved with hackish Python code. Interesting. What do you think is low quality about the design? In h5py you can do things like file[/dataset].write(...) -- this just wouldn't work as is in a strictly typed language since the indexing operator generally returns you something of a Location type (or an interface, rather) which can be a group/datatype/dataset which is only known at runtime. Well, if you don't mind recompiling your code when the data set type changes (or you encounter a new data set) then you can do that (which is what I posted a link to earlier). It depends on your use case. It's hard to think of an application more dynamic than web sites, and yet people seem happy enough with vibed's use of compiled diet templates as the primary implementation. They would like the option of dynamic ones too, and I think this would be useful in this domain too, since one does look at foreign data on occasion. One could of course use the quick compilation of D to regenerate parts of the code when this happens. Whether or not this is acceptable depends on your use case - for some it might be okay, but obviously it is no good if you are writing a generic H5 browser/charting tool. So I think if you don't allow static dataset typing it means the flexibility of dynamic typing gets in the way for some uses (which might be most of them), but you need to add dynamic typing too. Shall we move this to a different thread and/or email, as I am afraid I have hijacked the poor original poster's request. On the refcounting question, I confess that I do not fully understand your concern, which may well reflect a lack of deep experience with D on my part. Adam Ruppe suggests that it's generally okay to rely on a struct destructor to call C cleanup code. I can appreciate this may not be true with h5 and, if you can spare the time, I would love to understand more precisely why not. Out of all of them, only the dataset supports the write method but you don't know it's going to be a dataset. See the problem? In this case I didn't quite follow. Where does this fall down ? void h5write(T)(Dataset x, T data) I have your email somewhere and will drop you a line. Or you can email me laeeth at laeeth.com. And let's create a new thread. Laeeth.
Re: idiomatic D: what to use instead of pointers in constructing a tree data structure?
On Tuesday, 13 January 2015 at 17:41:53 UTC, Tobias Pankrath wrote: On Tuesday, 13 January 2015 at 17:19:42 UTC, Laeeth Isharc wrote: The GC is allowed to move structs around, as I undestand it. Under what circumstances do I get into trouble having a pointer to them? None, a GC that moves structs around must update every pointer afterwards and as far as I know, the standard GC doesn't do that (moving things around). You may not have a pointer inside a struct that points to the struct itself. This allows the compiler to elide some copies. Got it - thanks.
Re: vibe.d Subdirectory?
On Thursday, 15 January 2015 at 14:38:16 UTC, Steven Schveighoffer wrote: On 1/14/15 11:31 AM, Laeeth Isharc wrote: To be very clear: in the simple case when you compile your vibe application from multiple source files and diet templates etc, and you will end up with an executable. This can act as a server directly, or you can make it an internal server on localhost called by php, or you can call it from nginx as a CGI type application depending on the URL. Pardon me for asking kind of a side question, but is there any online information on how to do this? I am contemplating migrating my in-house php-based application to vibe.d, but I wondered if I have to do it whole-sale (not much gratification until the whole thing is done) or piecemeal (then some parts are php, some are vibe.d, until whole thing is ported). I thought I could do the latter, but I don't know how to interact apache/php with vibe.d. The biggest question I have is how vibe.d and php can share session data. But I also have no idea which server should be in front, and how to call one another ;) It may be that you have a reason for going with apache, but in case you have not looked into it (or the situation has changed since you last looked), you might consider nginx as a front end given better stability and performance. If you make D read the PHP authentication then you need to write a new auth scheme when you have gotten rid of PHP. So it might be worth at some point considering moving this to an external service called by both D and PHP when you can do so without breaking anything. This might be worth a read: http://merbist.com/2012/04/04/building-and-implementing-a-single-sign-on-solution/
Re: Getting a safe path for a temporary file
On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote: Is it currently possible to get the path to a safe temporary file, i.e. one that is guaranteed to be freshly created and will not override another existing file? There's `std.file.tempDir`, which doesn't create a unique file. Then there's `std.stdio.tmpfile()`, which does, but it returns a `File` object, and its `name` property is `null`. Did I miss something? IMO this is very import functionality. One use case is passing these names as command line arguments to an external program that doesn't support stdin/stdout. I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html
Re: Getting a safe path for a temporary file
On Saturday, 17 January 2015 at 16:55:42 UTC, Marc Schütz wrote: On Saturday, 17 January 2015 at 14:37:00 UTC, Laeeth Isharc wrote: On Saturday, 17 January 2015 at 13:47:39 UTC, Marc Schütz wrote: Is it currently possible to get the path to a safe temporary file, i.e. one that is guaranteed to be freshly created and will not override another existing file? There's `std.file.tempDir`, which doesn't create a unique file. Then there's `std.stdio.tmpfile()`, which does, but it returns a `File` object, and its `name` property is `null`. Did I miss something? IMO this is very import functionality. One use case is passing these names as command line arguments to an external program that doesn't support stdin/stdout. I agree that it would be useful. This is what I used, although there may be a better option: http://dlang.org/phobos/std_uuid.html Nice idea, but it still allows for intentional collision attacks :-( The only really safe solution is one that generates (probably) unique names, then opens the file with O_EXCL|O_CREAT (or whatever other means the OS provides), and if it fails, retries with a different name. `std.stdio.tmpfile()` already does that (it uses `tmpfile(3)` under the hood), but doesn't allow access to the name. I don't follow why a collision attack is applicable in this case. Your stage 1 of generating unique names: how is this different from using a random uuid?
Re: casting SysTime to ubyte[]
I really wouldn't advise doing that. SysTime contains a long which represents the time in hnsecs since midnight, January 1st, 1 A.D., and that could be written to a file quite easily. But it also contains a reference to a TimeZone object, so what you're doing would just be writing its address to disk, which wouldn't do you any good at all, since that's specific to each run of the program, even assuming that the object exists in both runs of the program (which it would for UTC or LocalTime but not for user-constructed time zones). So, writing the stdTime (horrible name, I know) property to disk would work just fine (that's the hnsecs as a long), but you're going to have to do something smarter than that if you want to retain the time zone. And you're not going to want to try and simply cast a SysTime to a ubyte[] and do anything practical with that regardless. - Jonathan M Davis Thanks for this. I still with my C habits had the idea the time would just be a flat struct. So in this case better to write it as a string, which is what I have already done. I just wondered why the other approach didn't work, and now I understand.
Re: vibe.d Subdirectory?
Actually I want to serve some JSON packed weather data (heck I also wrote my Global Climate Model partially in D and in C) - so I guess, I can use vibe.d to build a cgi. Cool. Do you incorporate the influence of solar activity via galactic rays / cloud formation and via volcanic activity? (This is not my field, but I am looking for a model that does). However I also want to recieve some userinput, and parse it - and sometimes, php is simpler, because for a very small parsing task you dont want to worry abotu types and all. Does vibe.d support any HTML Preprocessor in the first place? If not PHP then Neco? Falcon? So that I do not always have to access my vibe.d based app, but I can simply change the page itself a bit? It won't be quicker to write the parsing code in D if you are new to the language, but it is still pretty quick when you know how. There are regexes etc too. What Ketmar says. For a robust application it might be better to put it behind a proxy like nginx anyway. In which case you can use your vibed application to serve up parts of your site and php for others. Eg user submits a form and the result is posted to a url that nginx routes to your vibed app. You could dispense entirely with the web server and do the whole thing in vibed, but you don't need to. Although its a framework, you don't have to do much to turn it into a web server if you look at the documentation on the vibed page. The forum there is also quite helpful. Laeeth
Re: D Beginner Trying Manual Memory Management
In the hierarchy example above (c++ hdf hierarchy link), by
using UFCS to implement the shared methods (which are achieved
by multiple inheritance in the c++ counterpart) did you mean
something like this?
// id.d
struct ID { int id; ... }
// location.d
struct Location { ID _id; alias _id this; ... }
// file.d
public import commonfg; // ugh
struct File { Location _location; alias _location this; ... }
// group.d
public import commonfg;
struct File { Location _location; alias _location this; ... }
// commonfg.d { ... }
enum isContainer(T) = is(T: File) || is(T : Group);
auto method1(T)(T obj, args) if (isContainer!T) { ... }
auto method2(T)(T obj, args) if (isContainer!T) { ... }
I guess two of my gripes with UFCS is (a) you really have to
// another hdf-specific thing here but a good example in
general is that some functions return you an id for an object
which is one of the location subtypes (e.g. it could be a File
or could be a Group depending on run-time conditions), so it
kind of feels natural to use polymorphism and classes for that,
but what would you do with the struct approach? The only thing
that comes to mind is Variant, but it's quite meh to use in
practice.
Void unlink(File f){}
Void unlink(Group g){}
For simple cases maybe one can keep it simple, and despite the
Byzantine interface what one is trying to do when using HDF5 is
not intrinsically so complex.
Re: D Beginner Trying Manual Memory Management
struct File { Location _location; alias _location this; ... }
// group.d
public import commonfg;
struct File { Location _location; alias _location this; ... }
// commonfg.d { ... }
enum isContainer(T) = is(T: File) || is(T : Group);
auto method1(T)(T obj, args) if (isContainer!T) { ... }
auto method2(T)(T obj, args) if (isContainer!T) { ... }
I guess two of my gripes with UFCS is (a) you really have to
// another hdf-specific thing here but a good example in
general is that some functions return you an id for an object
which is one of the location subtypes (e.g. it could be a
File or could be a Group depending on run-time conditions),
so it kind of feels natural to use polymorphism and classes
for that, but what would you do with the struct approach? The
only thing that comes to mind is Variant, but it's quite meh
to use in practice.
Void unlink(File f){}
Void unlink(Group g){}
For simple cases maybe one can keep it simple, and despite the
Byzantine interface what one is trying to do when using HDF5
is not intrinsically so complex.
So your solution is copying and pasting the code?
But now repeat that for 200 other functions and a dozen more
types that can be polymorphic in weirdest ways possible...
If you are simply have a few lines calling the API and the
validation is different enough for file and group (I haven't
written unlink yet) then why not (and move proper shared code out
into helper functions). The alternative is a long method with
lots of conditions, which may be the best in some cases but may
be harder to follow.
I do like the h5py and pytables approaches. One doesn't need to
bother too much with the implementation when using their library.
However, what I am doing is quite simple from a data perspective
- a decent amount of it, but it is not an interesting problem
from a theoretical perspective - just execution. Now if you are
higher octane as a user you may be able to see what I cannot.
But on the other hand, the Pareto principle applies, and in my
view a library should make it simple to do simple things. One
can't get there if the primary interface is a direct mapping of
the HDF5 hierarchy, and I also think that is unnecessary with D.
But I very much appreciate your work as the final result is
better for everyone that way, and you are evidently a much longer
running user of D than me. I never used C++ as it just seemed
too ugly! and I suspect the difference in backgrounds is shaping
perspectives.
What do you think the trickiest parts are with HDF5? (You
mention weird polymorphism).
Laeeth
Re: vibe.d Subdirectory?
On Wednesday, 14 January 2015 at 11:40:26 UTC, seany wrote: I am new to vibe.d and plying a bit with it. I notice, that in case of Apache, there is a root directory, often by default under /var/www or /srv/http (resp. ftp) if you are using linux, and then every time the client sends a request, apache looks in to the root directory, deduces the subdirectory from the URI, and pulls the page up from there. Then it parses the PHP / other scripting commands, and prints the HTML as is, before serving it over http. I want to know the equivalent of all these in vibe.d. The website has a documentation, but all what I find is that you will need an app.d, in a predefined directory structure. I however, do not understand, what the root directory is going to be. Is it going to be the directry where vibe.d is started? Say, I start my vibe.d under /server, then I have an app.d under /server/a/app.d and /server/b/app.d Do I access them via http://top.level.domain/a, resp /b, and app.d is like index.html / index.php which vibe.d looks for by default, or do i have to use http://top.level.domain/a/app.d To be very clear: in the simple case when you compile your vibe application from multiple source files and diet templates etc, and you will end up with an executable. This can act as a server directly, or you can make it an internal server on localhost called by php, or you can call it from nginx as a CGI type application depending on the URL. The vibed directory layout is of no relevance after compilation (unless you put your static data there).
Re: Explicit Slicing of std.container.Array
On Saturday, 24 January 2015 at 13:11:33 UTC, Nordlöw wrote:
Is there a reason why std.container.Array have to be explicitly
sliced before being processed by range algorithms such as
filter typically as
import std.container: Array;
Array!int a;
foreach (e; a[].filter!true) {}
?
Does memory allocation play a role?
I would like to see it be used in replace of D builtin arrays
without any code changes.
One thing to mention in the revised docs in the introduction to
std.algorithms.
Re: Turning Executable into Application?
On Monday, 26 January 2015 at 03:36:32 UTC, Gan wrote: With Xamarin Studio I create a D project and run it. It runs an Executable Unix file through the terminal. How can I turn that into an Application that doesn't open the Terminal? Thanks. Have you tried running your executable from the command line? I suspect Xamarin is just opening a terminal for you as a convenience.
Re: BitArray - incomplete implementation?
To avoid confusion, the below is the code that fits the error
message:
import std.bitmanip;
import std.stdio;
import std.array;
import std.range:chain;
void test()
{
int[] a=[1,2,3,4,5];
int[] b=[5,4,3,2,1];
int[] c = chain(a,b).array; // chain two arrays of int
writefln(%s,c);
}
void test2()
{
BitArray a;
a.init([1,0,1,0]);
BitArray b;
b.init([0,1,0,1]);
BitArray[] d;
d~=a;
d~=b;
BitArray[] c=chain(a,b).array; // cannot chain two bitarrays
BitArray[] c=chain([a],[b]).array; // cannot chain two bitarrays
BitArray[] e=chain(d,d).array; // cannot chain two arrays of
bitarrays
writefln(%s,c);
}
int main(string[] args)
{
test();
test2();
return 1;
}
BitArray - incomplete implementation?
Hi.
Should the following code work?
import std.bitmanip;
import std.stdio;
import std.array;
import std.range:chain;
void test()
{
int[] a=[1,2,3,4,5];
int[] b=[5,4,3,2,1];
int[] c = chain(a,b).array; // chain two arrays of int
writefln(%s,c);
}
void test2()
{
BitArray a;
a.init([1,0,1,0]);
BitArray b;
b.init([0,1,0,1]);
BitArray[] d;
d~=a;
d~=b;
BitArray[] c=chain([a],[b]).array; // cannot chain two bitarrays
BitArray[] e=chain(d,d).array; // cannot chain two arrays of
bitarrays
writefln(%s,c);
}
int main(string[] args)
{
test();
test2();
return 1;
}
bitmanip.d(23): Error: template std.range.chain cannot deduce
function from argument types !()(BitArray, BitArray), candidates
are:
/usr/include/dmd/phobos/std/range.d(2493):
std.range.chain(Ranges...)(Ranges rs) if (Ranges.length 0
allSatisfy!(isInputRange, staticMap!(Unqual, Ranges))
!is(CommonType!(staticMap!(ElementType, staticMap!(Unqual,
Ranges))) == void))
bitmanip.d(24): Error: template std.range.chain cannot deduce
function from argument types !()(BitArray[], BitArray[]),
candidates are:
/usr/include/dmd/phobos/std/range.d(2493):
std.range.chain(Ranges...)(Ranges rs) if (Ranges.length 0
allSatisfy!(isInputRange, staticMap!(Unqual, Ranges))
!is(CommonType!(staticMap!(ElementType, staticMap!(Unqual,
Ranges))) == void))
bitmanip.d(24): Error: declaration bitmanip.test2.c is already
defined
bitmanip.d(25): Error: template std.range.chain cannot deduce
function from argument types !()(BitArray[], BitArray[]),
candidates are:
/usr/include/dmd/phobos/std/range.d(2493):
std.range.chain(Ranges...)(Ranges rs) if (Ranges.length 0
allSatisfy!(isInputRange, staticMap!(Unqual, Ranges))
!is(CommonType!(staticMap!(ElementType, staticMap!(Unqual,
Ranges))) == void))
I cannot seem to concatenate - directly, or using chain - two
bitarrays, or two arrays of bitarrays.
Re: BitArray - incomplete implementation?
Yes, that error is caused by a bug of BitArray(https://issues.dlang.org/show_bug.cgi?id=13806). Having init function broke template constraints of chain(and must break dozen of other templates). pragma(msg, ElementType!(BitArray[])) // prints 'pure nothrow void(bool[] ba)' - ElementType uses init property to determine types. Thanks for this. If I recompile phobos with the pull request (having edited out the init property entirely) then the code mostly works. However, I still can't use join, joiner, or chain on two BitArrays. Eg auto x=joiner(a,b).array; I can chain two arrays of BitArrays, but not join or joiner them. If this is what ought to happen, why?
Re: foreach - premature optimization vs cultivating good habits
Thank you Adam, Bbaz and Ola for the helpful thoughts. I dumped them in a wiki page off the sandbox but needs editing and refining.
Re: foreach - premature optimization vs cultivating good habits
On Friday, 30 January 2015 at 12:55:20 UTC, Adam D. Ruppe wrote: On Friday, 30 January 2015 at 11:55:16 UTC, Laeeth Isharc wrote: As I understand it, foreach allocates when a simple C-style for using an array index would not. foreach is just syntax sugar over a for loop. If there's any allocations, it is because your code had some, it isn't inherit to the loop. The doc definition even lists the translation of foreach to for in the case of ranges explicitly: http://dlang.org/statement.html#ForeachStatement The most likely allocation would be to a user-defined opApply delegate, and you can prevent that by making it opApply(scope your_delegate) - the scope word prevents any closure allocation. Thanks, Adam. That's what I had thought (your first paragraph), but something Ola on a different thread confused me and made me think I didn't understand it, and I wanted to pin it down. The second paragraph is very helpful - appreciate it. Laeeth.
Re: simple dub question - avoiding creating a vibed project
On Monday, 5 January 2015 at 10:46:17 UTC, Rikki Cattermole wrote: On 5/01/2015 11:42 p.m., Laeeth Isharc wrote: Figured out a fix: versions: [VibeCustomMain], It is still mysterious as to why it is pulling in vibed though (I don't import it, and I didn't think ddbc did). https://github.com/mysql-d/mysql-native/blob/master/package.json aha. isn't this a poor default for dub though? ie if your parent project itself does not depend on vibed, the default should be that you take care of main yourself, and it does not try and use vibed's, no ?
simple dub question - avoiding creating a vibed project
Hi.
I am building an example for hibernated (I put a main around the
sample code extract from the website).
How do I stop dub trying to build a vibed project?
Here is my dub.json
{
name: ddbc example,
description: example for DB Connector for D language,
similar to JDBC,
authors: [Vadim Lopatin,Laeeth Isharc],
homepage: https://github.com/buggins/ddbc;,
license: Boost Software License (BSL 1.0),
dependencies: {
mysql-native: =0.0.12,
ddbc: =0.2.16,
},
targetType: executable,
libs-posix: [sqlite3, pq],
libs-windows: [sqlite3, libpq],
copyFiles-windows-x86: [ libs/win32/sqlite3.dll,
libs/win32/libpq.dll, libs/win32/intl.dll ],
sourceFiles-windows-x86 : [ libs/win32/sqlite3.lib,
libs/win32/libpq.lib ],
targetPath: .,
}
Re: simple dub question - avoiding creating a vibed project
Figured out a fix: versions: [VibeCustomMain], It is still mysterious as to why it is pulling in vibed though (I don't import it, and I didn't think ddbc did).
Re: simple dub question - avoiding creating a vibed project
I opened an issue about this last year: https://github.com/mysql-d/mysql-native/issues/44 Thanks. Laeeth.
Any chance of a linux dtoh?
I realize Walter has far better things to work on, but value of having a translation tool is considerable, since it opens up easy access to an enormous range of libraries. It is not much work to do the translation oneself, but in the world as it is small frictions cumulatively have large consequences. Access to libraries is a big advantage of python, and the like. Once you have the wrapper in place, I find D as productive as python, but for now it is still a bit of work first. There is also dstep, but this doesn't seem to compile right now, and I don't have time to figure out why and fix it. (I took a look, and it is not so easy to figure out). Something mambo/tango related. I have raised a bug report. Laeeth
Re: Any chance of a linux dtoh?
On Tuesday, 6 January 2015 at 14:11:19 UTC, Dicebot wrote: dstep is your only realistic chance currently. htod is completely unmaintained, even on Windows. Please report any issues found with it in relevant issue tracker. I got it the wrong way around - yes, I meant htod. I have reported the issue, and author doesn't know what is wrong.
Re: idiomatic D: what to use instead of pointers in constructing a tree data structure?
Small recommendation (apart from the reserved word issue which you fixed): it's generally considered good D style to give structs and classes names that start with capital letters, JustLikeThis. So, I suggest Node rather than node. Very minor point, and of course, your code is yours to style as you wish, but it can be helpful to meet the standard style conventions in order to make it as easy as possible for everyone else to understand. Thanks for the reminder. I use D Style when I am writing properly, but haven't yet internalized it for a quick example and old habits die hard.
Re: Wrapping a C library with its own GC + classes vs refcounted structs
Hi Aldanor. I wrote a slightly longer reply, but mislaid the file somewhere. I guess your question might relate to wrapping the HDF5 library - something that I have already done in a basic way, although I welcome your project, as no doubt we will get to a higher quality eventual solution that way. One question about accurately representing the HDF5 object hierarchy. Are you sure you wish to do this rather than present a flattened approach oriented to what makes sense to make things easy for the user in the way that is done by h5py and pytables? In terms of the actual garbage generated by this library - there are lots of small objects. The little ones are things like a file access attribute, or a schema for a dataset. But really the total size taken up by the small ones is unlikely to amount to much for scientific computing or for quant finance if you have a small number of users and are not building some kind of public web server. I think it should be satisfactory for the little objects just to wrap the C functions with a D wrapper and rely on the object destructor calling the C function to free memory. On the rare occasions when not, it will be pretty obvious to the user and he can always call destroy directly. For the big ones, maybe reference counting brings enough value to be useful - I don't know. But mostly you are either passing data to HDF5 to write, or you are receiving data from it. In the former case you pass it a pointer to the data, and I don't think it keeps it around. In the latter, you know how big the buffer needs to be, and you can just allocate something from the heap of the right size (and if using reflection, type) and use destroy on it when done. So I don't have enough experience yet with either D or HDF5 to be confident in my view, but my inclination is to think that one doesn't need to worry about reference counting. Since objects are small and there are not that many of them, relying on the destructor to be run (manually if need be) seems likely to be fine, as I understand it. I may well be wrong on this, and would like to understand the reasons if so. Laeeth.
endsWith - for a string vs an array of strings
I understand from previous discussion there is some difficulty
over immutability. I did not quite figure out what the solution
was in this case:
import std.array;
import std.string;
import std.stdio;
void main(string[] args)
{
string[] test=[1,two,three!];
auto a=arghtwo.endsWith(test);
writefln(%s,a);
}
This does not compile...
test.d(7): Error: template std.algorithm.endsWith cannot deduce
function from argument types !()(string, string[]), candidates
are:
/usr/include/dmd/phobos/std/algorithm.d(6143):
std.algorithm.endsWith(alias pred = a == b, Range,
Needles...)(Range doesThisEnd, Needles withOneOfThese) if
(isBidirectionalRange!Range Needles.length 1
is(typeof(.endsWith!pred(doesThisEnd, withOneOfThese[0])) : bool)
is(typeof(.endsWith!pred(doesThisEnd,
withOneOfThese[1..__dollar])) : uint))
/usr/include/dmd/phobos/std/algorithm.d(6210):
std.algorithm.endsWith(alias pred = a == b, R1, R2)(R1
doesThisEnd, R2 withThis) if (isBidirectionalRange!R1
isBidirectionalRange!R2
is(typeof(binaryFun!pred(doesThisEnd.back, withThis.back)) :
bool))
/usr/include/dmd/phobos/std/algorithm.d(6237):
std.algorithm.endsWith(alias pred = a == b, R, E)(R
doesThisEnd, E withThis) if (isBidirectionalRange!R
is(typeof(binaryFun!pred(doesThisEnd.back, withThis)) : bool))
Thanks.
Laeeth.
casting SysTime to ubyte[]
import std.datetime;
import std.stdio;
import std.conv;
void main(string[] arg)
{
auto a=Clock.currTime();
auto b=cast(ubyte[])a;
writefln(%s,b);
}
how do i get the time as a binary representation I can write to a
file?
Thanks.
Re: endsWith - for a string vs an array of strings
Thanks for the help to everyone. It seems a common thing to want
to check an array as one may not know the variables at compile
time. Not that it's more than a few lines to do in D. But in
terms of language adoption, small frictions can have large
consequences over time. (Modern people don't have much patience
or sticking power even if viewed rationally the ROI makes these
things trivial. That's also because in a corporate environment
management are not always patient when somebody tries something
new and not yet mainstream). Making the standard library easy
with no sharp edges might have a high payoff over time.
How would one add this note about needing to pass a tuple not an
array to the docs or wiki ? For docs, clone and submit a pull
request? How about wiki?
On Monday, 12 January 2015 at 07:49:16 UTC, Mengu wrote:
On Saturday, 10 January 2015 at 23:32:47 UTC, bearophile wrote:
Laeeth Isharc:
In D there is a feature that allows a function to accept
both an array of items and items,
yes - it is funny there is not an overloading that accepts
arrays
I meant this D feature:
void foo(T)(T[] items...) {
import std.stdio;
items.writeln;
}
void main() {
foo(red, green, blue);
foo([red, green, blue]);
auto a = [red, green, blue];
foo(a);
}
Bye,
bearophile
for the curious, expanding tuples and typetuples on ali's book
is explained at
http://ddili.org/ders/d.en/tuples.html#ix_tuples..expand and at
http://ddili.org/ders/d.en/tuples.html#ix_tuples.TypeTuple,
std.typetuple.
Re: Wrapping a C library with its own GC + classes vs refcounted structs
Laeeth. Thanks for the reply. Yes, this concerns my HDF5 wrapper project; the main concern is not that the memory consumption of course, but rather explicitly controlling lifetimes of the objects (especially objects like files -- so you are can be sure there are no zombie handles floating around). An easy way is to just use scope(exit) to either close the HDF5 object directly, or indirectly call destroy on the wrapper. If you want to make it 'idiot proof', maybe ref counting structs will get you there (at possible cost of small overhead). I personally don't tend to forget to close a file or dataset; its much easier up forget to close a data type or data space descriptor. But struct vs class depends somewhat on how you want to represent the object hierarchy in D, no ? Incidentally there are some nice things one can do using compile time code to map D structs to HDF5 types (I have implemented a simple version of this in my wrapper). A bit more work the other way around if you don't know what's in the file beforehand.
Re: How to prevent sensitive information is displayed when the extension 'exe' is modified to 'txt' on windows?
What you want is some kind of code obfuscation. The easiest thing for you is to use exe compression. It is not going to stop a dedicated attacker, but ordinary people will not be able to extract any information from it. And I guess as an alternative to the utility you linked to, you could use D's ability to run code at compile time to encrypt your sensitive literals during compilation and then decrypt them on program startup.
Re: D Beginner Trying Manual Memory Management
I see, thanks! :) I've started liking structs more and more recently as well and been pondering on how to convert a class-based code that looks like this (only the base class has any data): it's hard to tell by brief description. but having multiple inheritance immediately rings an alarm ring for me. something is very-very-very wrong if you need to have a winged whale. ;-) A real-world example: http://www.hdfgroup.org/HDF5/doc/cpplus_RM/hierarchy.html H5::File is both an H5::Location and H5::CommonFG (but not an H5::Object) H5::Group is both an H5::Object (subclass of H5::Location) and H5::CommonFG H5::Dataset is an H5::Object i see something named CommonFG here, which seems to good thing to move out of hierarchy altogether. bwah, i don't even sure that given hierarchy is good for D. C++ has no UFCS, and it's incredibly hard to check if some entity has some methods/properties in C++, so they have no other choice than to work around that limitations. it may be worthful to redesign the whole thing for D, exploiting D shiny UFCS and metaprogramming features. and, maybe, moving some things to interfaces too. I just finished reading aldanor's blog, so I know he is slightly allergic to naked functions and prefers classes ;) With Ketmar, I very much agree (predominantly as a user of HDF5 and less so as an inexperienced D programmr writing a wrapper for it). It's a pain to figure out just how to do simple things until you know the H5 library. You have to create an object for file permissions before you even get started, then similarly for the data series (datasets) within, another for the dimensions of the array, etc etc - that doesn't fit with the intrinsic nature of the domain. There is a more general question of bindings/wrappers - preserve the original structure and naming so existing code can be ported, or write a wrapper that makes it easy for the user to accomplish his objectives. It seems like for the bindings preserving the library structure is fine, but for the wrapper one might as well make things easy. Eg here https://gist.github.com/Laeeth/9637233db41a11a9d1f4 line 146. (sorry for duplication and messiness of code, which I don't claim to be perfectly written - I wanted to try something quickly and have not yet tidied up). So rather than navigate the Byzantine hierarchy, one can just do something like this (which will take a struct of PriceBar - date,open,high,low,close - and put it in your desired dataset and file, appending or overwriting as you prefer). dumpDataSpaceVector!PriceBar(file,ticker,array(priceBars[ticker]),DumpMode.truncate); which is closer to h5py in Python. (It uses reflection to figure out the contents of a non-nested struct, but won't yet cope with arrays and nested structs inside). And of course a full wrapper might be a bit more complicated, but I truly think one can do better than mapping the HDF5 hierarchy one for one. Laeeth.
Re: idiomatic D: what to use instead of pointers in constructing a tree data structure?
On Wednesday, 7 January 2015 at 14:59:58 UTC, H. S. Teoh via Digitalmars-d-learn wrote: On Wed, Jan 07, 2015 at 02:52:51PM +, Laeeth Isharc via Digitalmars-d-learn wrote: Another schoolboy question. Suppose I am constructing a tree (in this case it is an AST). In C I would have a pointer for the child to find the parent, and an array or linked list of pointers to find the children from the parent. Obviously, I could still use pointers, but that would not be idiomatic. Not true. If you're using a tree structure, you *should* use pointers. Unless you're using classes, which are by-reference, in which case you can just use the class as-is. :-) --T The GC is allowed to move structs around, as I undestand it. Under what circumstances do I get into trouble having a pointer to them?
Re: idiomatic D: what to use instead of pointers in constructing a tree data structure?
this conversation is so funny: well what's wrong with this . It's a keyword... Aa Ha ha ha ha , rol. Seriously, is it so complicated to use a D editor ? I mean with syntax color... Man afraid to ask stoopid questions stays stoopid. And compiler error message far from informative. Not everyone is very visually oriented, and sometimes vi is quicker.
Re: idiomatic D: what to use instead of pointers in constructing a tree data structure?
ref is a reserved keyword. doh! Thanks.
idiomatic D: what to use instead of pointers in constructing a tree data structure?
Another schoolboy question. Suppose I am constructing a tree (in this case it is an AST). In C I would have a pointer for the child to find the parent, and an array or linked list of pointers to find the children from the parent. Obviously, I could still use pointers, but that would not be idiomatic. I also could just use integer array index references. A slice seems overkill to refer to just one object, but is that the best way ?
tango linking problem
when trying to build dstep. any thoughts? (since I thought it may be a problem on my machine rather than something dstep specific). Fedora 21 64 bit, release dmd. Target tango 1.0.0+2.066 is up to date. Use --force to rebuild. Target mambo 0.0.3 is up to date. Use --force to rebuild. Target dstack 0.0.2 is up to date. Use --force to rebuild. Building dstep 0.1.1 configuration application, build type debug. Compiling using dmd... Linking... ../../../root/.dub/packages/dstack-0.0.2/libdstack.a(Application_1_474.o): In function `_D5tango4text7convert6Layout13__T6LayoutTaZ6Layout6sprintMFAaAxaYAa': /opt/git/dstep/../../../root/.dub/packages/tango-1.0.0_2.066/tango/text/convert/Layout.d:128: undefined reference to `_D4core4stdc6stdarg6va_endFNbPS4core4stdc6stdarg13__va_list_tagZv' ../../../root/.dub/packages/tango-1.0.0_2.066/libtango.a(Format_224_6b9.o): In function `_D5tango2io6stream6Format20__T12FormatOutputTaZ12FormatOutput6formatMFxAaYC5tango2io6stream6Format20__T12FormatOutputTaZ12FormatOutput': /opt/git/dstep/../../../root/.dub/packages/tango-1.0.0_2.066/tango/io/stream/Format.d:157: undefined reference to `_D4core4stdc6stdarg6va_endFNbPS4core4stdc6stdarg13__va_list_tagZv' ../../../root/.dub/packages/tango-1.0.0_2.066/libtango.a(Format_224_6b9.o): In function `_D5tango2io6stream6Format20__T12FormatOutputTaZ12FormatOutput8formatlnMFxAaYC5tango2io6stream6Format20__T12FormatOutputTaZ12FormatOutput': /opt/git/dstep/../../../root/.dub/packages/tango-1.0.0_2.066/tango/io/stream/Format.d:181: undefined reference to `_D4core4stdc6stdarg6va_endFNbPS4core4stdc6stdarg13__va_list_tagZv' ../../../root/.dub/packages/tango-1.0.0_2.066/libtango.a(Format_224_6b9.o): In function `_D5tango2io6stream6Format20__T12FormatOutputTaZ12FormatOutput5printMFYC5tango2io6stream6Format20__T12FormatOutputTaZ12FormatOutput': /opt/git/dstep/../../../root/.dub/packages/tango-1.0.0_2.066/tango/io/stream/Format.d:217: undefined reference to `_D4core4stdc6stdarg6va_endFNbPS4core4stdc6stdarg13__va_list_tagZv' ../../../root/.dub/packages/tango-1.0.0_2.066/libtango.a(Format_224_6b9.o): In function `_D5tango4text7convert6Layout13__T6LayoutTaZ6Layout6sprintMFAaAxaYAa': /opt/git/dstep/../../../root/.dub/packages/tango-1.0.0_2.066/tango/text/convert/Layout.d:128: undefined reference to `_D4core4stdc6stdarg6va_endFNbPS4core4stdc6stdarg13__va_list_tagZv' ../../../root/.dub/packages/tango-1.0.0_2.066/libtango.a(Format_224_6b9.o):/opt/git/dstep/../../../root/.dub/packages/tango-1.0.0_2.066/tango/text/convert/Layout.d:203: more undefined references to `_D4core4stdc6stdarg6va_endFNbPS4core4stdc6stdarg13__va_list_tagZv' follow collect2: error: ld returned 1 exit status --- errorlevel 1 FAIL .dub/build/application-debug-linux.posix-x86_64-dmd_2066-0E8A57E2F49ABF92B2E545BD241FBEE4/ dstep executable Error executing command build: dmd failed with exit code 1.
Re: Any chance of a linux dtoh?
On Tuesday, 6 January 2015 at 14:25:19 UTC, Dicebot wrote: On Tuesday, 6 January 2015 at 14:14:28 UTC, Laeeth Isharc wrote: On Tuesday, 6 January 2015 at 14:11:19 UTC, Dicebot wrote: dstep is your only realistic chance currently. htod is completely unmaintained, even on Windows. Please report any issues found with it in relevant issue tracker. I got it the wrong way around - yes, I meant htod. I have reported the issue, and author doesn't know what is wrong. You must be referring to https://github.com/jacob-carlborg/dstep/issues/33 You mention using DMD beta there. Please don't. Use last currently available public release. This specific beta is especially harmful being left-over from unfinished beta cycle. Thanks for coming back, Dicebot - I had not seen the update to the ticket. I downgraded dmd and now have a different problem with tango at link stage, which is progress of a sort. (I updated the ticket).
Re: idiomatic D: what to use instead of pointers in constructing a tree data structure?
Not true. If you're using a tree structure, you *should* use
pointers.
Unless you're using classes, which are by-reference, in which
case you
can just use the class as-is. :-)
Thanks v much.
I just came to that realization also when I stepped away.
class node
{
string name;
node ref;
}
what's wrong with the code above ? i get an error no identifier
for declarator node. (I have not used classes much, since
structs often seem to be enough for what I need to do mostly).
Re: calling C variadic arguments with no 'argc' and only variadic arguments
On Wednesday, 18 March 2015 at 05:38:40 UTC, Ali Çehreli wrote: On 03/17/2015 06:13 PM, Laeeth Isharc wrote: DMD gave me an error message for the following declarations: double mgl_rnd (...); double mgl_rnd_ (...); Are you sure those are the right signatures? I don't think those functions take any parameters at all. I would try these: double mgl_rnd (); double mgl_rnd_ (); Ali My mistake. Dstep got confused in the translation and wrote the sig as ...and I was tired and didnt check. Thanks for looking into it.
calling C variadic arguments with no 'argc' and only variadic arguments
So I ported the C API for MathGL to D, and it is up at code.dlang.org (under dmathgl). MathGL is a nice plotting library. http://mathgl.sourceforge.net/doc_en/Pictures.html#Pictures Later I will work on porting the C++ interface, but so far it at least works for the simplest sample. (Not tried anything else as they are all written in C++). DMD gave me an error message for the following declarations: double mgl_rnd (...); double mgl_rnd_ (...); It says I need at least one fixed argument first. But the C headers are as they are. I could work around this by writing a C stub double mgl_rnd_(int dummy, ...) but is there any way to call these functions without this workaround?
Re: pyd and converting arrays of strings
On Wednesday, 18 March 2015 at 20:58:56 UTC, John Colvin wrote:
On Wednesday, 18 March 2015 at 17:09:27 UTC, Laeeth Isharc
wrote:
Hi
I am using PyD with latest stable LDC on Arch Linux 64 bit.
I have various structures I am trying to wrap to pass to
python. For example:
struct PlotLines
{
KPDateTime[] start_dates;
KPDateTime[] end_dates;
double[] y;
string[] colors;
long[] weights;
string[] styles;
}
import pyd.pyd;
extern(C) void PydMain()
{
def!(bokehDownload,PyName!bokeh_download, BokehBars
function(string,bool))();
def!(bokehCountdown,PyName!bokeh_countdown,CanvasTable
function(BokehBars,bool,bool))();
module_init();
wrap_struct!(BokehBars,Member!(dates),Member!(opens),Member!(highs),Member!(lows),Member!(closes),Member!(volumes),Member!(openinterests));
wrap_struct!(KPDateTime,Member!year,Member!month,Member!day,Member!hour,Member!minute,Member!second);
wrap_struct!(CanvasTable,Member!plot_title,Member!x_label,Member!y_label,Member!annotations,Member!lines,Member!bars,Member!studies,Member!grid_sequence);
wrap_struct!(PlotLines,Member!start_dates,Member!end_dates,Member!y,Member!colors,Member!weights,Member!styles);
wrap_struct!(PlotStudy,Member!title,Member!dates,Member!lines,Member!overbought,Member!oversold);
wrap_struct!(PlotAnnotations,Member!directions,Member!dates,Member!y,Member!weights,Member!values,Member!colors,Member!styles);
}
PyD doesn't seem to want to wrap a string[] member:
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/make_object.d(773):
Error: cannot implicitly convert expression
(python_to_d(item)) of type string to dchar[]
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/make_object.d(740):
Error: function
pyd.make_object.python_iter_to_d!(string[]).python_iter_to_d
no return exp; or assert(0); at end of function
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/make_object.d(482):
Error: template instance
pyd.make_object.python_iter_to_d!(string[]) error instantiating
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/struct_wrap.d-mixin-73(73):
instantiated from here: python_to_d!(string[])
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/struct_wrap.d(105):
instantiated from here: wrapped_member!(PlotLines*,
colors, rw)
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/class_wrap.d(1503):
instantiated from here: call!(PlotLines, PlotLines*)
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/class_wrap.d(1474):
instantiated from here: _wrap_class!(PlotLines,
PlotLines, , , Member!start_dates, Member!end_dates,
Member!y, Member!colors, Member!weights, Member!styles)
/btrfs0/mirrored/kprop/core/prod/kprop/technical/demark/countdown/main13.d(72):
instantiated from here: wrap_struct!(PlotLines,
Member!start_dates, Member!end_dates, Member!y,
Member!colors, Member!weights, Member!styles)
error: command 'ldc2' failed with exit status 1
Any thoughts on whether I am doing something wrong, and if it
is a PyD problem, how to circumvent it?
Thanks.
Laeeth.
A quick glance over class_wrap.d suggests that pyd may not
handle string[] correctly. I think it has the necessary special
casing to deal with ElementType!string being dchar (not char),
but this isn't triggered for string[]. I see you've filed a
bug, hopefully someone will take a look. I might have time soon
if no one else jumps on it.
Can you use immutable(dchar)[][] instead of string[]? That
might work, but I haven't tested it.
Hi John.
dchar[] works, but I am not sure about immutable. I turned the
strings used to populate the struct into dchar by appending
.array to the end (there may be a better way).
Thanks for looking at it.
Laeeth
pyd and converting arrays of strings
Hi
I am using PyD with latest stable LDC on Arch Linux 64 bit.
I have various structures I am trying to wrap to pass to python.
For example:
struct PlotLines
{
KPDateTime[] start_dates;
KPDateTime[] end_dates;
double[] y;
string[] colors;
long[] weights;
string[] styles;
}
import pyd.pyd;
extern(C) void PydMain()
{
def!(bokehDownload,PyName!bokeh_download, BokehBars
function(string,bool))();
def!(bokehCountdown,PyName!bokeh_countdown,CanvasTable
function(BokehBars,bool,bool))();
module_init();
wrap_struct!(BokehBars,Member!(dates),Member!(opens),Member!(highs),Member!(lows),Member!(closes),Member!(volumes),Member!(openinterests));
wrap_struct!(KPDateTime,Member!year,Member!month,Member!day,Member!hour,Member!minute,Member!second);
wrap_struct!(CanvasTable,Member!plot_title,Member!x_label,Member!y_label,Member!annotations,Member!lines,Member!bars,Member!studies,Member!grid_sequence);
wrap_struct!(PlotLines,Member!start_dates,Member!end_dates,Member!y,Member!colors,Member!weights,Member!styles);
wrap_struct!(PlotStudy,Member!title,Member!dates,Member!lines,Member!overbought,Member!oversold);
wrap_struct!(PlotAnnotations,Member!directions,Member!dates,Member!y,Member!weights,Member!values,Member!colors,Member!styles);
}
PyD doesn't seem to want to wrap a string[] member:
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/make_object.d(773):
Error: cannot implicitly convert expression (python_to_d(item))
of type string to dchar[]
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/make_object.d(740):
Error: function
pyd.make_object.python_iter_to_d!(string[]).python_iter_to_d no
return exp; or assert(0); at end of function
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/make_object.d(482):
Error: template instance
pyd.make_object.python_iter_to_d!(string[]) error instantiating
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/struct_wrap.d-mixin-73(73):
instantiated from here: python_to_d!(string[])
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/struct_wrap.d(105):
instantiated from here: wrapped_member!(PlotLines*,
colors, rw)
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/class_wrap.d(1503):
instantiated from here: call!(PlotLines, PlotLines*)
/home/laeeth/anaconda/lib/python2.7/site-packages/pyd/infrastructure/pyd/class_wrap.d(1474):
instantiated from here: _wrap_class!(PlotLines,
PlotLines, , , Member!start_dates, Member!end_dates,
Member!y, Member!colors, Member!weights, Member!styles)
/btrfs0/mirrored/kprop/core/prod/kprop/technical/demark/countdown/main13.d(72):
instantiated from here: wrap_struct!(PlotLines,
Member!start_dates, Member!end_dates, Member!y,
Member!colors, Member!weights, Member!styles)
error: command 'ldc2' failed with exit status 1
Any thoughts on whether I am doing something wrong, and if it is
a PyD problem, how to circumvent it?
Thanks.
Laeeth.
Re: Windows - std.process - Setting env variables from D
On Monday, 30 March 2015 at 12:28:19 UTC, wobbles wrote:
I'm trying to set environment variables that will be visible
when my D program exits.
It is possible in a windows batch file using the set command
(like set VAR=VALUE )
However, running this in D using:
import std.process;
import std.stdio;
void main(){
auto pid1 = spawnShell(`set VAR=VALUE`);
pid1.wait();
auto pid2 = spawnShell(`set`);
pid2.wait();
}
however, upon exit, there is no VAR=VALUE in the environment.
Using std.process.environment[VAR]= VALUE; doesnt store the
variable in the parent either.
Any solutions that people know of?
Type setx /? in the command shell. (Note the x).
http://stackoverflow.com/questions/5898131/set-a-persistent-environment-variable-from-cmd-exe
Re: Windows - std.process - Setting env variables from D
On Monday, 30 March 2015 at 13:29:06 UTC, wobbles wrote: On Monday, 30 March 2015 at 12:54:28 UTC, Adam D. Ruppe wrote: On Monday, 30 March 2015 at 12:28:19 UTC, wobbles wrote: Any solutions that people know of? You can't from an exe, it is a limitation of the operating system (same on Linux btw, environment variable inheritance is always from parent to child, never from child to parent). The reason batch files can do it is that they don't run in a separate process, they just run a batch of commands inside the shell itself. https://msdn.microsoft.com/en-us/library/windows/desktop/ms682009%28v=vs.85%29.aspx Altering the environment variables of a child process during process creation is the only way one process can directly change the environment variables of another process. A process can never directly change the environment variables of another process that is not a child of that process. If you're an administrator, you could poke the system-wide variables in the registry and tell the processes to reload them: https://msdn.microsoft.com/en-us/library/windows/desktop/ms682653%28v=vs.85%29.aspx but of course, changing system-wide registry entries affects way more than just your parent shell! If you need to change a parent shell variable, the only way is to do it from a batch file. You could perhaps run a .bat which sets the variable and calls your exe to help it do some work. Thanks Adam, Yeah, I knew it was the case in Linux, I just figured as 'set' worked in a batch file that it must be possible in Windows. I think what I'm going to do is have my D program output the commands as strings that are required to set the ENV variables in the parent and then have a batch file to run the program, get its output and run the commands outputted from the D program. Can also have a bash file to do the same (using the source command). This is for setting up a build system we're using, and is normally run via Jenkins, so running it in a kind of ugly way doesnt really matter. We're currently maintaining two seperate scripts to do this work, I'm trying to consolidate them. Maintaining one large-ish D script to do this work and 2 mini scripts to call them should be easier to maintain than 2 large bash/batch scripts. Thanks! You tried setx, and it didn't work ? Or you don't want to set permanent environmental variables ?
Re: Windows - std.process - Setting env variables from D
You tried setx, and it didn't work ? Or you don't want to set permanent environmental variables Yep, correct. Don't want them to be permanent. The systems have to be clean for other tests at all times, so they need to be on a shell by shell basis sadly. Thanks - was curious to know. Laeeth.
Maxime's micro allocation benchmark much faster ?
I was curious to see if new DMD had changed speed on Maxime
Chevalier-Boisvert's allocation benchmark here:
http://pointersgonewild.com/2014/10/26/circumventing-the-d-garbage-collector/
I haven't had time to look at the Phobos test suite to know if
this was one of those that were included, but the difference
seems to be striking. I am using two machines in my office -
both of which are old x64 boxes running Arch Linux and are quite
old (8 Gb RAM only). Same manufacturer and similar models so
should be same spec CPUwise. Have not got time to install and
compare different versions of dmd on same machine, so fwiw:
1mm objects
---
dmd 2.07 release: 0.56 seconds
dmd 2.067-devel-639bcaa: 0.88 seconds
dmd 2.07 release: between 4.44 and 6.57 seconds
dmd 2.067-devel-639bcaa: 90 seconds
In case I made a typo in code:
import std.conv;
class Node
{
Node next;
size_t a,b,c,d;
}
void main(string[] args)
{
auto numNodes=to!size_t(args[1]);
Node head=null;
for(size_t i=0;inumNodes;i++)
{
auto n=new Node();
n.next=head;
head=n;
}
}
Re: Maxime's micro allocation benchmark much faster ?
oops - scratch that. may have made a mistake with versions and
be comparing 2.067 with some unstable dev version.
On Tuesday, 31 March 2015 at 11:46:41 UTC, Laeeth Isharc wrote:
I was curious to see if new DMD had changed speed on Maxime
Chevalier-Boisvert's allocation benchmark here:
http://pointersgonewild.com/2014/10/26/circumventing-the-d-garbage-collector/
I haven't had time to look at the Phobos test suite to know if
this was one of those that were included, but the difference
seems to be striking. I am using two machines in my office -
both of which are old x64 boxes running Arch Linux and are
quite old (8 Gb RAM only). Same manufacturer and similar
models so should be same spec CPUwise. Have not got time to
install and compare different versions of dmd on same machine,
so fwiw:
1mm objects
---
dmd 2.07 release: 0.56 seconds
dmd 2.067-devel-639bcaa: 0.88 seconds
dmd 2.07 release: between 4.44 and 6.57 seconds
dmd 2.067-devel-639bcaa: 90 seconds
In case I made a typo in code:
import std.conv;
class Node
{
Node next;
size_t a,b,c,d;
}
void main(string[] args)
{
auto numNodes=to!size_t(args[1]);
Node head=null;
for(size_t i=0;inumNodes;i++)
{
auto n=new Node();
n.next=head;
head=n;
}
}
Re: Maxime's micro allocation benchmark much faster ?
On Wednesday, 1 April 2015 at 10:35:05 UTC, John Colvin wrote: On Wednesday, 1 April 2015 at 10:09:12 UTC, FG wrote: On 2015-03-31 at 22:56, Laeeth Isharc wrote: 1mm allocations 2.066: 0.844s 2.067: 0.19s That is great news, thanks! OT: it's a nasty financier's habit to write 1M and 1MM instead of 1k and 1M. :P Yeah, what's with that? I've never seen it before. One cannot entirely escape déformation professionnelle ;) [People mostly write 1,000 but 1mm although 1m is pedantically correct for 1,000). Better internalize the conventions if one doesn't want to avoid expensive mistakes under pressure.
OT; Donald Knuth on beauty, efficiency, and the programmer as artist
An old essay that may yet be relevant today at a time when intellectual fashion has continued in the direction he was moved to address in his speech. there is a way to make a big improvement: it is still a pleasure to do routine jobs if we have beautiful things to work with. For example, a person will really enjoy wiping off the dining room table, day after day, if it is a beautifully designed table made from some fine quality hardwood. Language designers also have an obligation to provide languages that encourage good style, since we all know that style is strongly influenced by the language in which it is expressed. The present surge of interest in structured programming has revealed that none of our existing languages is really ideal for dealing with program and data structure http://www.paulgraham.com/knuth.html CACM, December 1974 When Communications of the ACM began publication in 1959, the members of ACM'S Editorial Board made the following remark as they described the purposes of ACM'S periodicals [2]: If computer programming is to become an important part of computer research and development, a transition of programming from an art to a disciplined science must be effected. Such a goal has been a continually recurring theme during the ensuing years; for example, we read in 1970 of the first steps toward transforming the art of programming into a science [26]. Meanwhile we have actually succeeded in making our discipline a science, and in a remarkably simple way: merely by deciding to call it computer science. Implicit in these remarks is the notion that there is something undesirable about an area of human activity that is classified as an art; it has to be a Science before it has any real stature. On the other hand, I have been working for more than 12 years on a series of books called The Art of Computer Programming. People frequently ask me why I picked such a title; and in fact some people apparently don't believe that I really did so, since I've seen at least one bibliographic reference to some books called The Act of Computer Programming. In this talk I shall try to explain why I think Art is the appropriate word. I will discuss what it means for something to be an art, in contrast to being a science; I will try to examine whether arts are good things or bad things; and I will try to show that a proper viewpoint of the subject will help us all to improve the quality of what we are now doing. ... As I was looking up these things about the meanings of art, I found that authors have been calling for a transition from art to science for at least two centuries. For example, the preface to a textbook on mineralogy, written in 1784, said the following [17]: Previous to the year 1780, mineralogy, though tolerably understood by many as an Art, could scarce be deemed a Science. According to most dictionaries science means knowledge that has been logically arranged and systematized in the form of general laws. The advantage of science is that it saves us from the need to think things through in each individual case; we can turn our thoughts to higher-level concepts. As John Ruskin wrote in 1853 [32]: The work of science is to substitute facts for appearances, and demonstrations for impressions. It seems to me that if the authors I studied were writing today, they would agree with the following characterization: Science is knowledge which we understand so well that we can teach it to a computer; and if we don't fully understand something, it is an art to deal with it. Since the notion of an algorithm or a computer program provides us with an extremely useful test for the depth of our knowledge about any given subject, the process of going from an art to a science means that we learn how to automate something. ... From this standpoint it is certainly desirable to make computer programming a science, and we have indeed come a long way in the 15 years since the publication ot the remarks I quoted at the beginning of this talk. Fifteen years ago computer programming was so badly understood that hardly anyone even thought about proving programs correct; we just fiddled with a program until we knew it worked. At that time we didn't even know how to express the concept that a program was correct, in any rigorous way. It is only in recent years that we have been learning about the processes of abstraction by which programs are written and understood; and this new knowledge about programming is currently producing great payoffs in practice, even though few programs are actually proved correct with complete rigor, since we are beginning to understand the principles of program structure. ... A scientific approach is generally characterized by the words logical, systematic, impersonal, calm, rational, while an artistic approach is characterized by the words aesthetic, creative, humanitarian, anxious, irrational. It seems to me
Re: Maxime's micro allocation benchmark much faster ?
Trying on a different beefier machine with 2.066 and 2.067 release versions installed: 1mm allocations: 2.066: 0.844s 2.067: 0.19s 10mm allocations 2.066: 1m 17.2 s 2.067: 0m 1.15s So numbers were ballpark right before, and allocation on this micro-benchmark much faster.
Re: Maxime's micro allocation benchmark much faster ?
On Tuesday, 31 March 2015 at 22:00:39 UTC, weaselcat wrote: On Tuesday, 31 March 2015 at 20:56:09 UTC, Laeeth Isharc wrote: Trying on a different beefier machine with 2.066 and 2.067 release versions installed: 1mm allocations: 2.066: 0.844s 2.067: 0.19s 10mm allocations 2.066: 1m 17.2 s 2.067: 0m 1.15s So numbers were ballpark right before, and allocation on this micro-benchmark much faster. Wow! props to the people that worked on the GC. Yes - should have said that. And I do appreciate very much all the hard work that has been done on this (and also by the GDC and LDC maintainers who have to keep up with each release). Don't trust these numbers till someone else has verified them, as I am not certain I haven't messed up transliterating the code, or doing something else stoopid. And of course it's a very specific micro benchmark, but it's one that matters beyond the direct implications given the discussion over it when her post came out. I would be really curious to see if Maxime finds the overall performance of her JIT improved.
Re: OT; Donald Knuth on beauty, efficiency, and the programmer as artist
The whole art/science vein of these Knuth quotes seems like a lot of BS, trying to situate computer programming in the long-standing and overblown science/humanities divide. I should like to see an argument rather than mere assertion. Steve Jobs is not an authority on this subject, but I found his conception of placing Apple at the juncture of science and the humanities rather intriguing. He characterizes the artistic approach as aesthetic, creative, humanitarian, anxious, irrational. He was making a statement about how it was characterized in his time, in order to set the context for his further remarks. He didn't express a view one way or another about whether this was a good characterization. But of course, there are aesthetic elements to any human endeavor: the scientific camp simply favors different aesthetics.Anything beyond the most rudimentary science requires a great deal of imagination and creativity: it just requires learning a great deal of technical concepts first that may be harder to manipulate mentally. Yes - so we agree about the most important part, because this has implications for how one thinks about language design, programmer productivity, quality, questions of measurement, and the like. Of course art is humanitarian, as it's subjective and aimed at a human audience. Hard to argue the science that brought us modern civilization isn't a hundred times more humanitarian, far beyond the superficial sheen of humanitarian art he's talking about. Anxious and irrational are human emotions often possessed by artists, not qualities applied to art, fitting given they cannot produce anything of the great value of science. Interesting perspective; I will leave it at that. He is really arguing against the likely prevailing view of the time of programming as a dry, functional process and pointing out and pushing for the aesthetic qualities in programming, which is all fine. But he then gets caught up in the false dichotomy and claptrap of the science/humanities debate, a lot of mumbo jumbo that is not worth getting caught up in. He should have stuck to calling for more aesthetically pleasing programming languages and tools- the best contribution he could have made is to more precisely define what he thinks that aesthetic should look like- instead of needlessly laying out worthless and overly simplistic platonic definitions like the mindsets of Art and Science. At least his piece gave us that premature optimization line, which is worth far more than everything around it. That's rather the point - one cannot 'precisely define' what aesthetic excellence looks like because it uses at a basic cognitive level a different kind of mental process to that deployed in Cartesian analysis. Analysis meaning breaking things down into their constituent parts, and thinking in terms of gestalts being the antithesis. (That's one of the fascinating things about programming: non-programmers perceive it as mostly about analysis, but that is not the case). http://www.ttbook.org/book/transcript/transcript-iain-mcgilchrist-uncut https://www.youtube.com/watch?v=MDO0yXgpD2w There is a great desire by programmers and many other professions to see themselves as artists. Well, there is art in everything, but they're not, and they should be grateful they aren't, or they wouldn't be paid anywhere near so well. ;) There is more art in C and D than Java and C++. I don't think that Knuth was playing a relative status game, but trying to bring peoples' attention to a facet of reality that he believed to be neglected yet important. I believe it remains neglected and that the aesthetic element matters for good design, but we blind ourselves to this if we pretend the field is drier than its intrinsic nature truly is.
Re: OT; Donald Knuth on beauty, efficiency, and the programmer as artist
On Saturday, 28 March 2015 at 09:04:51 UTC, Messenger wrote: On Saturday, 28 March 2015 at 01:09:44 UTC, Laeeth Isharc wrote: On Friday, 27 March 2015 at 11:33:39 UTC, Kagamin wrote: Hmm... science exists only as long as we don't understand something, then it disappears and only knowledge remains. Looks like he talks about engineering, but calls it science. One dismisses Knuth discussing the topic for which he is rightfully exceptionally well known at one's peril. He may be wrong, but one should agonize before being sure this is the case and read widely and carefully about the topic first. Often in our society people resolve cognitive dissonance by dismissing the awkward perspective that causes such discomfort, when one would reach a fuller perspective by putting up with the discomfort for a while before attempting to reach intellectual closure. I don't believe he is making the confusion you suggest. Without touching on what Kagamin said, one must also take care not to fall to argumentum ab auctoritate. Argument from authority was not a fallacy in the time this began to be deployed. After the European Dark Ages, there simply was not anyone alive of the ability, learning and discernment of the best minds of the past whose works had survived, and during the period of catching up (remember that as late as at the time Gibbon wrote classical achievements in urban infrastructure were way ahead of what we had in London - no sewers!) This being said, I did not make anything tending towards an argument from authority. I did suggest a man such as Knuth probably understood well the difference between engineering and science, and that in these sorts of cases where one is offended by reading something because it causes cognitive dissonance, one may benefit from suspending judgement temporarily and reading more by other authors resonant with the offending piece to try to make sense of the discomfort and either reject it on the basis of a now deeper understanding, or integrate some or part of it. Irritation triggers the growth of pearls, if one lets it. I appreciate that many of us have better things to do. But I had been thinking about why I find D appealing, and how I would get this across to future partners, and had also been thinking about various forum comments equating measurement with science, and so I found this Knuth piece highly thought-provoking. Because it goes against the grain of the prevailing tendency, I shouldn't expect many to agree. But there is nothing wrong with appealing to minority opinion, provided one does not become a crank. In a sense that is in any case part of how I make a living.
Re: OT; Donald Knuth on beauty, efficiency, and the programmer as artist
On Sunday, 29 March 2015 at 18:51:19 UTC, Joakim wrote: On Sunday, 29 March 2015 at 18:05:28 UTC, Laeeth Isharc wrote: I appreciate that many of us have better things to do. But I had been thinking about why I find D appealing, and how I would get this across to future partners, and had also been thinking about various forum comments equating measurement with science, and so I found this Knuth piece highly thought-provoking. Because it goes against the grain of the prevailing tendency, I shouldn't expect many to agree. But there is nothing wrong with appealing to minority opinion, provided one does not become a crank. In a sense that is in any case part of how I make a living. I completely agree with you about the aesthetic appeal of writing and reading D, it is one of the major draws of the language to me. I've recently been dealing with some C code and it feels like going back to punch cards by comparison. It is not a minor issue and Walter has often talked about optimizing for it. The white-space formatting requirements of Python were one of the main reasons I rejected it early on. Call it a superficial, knee-jerk reaction if you like, but I can't be bothered with a language that won't even let me insert temporary debugging code without formatting it just right. Beauty will save the world. But it's not mere indulgence because an accurate image of reality is beautiful, and programming is a human activity and emotional and psychological considerations cannot be wished away. I can put up with the white-space stuff, but the lack of compile-time checking disturbs me and it seems like one needs the scarce resource of discipline more to write good code there. (I do not claim to be an expert on Python).
using vibe.d to parse json
Hi.
struct RawGoogleResults
{
string version_;
string status;
string sig;
string[string][][string] table;
}
enum json =
{version:0.6,status:ok,sig:717451517,table:{cols:[{id:date,label:Date,type:date,pattern:},{id:query0,label:euro
crisis,type:number,pattern:}],rows:[{c:[{v:2004-01-02),f:January
2004},{v:0.0,f:0}]},{c:[{v:2004-02-02),f:February
2004},{v:0.0,f:0}]},{c:[{v:2004-03-02),f:March
2004},{v:0.0,f:0}]},{c:[{v:2004-04-02)...
auto table = deserialize!(JsonSerializer, RawGoogleResults)(json);
I cannot pass a string to deserialize (the documentation suggests
an input range should be fine):
http://vibed.org/api/vibe.data.serialization/deserialize
I have a feeling maybe deserialize doesn't do what I want, but
what should I be using instead. (I would like to parse the json
and shove the results in the struct above).
Thanks.
Laeeth.
Re: using vibe.d to parse json
On Tuesday, 24 March 2015 at 04:53:39 UTC, Laeeth Isharc wrote:
Hi.
struct RawGoogleResults
{
string version_;
string status;
string sig;
string[string][][string] table;
}
enum json =
{version:0.6,status:ok,sig:717451517,table:{cols:[{id:date,label:Date,type:date,pattern:},{id:query0,label:euro
crisis,type:number,pattern:}],rows:[{c:[{v:2004-01-02),f:January
2004},{v:0.0,f:0}]},{c:[{v:2004-02-02),f:February
2004},{v:0.0,f:0}]},{c:[{v:2004-03-02),f:March
2004},{v:0.0,f:0}]},{c:[{v:2004-04-02)...
auto table = deserialize!(JsonSerializer,
RawGoogleResults)(json);
I cannot pass a string to deserialize (the documentation
suggests an input range should be fine):
http://vibed.org/api/vibe.data.serialization/deserialize
I have a feeling maybe deserialize doesn't do what I want, but
what should I be using instead. (I would like to parse the
json and shove the results in the struct above).
Thanks.
Laeeth.
Okay - figured it out from the source code.
auto results = deserialize!(JsonStringSerializer!string,
RawGoogleResults)(json);
and easier to write:
struct RawGoogleResults
{
string version_;
string status;
string sig;
//Json!array[string][][string] table;
Json table;
}
Re: using vibe.d to parse json
Yeah, it is not very intuitive. But it works. Thanks. Next question - how can I correctly deal with inconsiderately chosen JSON field names like 'private' (which conflict in a struct declaration with D language keywords). A hack is to do a search and replace on the JSON before presenting it to vibe.d, but I wondered if there was a better way. Incidentally, vibe doesn't seem to like my replacing private with private_ - whereas privateX works. Laeeth.
