Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-03 Thread John Colvin via Digitalmars-d-announce

On Friday, 3 July 2015 at 10:52:38 UTC, John Colvin wrote:


Also, there would have to be some serious warning signs about 
translating from python to D, in that the micro-seconds will be 
truncated. Doesn't a lack of microseconds make it unusable for 
tick data?


Woops, I means milliseconds.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-03 Thread Laeeth Isharc via Digitalmars-d-announce

On Friday, 3 July 2015 at 10:52:38 UTC, John Colvin wrote:

On Friday, 3 July 2015 at 10:23:14 UTC, Laeeth Isharc wrote:

On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:

It would be v helpful to have a Datetime conversion from D.
 Looks like there is a macro for converting from ymd in 
datetime.h, so I guess one could just write some code 
against this API in C, D, or Cython and link it in with D 
so one can transfer data structures over more easily.


I know just enough about that topic to be very scared.


I should be able to call these from D without changing PyD, I 
think, for D to Python.  Not sure about Python to D, but 
maybe.


https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d


Doh!

It's already in deimos, but you need to call 
PyDateTime_IMPORT(); before calling the conversion function.


So all that is needed is the following:

module example;
import std.datetime;
import std.stdio;
import pyd.pyd;
import std.conv;
import deimos.python.datetime;


DateTime foo()
{
return DateTime(1999,1,1);
}
extern(C) void PydMain() {
ex_d_to_python((DateTime dt) = 
PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0));

def!foo();
module_init();
PyDateTime_IMPORT();
}

Ideally, shouldn't PyD do this for you ?  I think the code is 
already there in make_object.d, but it doesn't work - maybe 
because PyDateTime_IMPORT() has not been called if it needs to 
be.


Thanks.


Laeeth.


Aren't there time-zone concerns? Or is this just a mapping 
between D's std.datetime.DateTime and python's 
datetime.datetime with tzinfo==None, i.e. a naive date?


Also, there would have to be some serious warning signs about 
translating from python to D, in that the micro-seconds will be 
truncated. Doesn't a lack of microseconds make it unusable for 
tick data?


I just put 0 in the microseconds field as I was too tired to look 
up how D does that for now and it wasn't so important for my 
proof of concept.  But python has microseconds (though not 
smaller, I think).  Numpy and pandas have their own datetime64 
concept - I have not so nice memories of using these, but I am 
sure it must be possible to convert at C API level.  Worst case 
if there are no system calls to do the conversion, just hack up 
the current binary representation of the struct and deal with the 
mess when they change the version.  But I wouldn't suggest you 
put that in PyD unless we can do it via official call!  I do 
think regular datetime.datetime should be in there, though.  I 
think it would probably only need a handful of lines added to the 
file I mentioned - I thought they were already there, but that 
was my own change!


I am not sure where the init date call should go, and I am wary 
of tinkering with PyD when I don't yet know the codebase, nor the 
Python C API.  I think someone who knows it could do it in a few 
minutes.


Is it a problem not having sub micro second resolution?  It might 
be for some people, but then it's a core problem, and they 
probably have the budget to do it themselves and might have their 
own date representation anyway (which nonetheless might most 
conveniently be mapped to datetime64 given all the python 
libraries).


However, it's not my use case.  (I guess those who need are 
scientists and investors who analyze data at a high-frequency 
level, which is not me - 1 minute bars are enough for now, and my 
holding period will be much longer).


It's incrementally a big improvement to start by adding 
datetime.datetime...



Laeeth.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-03 Thread Laeeth Isharc via Digitalmars-d-announce
Aren't there time-zone concerns? Or is this just a mapping 
between D's std.datetime.DateTime and python's 
datetime.datetime with tzinfo==None, i.e. a naive date?


Also, there would have to be some serious warning signs about 
translating from python to D, in that the micro-seconds will be 
truncated. Doesn't a lack of microseconds make it unusable for 
tick data?


https://docs.python.org/3/c-api/datetime.html

The Python API has microseconds passed as an int in last 
argument.  You're right that I hadn't considered that D DateTime 
doesn't have fractions of a second (I think), so a SysTime would 
be better.


On the other hand, I think datetime.datetime and numpy datetime64 
are timezone-free.  There are libraries that implement datetimes 
with a timezone.


So sensible options are:

1. datetime.datetime-std.datetime.DateTime throwing away 
fractions of a second for python - D.
2. datetime.datetime-std.datetime.SysTime presuming python time 
is UTC and

2a) throwing away SysTime timezone and assuming it is UTC
2b) converting SysTime to a UTC time (and presuming python time 
is UTC).


I would go for 1 as default and allowing 2b as an option.  The 
problem is that if you implement the mapping in the pyd main code 
then user cannot override it in custom conversion code because it 
is only called if the inbuilt conversion fails.


So maybe 2b is the only sensible feasible option.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-03 Thread Laeeth Isharc via Digitalmars-d-announce

On Saturday, 4 July 2015 at 00:14:51 UTC, Laeeth Isharc wrote:

On Friday, 3 July 2015 at 21:46:09 UTC, John Colvin wrote:

On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote:
Aren't there time-zone concerns? Or is this just a mapping 
between D's std.datetime.DateTime and python's 
datetime.datetime with tzinfo==None, i.e. a naive date?


Also, there would have to be some serious warning signs 
about translating from python to D, in that the 
micro-seconds will be truncated. Doesn't a lack of 
microseconds make it unusable for tick data?


https://docs.python.org/3/c-api/datetime.html

The Python API has microseconds passed as an int in last 
argument.  You're right that I hadn't considered that D 
DateTime doesn't have fractions of a second (I think), so a 
SysTime would be better.


On the other hand, I think datetime.datetime and numpy 
datetime64 are timezone-free.


both datetime.datetime and numpy.datetime64 have timezone 
support.



Great - in that case 2b makes sense.


or rather, not 2b, but just mapping python + numpy to a SysTime


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-03 Thread Laeeth Isharc via Digitalmars-d-announce

On Friday, 3 July 2015 at 21:46:09 UTC, John Colvin wrote:

On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote:
Aren't there time-zone concerns? Or is this just a mapping 
between D's std.datetime.DateTime and python's 
datetime.datetime with tzinfo==None, i.e. a naive date?


Also, there would have to be some serious warning signs about 
translating from python to D, in that the micro-seconds will 
be truncated. Doesn't a lack of microseconds make it unusable 
for tick data?


https://docs.python.org/3/c-api/datetime.html

The Python API has microseconds passed as an int in last 
argument.  You're right that I hadn't considered that D 
DateTime doesn't have fractions of a second (I think), so a 
SysTime would be better.


On the other hand, I think datetime.datetime and numpy 
datetime64 are timezone-free.


both datetime.datetime and numpy.datetime64 have timezone 
support.



Great - in that case 2b makes sense.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-03 Thread John Colvin via Digitalmars-d-announce

On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote:
Aren't there time-zone concerns? Or is this just a mapping 
between D's std.datetime.DateTime and python's 
datetime.datetime with tzinfo==None, i.e. a naive date?


Also, there would have to be some serious warning signs about 
translating from python to D, in that the micro-seconds will 
be truncated. Doesn't a lack of microseconds make it unusable 
for tick data?


https://docs.python.org/3/c-api/datetime.html

The Python API has microseconds passed as an int in last 
argument.  You're right that I hadn't considered that D 
DateTime doesn't have fractions of a second (I think), so a 
SysTime would be better.


On the other hand, I think datetime.datetime and numpy 
datetime64 are timezone-free.


both datetime.datetime and numpy.datetime64 have timezone support.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-03 Thread John Colvin via Digitalmars-d-announce

On Friday, 3 July 2015 at 10:23:14 UTC, Laeeth Isharc wrote:

On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
It would be v helpful to have a Datetime conversion from D.  
Looks like there is a macro for converting from ymd in 
datetime.h, so I guess one could just write some code 
against this API in C, D, or Cython and link it in with D so 
one can transfer data structures over more easily.


I know just enough about that topic to be very scared.


I should be able to call these from D without changing PyD, I 
think, for D to Python.  Not sure about Python to D, but maybe.


https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d


Doh!

It's already in deimos, but you need to call 
PyDateTime_IMPORT(); before calling the conversion function.


So all that is needed is the following:

module example;
import std.datetime;
import std.stdio;
import pyd.pyd;
import std.conv;
import deimos.python.datetime;


DateTime foo()
{
return DateTime(1999,1,1);
}
extern(C) void PydMain() {
ex_d_to_python((DateTime dt) = 
PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0));

def!foo();
module_init();
PyDateTime_IMPORT();
}

Ideally, shouldn't PyD do this for you ?  I think the code is 
already there in make_object.d, but it doesn't work - maybe 
because PyDateTime_IMPORT() has not been called if it needs to 
be.


Thanks.


Laeeth.


Aren't there time-zone concerns? Or is this just a mapping 
between D's std.datetime.DateTime and python's datetime.datetime 
with tzinfo==None, i.e. a naive date?


Also, there would have to be some serious warning signs about 
translating from python to D, in that the micro-seconds will be 
truncated. Doesn't a lack of microseconds make it unusable for 
tick data?


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-03 Thread Laeeth Isharc via Digitalmars-d-announce

On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
It would be v helpful to have a Datetime conversion from D.  
Looks like there is a macro for converting from ymd in 
datetime.h, so I guess one could just write some code against 
this API in C, D, or Cython and link it in with D so one can 
transfer data structures over more easily.


I know just enough about that topic to be very scared.


I should be able to call these from D without changing PyD, I 
think, for D to Python.  Not sure about Python to D, but maybe.


https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d


Doh!

It's already in deimos, but you need to call PyDateTime_IMPORT(); 
before calling the conversion function.


So all that is needed is the following:

module example;
import std.datetime;
import std.stdio;
import pyd.pyd;
import std.conv;
import deimos.python.datetime;


DateTime foo()
{
return DateTime(1999,1,1);
}
extern(C) void PydMain() {
ex_d_to_python((DateTime dt) = 
PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0));

def!foo();
module_init();
PyDateTime_IMPORT();
}

Ideally, shouldn't PyD do this for you ?  I think the code is 
already there in make_object.d, but it doesn't work - maybe 
because PyDateTime_IMPORT() has not been called if it needs to be.


Thanks.


Laeeth.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-02 Thread John Colvin via Digitalmars-d-announce

On Thursday, 2 July 2015 at 19:51:19 UTC, Laeeth Isharc wrote:
What is the benefit from using distutils for working with D in 
a notebook?  There are two standards - the Python one, and the 
D one.  The advantage of using dub is that it becomes 
wonderfully easy to pull in D projects from code.dlang.org and 
to compile your own work developed under dub.  (And dub itself 
continues to improve).  Linking to a D project of decent 
complexity via distutils is not my idea of fun.


This was my train of thought too.

I do agree that for pyd itself it would be nice to retain the 
option of distutils (although I would love to see dub added 
also),


pyd does have rudimentary dub support, but it's only for 
embedding python in D, not the other way around. Extension 
support is on my TODO list, it's not complicated.


One should look at this as an alpha, extremely promising 
project.
 It is not any rough edges that are important at this stage, 
but that it has been done at all (in a form that is already 
very valuable).


The set of people that want to use computers to explore larger 
data sets than lately considered comfortable in an iterative 
manner is not small, even confining It just to finance.  It's 
of real value to be able to hook in to a proper back end that 
manages market and static data, with also any data processing 
and analytics also in D, and then to have a pretty and friendly 
front end that can just talk to this code on the back with 
minimal messing around to get there.  It's also very nice to 
have both Jupyter and an Excel spreadsheet at windows onto the 
data on your local machine or in the enterprise cloud.


The frictions to starting to play with D in a notebook are much 
lower than going via the command line, so I really am not sure 
if we should be worried about making it marketable at this 
stage rather than useful for doing real work.  It's potentially 
a nice debugging tool where you are trying to make sense of 
things that don't tidily fit in a debugging window, and where 
there is just too much stuff to do it via writefln or logging 
without putting lots of effort into the infrastructure to find 
events first.


I do like the simpler syntax (than PyD).  One question is 
whether you need to wrap every member of a struct.


@pdefRecursive!() or @pdef!(Recursive.yes) or @pdef!(Recursive) ? 
It's totally doable.



But it's a very useful start as it stands.

As an addendum: not many people prefer to do non system type 
things in C than a higher level language when time is money.  
So the mindset is you build a C extension to address the worst 
bits where Python's true colours shine through (ie when you are 
in a tight loop within Python interpreter and not spending most 
of the time in its C libraries.  Or writing C glue to connect 
Python to another library.


In my view, D is different.  I would rather write in D than 
Python, and to me it's much better Tor doing serious work.  
Even for parsing a CSV I prefer it (although one could debate 
the point).  In any case D is not especially slower to write in 
than Python (particularly when you include time spent getting 
the bugs out), and in a decent number of cases it may be more 
productive.


So why bother with Python at all ?  Better ecosystem for 
charting and exploring data, and an interpreter is better 
suited generally for certain kinds of tasks.  Also in some 
areas more libraries, which can be helpful to get a quick 
result, so that one can go back and do it properly later.


Libraries, libraries, libraries. In a perfect world we can 
reimplement everything in D and it'll be awesome, but in the 
getting shit done world that normal people inhabit they need it 
to work, today. Bridges to other languages gives us that. E.g. I 
have a 3D plotting library in D (still in stealth mode on that 
one) that uses matplotlib via pyd to generate tick labels using 
either the builtin TeX parser  renderer (OK, fast) or actually 
calling out to latex (perfect, slow). Could I have done that all 
myself in D? Sure. Would I? No, i would still have terribly 
rendered tick labels with no TeX support. I don't have time to 
write a TeX implementation, even one as rudimentary and limited 
as matplotlib's.


In short, you're right.

It would be v helpful to have a Datetime conversion from D.  
Looks like there is a macro for converting from ymd in 
datetime.h, so I guess one could just write some code against 
this API in C, D, or Cython and link it in with D so one can 
transfer data structures over more easily.


I know just enough about that topic to be very scared.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-02 Thread Laeeth Isharc via Digitalmars-d-announce
It would be v helpful to have a Datetime conversion from D.  
Looks like there is a macro for converting from ymd in 
datetime.h, so I guess one could just write some code against 
this API in C, D, or Cython and link it in with D so one can 
transfer data structures over more easily.


I know just enough about that topic to be very scared.


I should be able to call these from D without changing PyD, I 
think, for D to Python.  Not sure about Python to D, but maybe.


https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d

From D to Python datetime.datetime
===
PyObject *PyDate_FromDate()(int year, int month, int day) {
  return PyDateTimeAPI.Date_FromDate(year, month, day, 
PyDateTimeAPI.DateType);

}
PyObject *PyDateTime_FromDateAndTime()(int year, int month, int 
day, int hour, int min, int sec, int usec) {
  return PyDateTimeAPI.DateTime_FromDateAndTime(year, month, day, 
hour,

min, sec, usec, Py_None, PyDateTimeAPI.DateTimeType);
}

From Python datetime.datetime to D
===
// D translations of C macros:
int PyDateTime_GET_YEAR()(PyObject *o) {
  PyDateTime_Date *ot = cast(PyDateTime_Date *) o;
  return (ot.data[0]  8) | ot.data[1];
}
int PyDateTime_GET_MONTH()(PyObject *o) {
  PyDateTime_Date *ot = cast(PyDateTime_Date *) o;
  return ot.data[2];
}
int PyDateTime_GET_DAY()(PyObject *o) {
  PyDateTime_Date *ot = cast(PyDateTime_Date *) o;
  return ot.data[3];
}

int PyDateTime_DATE_GET_HOUR()(PyObject *o) {
  PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
  return ot.data[4];
}
int PyDateTime_DATE_GET_MINUTE()(PyObject *o) {
  PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
  return ot.data[5];
}
int PyDateTime_DATE_GET_SECOND()(PyObject *o) {
  PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
  return ot.data[6];
}
int PyDateTime_DATE_GET_MICROSECOND()(PyObject *o) {
  PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
  return (ot.data[7]  16) | (ot.data[8]  8) | ot.data[9];
}


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-02 Thread Russel Winder via Digitalmars-d-announce
On Tue, 2015-06-30 at 18:20 +, John Colvin via Digitalmars-d
-announce
 […]
 
 Ditched distutils in favour of dub. This is easier for me to 
 maintain and fits much better with the rest of the D ecosystem

I am not convinced by this, though cleary my feeling carry no weight in
decision making :-)

For building C, and C++ extension (and indeed Chapel ones) it is
assumed distutils will be used, allowing for:

python3 setup.py build

python3 setup.py install

SCons can do this but every one demands distutils. Can dub really
replace distutils for installing extensions as well as building them?
Will people installing extensions be prepared to switch to a non
-standard system? Whilst perhaps they should, they won't. I fear that
without a distutils installation, the extension will never be installed
outside the development team. It's not the D community that must be
convinced, it is the Python one.

-- 
Russel.
=
Dr Russel Winder  t: +44 20 7585 2200   voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


signature.asc
Description: This is a digitally signed message part


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-02 Thread John Colvin via Digitalmars-d-announce

On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote:

On Tue, 2015-06-30 at 18:20 +, John Colvin via Digitalmars-d
-announce
 […]


Ditched distutils in favour of dub. This is easier for me to 
maintain and fits much better with the rest of the D ecosystem


I am not convinced by this, though cleary my feeling carry no 
weight in decision making :-)


For building C, and C++ extension (and indeed Chapel ones) it 
is assumed distutils will be used, allowing for:


python3 setup.py build

python3 setup.py install

SCons can do this but every one demands distutils. Can dub 
really replace distutils for installing extensions as well as 
building them? Will people installing extensions be prepared to 
switch to a non -standard system? Whilst perhaps they should, 
they won't. I fear that without a distutils installation, the 
extension will never be installed outside the development team. 
It's not the D community that must be convinced, it is the 
Python one.


There are 2 seperate points where a build/install system is 
needed:


1) Installing the extension. It's just one python file, which 
makes it a good fit for %install_ext but of course I could wrap 
it in a trivial setup.py to make people feel more comfortable / 
make it fit in with easy_install/pip or related tools.


2) building the inline D code and sorting out linking to other 
libraries etc. The basics of this can be done with distutils 
(it's how pyd traditionally does it), but seeing as it's all D 
code from this point, dub is a better fit, otherwise people have 
to have manually install any D libraries they want to use.


I originally modelled PydMagic on how the %%cython inline magic 
works. %%cython uses distutils under the scenes with minor 
leakage for the compilation options and PydMagic uses dub, with 
similar levels of leakage.



The recently-spun-off-from-PydMagic work of making a nicer API 
for pyd ( github.com/John-Colvin/ppyd which PydMagic pulls in via 
dub) is independent of the build system, so you can use pyd's 
distutils extensions or dub. This is the project you would use if 
you want to write a proper python extension in D, as opposed to 
little snippets in a Jupyter notebook.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-02 Thread Laeeth Isharc via Digitalmars-d-announce

On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote:

On Tue, 2015-06-30 at 18:20 +, John Colvin via Digitalmars-d
-announce
 […]


Ditched distutils in favour of dub. This is easier for me to 
maintain and fits much better with the rest of the D ecosystem


I am not convinced by this, though cleary my feeling carry no 
weight in decision making :-)


For building C, and C++ extension (and indeed Chapel ones) it 
is assumed distutils will be used, allowing for:


python3 setup.py build

python3 setup.py install

SCons can do this but every one demands distutils. Can dub 
really replace distutils for installing extensions as well as 
building them? Will people installing extensions be prepared to 
switch to a non -standard system? Whilst perhaps they should, 
they won't. I fear that without a distutils installation, the 
extension will never be installed outside the development team. 
It's not the D community that must be convinced, it is the 
Python one.


What is the benefit from using distutils for working with D in a 
notebook?  There are two standards - the Python one, and the D 
one.  The advantage of using dub is that it becomes wonderfully 
easy to pull in D projects from code.dlang.org and to compile 
your own work developed under dub.  (And dub itself continues to 
improve).  Linking to a D project of decent complexity via 
distutils is not my idea of fun.


I do agree that for pyd itself it would be nice to retain the 
option of distutils (although I would love to see dub added also),


One should look at this as an alpha, extremely promising project. 
 It is not any rough edges that are important at this stage, but 
that it has been done at all (in a form that is already very 
valuable).


The set of people that want to use computers to explore larger 
data sets than lately considered comfortable in an iterative 
manner is not small, even confining It just to finance.  It's of 
real value to be able to hook in to a proper back end that 
manages market and static data, with also any data processing and 
analytics also in D, and then to have a pretty and friendly front 
end that can just talk to this code on the back with minimal 
messing around to get there.  It's also very nice to have both 
Jupyter and an Excel spreadsheet at windows onto the data on your 
local machine or in the enterprise cloud.


The frictions to starting to play with D in a notebook are much 
lower than going via the command line, so I really am not sure if 
we should be worried about making it marketable at this stage 
rather than useful for doing real work.  It's potentially a nice 
debugging tool where you are trying to make sense of things that 
don't tidily fit in a debugging window, and where there is just 
too much stuff to do it via writefln or logging without putting 
lots of effort into the infrastructure to find events first.


I do like the simpler syntax (than PyD).  One question is whether 
you need to wrap every member of a struct.


But it's a very useful start as it stands.



Re: D extensions to python, inline in an ipython/jupyter notebook

2015-07-02 Thread Laeeth Isharc via Digitalmars-d-announce

On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote:

On Tue, 2015-06-30 at 18:20 +, John Colvin via Digitalmars-d
-announce
 […]


Ditched distutils in favour of dub. This is easier for me to 
maintain and fits much better with the rest of the D ecosystem


I am not convinced by this, though cleary my feeling carry no 
weight in decision making :-)


For building C, and C++ extension (and indeed Chapel ones) it 
is assumed distutils will be used, allowing for:


python3 setup.py build

python3 setup.py install

SCons can do this but every one demands distutils. Can dub 
really replace distutils for installing extensions as well as 
building them? Will people installing extensions be prepared to 
switch to a non -standard system? Whilst perhaps they should, 
they won't. I fear that without a distutils installation, the 
extension will never be installed outside the development team. 
It's not the D community that must be convinced, it is the 
Python one.


As an addendum: not many people prefer to do non system type 
things in C than a higher level language when time is money.  So 
the mindset is you build a C extension to address the worst bits 
where Python's true colours shine through (ie when you are in a 
tight loop within Python interpreter and not spending most of the 
time in its C libraries.  Or writing C glue to connect Python to 
another library.


In my view, D is different.  I would rather write in D than 
Python, and to me it's much better Tor doing serious work.  Even 
for parsing a CSV I prefer it (although one could debate the 
point).  In any case D is not especially slower to write in than 
Python (particularly when you include time spent getting the bugs 
out), and in a decent number of cases it may be more productive.


So why bother with Python at all ?  Better ecosystem for charting 
and exploring data, and an interpreter is better suited generally 
for certain kinds of tasks.  Also in some areas more libraries, 
which can be helpful to get a quick result, so that one can go 
back and do it properly later.


It would be v helpful to have a Datetime conversion from D.  
Looks like there is a macro for converting from ymd in 
datetime.h, so I guess one could just write some code against 
this API in C, D, or Cython and link it in with D so one can 
transfer data structures over more easily.




Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-30 Thread Laeeth Isharc via Digitalmars-d-announce

On Tuesday, 30 June 2015 at 18:20:21 UTC, John Colvin wrote:

On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
Bare-bones at the moment, but hopefully of interest to some 
people


https://github.com/DlangScience/PydMagic

This allows you to write D extensions to python, as inline 
cells in an ipython/jupyter notebook.


v0.1.0 is released, with substantial improvements. This is more 
like a usable tool and less like a proof of concept now, 
although I would definitely say it is still alpha quality due 
to lack of testing.


Highlights:

Ditched distutils in favour of dub. This is easier for me to 
maintain and fits much better with the rest of the D ecosystem


As a result of the above, the %%pyd magic interface has 
changed, see the README.md or %%pyd? for info on the new system


Extended the @pdef!() UDA mechanism to work for not only 
functions, but pretty much everything else that pyd supports. 
You can now use it to automatically expose types, members, 
properties and more.


Many bugs fixed.


This is great - looking forward to playing with it.

One nice idea I have is to be able to flip between the notebook 
for charting and specifying reports and Excel for viewing and 
exploring the results.  Not just for me, but some people are much 
more comfortable with Excel.


(would use a backend server that is communicated with via nanomsg 
IPC so excel can see same data)


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-30 Thread John Colvin via Digitalmars-d-announce

On Tuesday, 30 June 2015 at 18:20:21 UTC, John Colvin wrote:

On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:

[...]


v0.1.0 is released, with substantial improvements. This is more 
like a usable tool and less like a proof of concept now, 
although I would definitely say it is still alpha quality due 
to lack of testing.


Highlights:

Ditched distutils in favour of dub. This is easier for me to 
maintain and fits much better with the rest of the D ecosystem


As a result of the above, the %%pyd magic interface has 
changed, see the README.md or %%pyd? for info on the new system


Extended the @pdef!() UDA mechanism to work for not only 
functions, but pretty much everything else that pyd supports. 
You can now use it to automatically expose types, members, 
properties and more.


Many bugs fixed.


Please report any bugs, suggestions, problems, anything. It all 
helps.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-16 Thread John Colvin via Digitalmars-d-announce

On Monday, 15 June 2015 at 19:35:16 UTC, Laeeth Isharc wrote:

On Monday, 15 June 2015 at 17:28:52 UTC, John Colvin wrote:

On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote:

On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:

[...]


Yes - I had noticed same, but don't yet have the experience 
(and don't have available time for now) to do much about it.  
Was looking at Facebook torch to see how that fitted with 
bokeh and inotebook, but glad you actually did something to 
make it happen.


I haven't looked at torch much, is it up to the hype?


To be direct: I don't know, as I haven't used it.  I was just 
interested in seeing how it fitted with ipython and bokeh in 
particular.


I liked your talk at dconf.


Thank you :)

Looks like progress is being made on planting the seeds of a 
matrix implementation and computation library on top.


Ilya Yaroshenko is doing great work here.

Having a way to use D code to explore data within an approach 
of rapid iteration will be useful too, and I think your work is 
a great start on this (seems like its value as it is is very 
much greater than you modestly suggest, and also I am amazed by 
how short the code is).


Well, as i briefly mentioned in my talk, it's really very easy to 
do, Jupyter/IPython makes it simple. Of course, behind the scenes 
Pyd is doing the hard work.


At the moment I am using a combination of the python interface 
to Bokeh and D code to do the work.  It would be great to port 
the bokeh server side stuff to D - I have made a start, but I 
haven't the space for a concerted effort.  The funny thing is 
there is nothing to the server side - all you do is have a 
shallow layer that translates the cumulative state from API 
calls to JSON; the hard work of drawing stuff is done in 
Javascript on the client.


Whether it's Bokeh or not, having some kind of visualisation 
solution that fits well with iteratively exploring data in D 
will be nice.  (What you have done means it's already enough to 
be very usable, but there may be benefits to moving the 
charting itself to D).


Visualisation is one of those things that no-one seems to have 
got right, despite the huge efforts that have gone in to it. I 
have ideas about D-based visualisation and already my own basic 
3-D plotting library for huge datasets, but for now the link to 
python is key to get quick and easy access to the latest tools 
that the scientific and finance world are creating.


Perhaps not for scientific applications, but for finance: 
reporting is also something that is an important and rising 
theme in use cases.  Not just for accounting, but for example 
to understand portfolio risk in a richer way.  You have for a 
long time been able to hook up reportlab with D code using PyD, 
but this element of rapid iteration was not so easy - small 
frictions have large consequences when you don't have a big 
team.


So in any case, I really appreciate your work.

One more thing - any thoughts on best spreadsheet like 
interface either purely on browser or within ipython to display 
results and allow user to enter parameters?  My friend, Giles 
Thomas, wrote Resolver but that's too big and aimed at 
something different, and I don't know if it is being maintained 
properly.


There is http://nbviewer.ipython.org/gist/rossant/9463955 (see 
https://www.youtube.com/watch?v=_JZFSFR6Yeo for demo) which looks 
like a good starting point, but it's very basic.


If https://github.com/quantopian/qgrid supported editing (perhaps 
it does already, but I can't see any mention of it) then that 
would probably be better.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-16 Thread Laeeth Isharc via Digitalmars-d-announce

On Tuesday, 16 June 2015 at 13:11:51 UTC, John Colvin wrote:
Looks like progress is being made on planting the seeds of a 
matrix implementation and computation library on top.


Ilya Yaroshenko is doing great work here.


Yes - Ilya's work looks great, and Vlad Levenfeld has some 
interesting ideas too (less specifically on matrices, but 
relating to them).


Well, as i briefly mentioned in my talk, it's really very easy 
to do, Jupyter/IPython makes it simple. Of course, behind the 
scenes Pyd is doing the hard work.


Yes - so often it's the last bit of integration that isn't so 
hard but removes frictions that makes it possible to realize the 
value created by all the foundational meta work.  And solving 
one's own problems is often a route to completing that 
integration.


Visualisation is one of those things that no-one seems to have 
got right, despite the huge efforts that have gone in to it. I 
have ideas about D-based visualisation and already my own basic 
3-D plotting library for huge datasets, but for now the link to 
python is key to get quick and easy access to the latest tools 
that the scientific and finance world are creating.


Would love to see it when it is ready (although I haven't used 3d 
plots much myself).  Agree about python, for better or for worse.


There is http://nbviewer.ipython.org/gist/rossant/9463955 (see 
https://www.youtube.com/watch?v=_JZFSFR6Yeo for demo) which 
looks like a good starting point, but it's very basic.


If https://github.com/quantopian/qgrid supported editing 
(perhaps it does already, but I can't see any mention of it) 
then that would probably be better.


Thank you for these - I will take a look.


Laeeth.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-15 Thread John Colvin via Digitalmars-d-announce

On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote:

On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:

[...]


Yes - I had noticed same, but don't yet have the experience 
(and don't have available time for now) to do much about it.  
Was looking at Facebook torch to see how that fitted with bokeh 
and inotebook, but glad you actually did something to make it 
happen.


I haven't looked at torch much, is it up to the hype?


[...]


Have some other constraints but will get in touch later if I 
can.


I hope this wasn't premature to submit:
http://www.reddit.com/r/programming/comments/39xmpj/run_inline_d_dlang_code_for_python_extensions_in/


Perhaps a little early but I'm sure it won't do any harm.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-15 Thread Laeeth Isharc via Digitalmars-d-announce

On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:
It's really only a beginning. Pyd's API is pretty clunky 
feeling by modern D standards, I hope to extend @pdef to 
automatically work for most language constructs, amongst other 
sugar.


Yes - I had noticed same, but don't yet have the experience (and 
don't have available time for now) to do much about it.  Was 
looking at Facebook torch to see how that fitted with bokeh and 
inotebook, but glad you actually did something to make it happen.


Any contributions are very much welcome, it's a fun thing to 
work on but I'm crazily busy at the moment. Anyone who's 
interested, get in touch :)


Have some other constraints but will get in touch later if I can.

I hope this wasn't premature to submit:
http://www.reddit.com/r/programming/comments/39xmpj/run_inline_d_dlang_code_for_python_extensions_in/


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-15 Thread Laeeth Isharc via Digitalmars-d-announce

On Monday, 15 June 2015 at 17:28:52 UTC, John Colvin wrote:

On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote:

On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:

[...]


Yes - I had noticed same, but don't yet have the experience 
(and don't have available time for now) to do much about it.  
Was looking at Facebook torch to see how that fitted with 
bokeh and inotebook, but glad you actually did something to 
make it happen.


I haven't looked at torch much, is it up to the hype?


To be direct: I don't know, as I haven't used it.  I was just 
interested in seeing how it fitted with ipython and bokeh in 
particular.


I liked your talk at dconf.  Looks like progress is being made on 
planting the seeds of a matrix implementation and computation 
library on top.  Having a way to use D code to explore data 
within an approach of rapid iteration will be useful too, and I 
think your work is a great start on this (seems like its value as 
it is is very much greater than you modestly suggest, and also I 
am amazed by how short the code is).


At the moment I am using a combination of the python interface to 
Bokeh and D code to do the work.  It would be great to port the 
bokeh server side stuff to D - I have made a start, but I haven't 
the space for a concerted effort.  The funny thing is there is 
nothing to the server side - all you do is have a shallow layer 
that translates the cumulative state from API calls to JSON; the 
hard work of drawing stuff is done in Javascript on the client.


Whether it's Bokeh or not, having some kind of visualisation 
solution that fits well with iteratively exploring data in D will 
be nice.  (What you have done means it's already enough to be 
very usable, but there may be benefits to moving the charting 
itself to D).


Perhaps not for scientific applications, but for finance: 
reporting is also something that is an important and rising theme 
in use cases.  Not just for accounting, but for example to 
understand portfolio risk in a richer way.  You have for a long 
time been able to hook up reportlab with D code using PyD, but 
this element of rapid iteration was not so easy - small frictions 
have large consequences when you don't have a big team.


So in any case, I really appreciate your work.

One more thing - any thoughts on best spreadsheet like interface 
either purely on browser or within ipython to display results and 
allow user to enter parameters?  My friend, Giles Thomas, wrote 
Resolver but that's too big and aimed at something different, and 
I don't know if it is being maintained properly.




Laeeth




Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-15 Thread Andrei Alexandrescu via Digitalmars-d-announce

On 6/15/15 10:28 AM, John Colvin wrote:

On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote:

On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote:

[...]


Yes - I had noticed same, but don't yet have the experience (and don't
have available time for now) to do much about it. Was looking at
Facebook torch to see how that fitted with bokeh and inotebook, but
glad you actually did something to make it happen.


I haven't looked at torch much, is it up to the hype?


Yann LeCun recommended it; I think that's a good endorsement. -- Andrei



Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-15 Thread John Colvin via Digitalmars-d-announce

On Monday, 15 June 2015 at 05:11:17 UTC, Laeeth Isharc wrote:

On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
Bare-bones at the moment, but hopefully of interest to some 
people


https://github.com/DlangScience/PydMagic

This allows you to write D extensions to python, as inline 
cells in an ipython/jupyter notebook.


This was just what I was looking for - great work, and I think 
this will be very useful.




Laeeth


It's really only a beginning. Pyd's API is pretty clunky feeling 
by modern D standards, I hope to extend @pdef to automatically 
work for most language constructs, amongst other sugar.


I also aim to factor it out in to a library that doesn't depend 
on ipython, so people can just get a nicer Pyd. Ultimately maybe 
it could replace the upstream API.


Any contributions are very much welcome, it's a fun thing to work 
on but I'm crazily busy at the moment. Anyone who's interested, 
get in touch :)


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-14 Thread John Colvin via Digitalmars-d-announce

On Sunday, 14 June 2015 at 16:14:58 UTC, Russel Winder wrote:
On Sun, 2015-06-14 at 15:51 +, John Colvin via 
Digitalmars-d -announce wrote:
Bare-bones at the moment, but hopefully of interest to some 
people


https://github.com/DlangScience/PydMagic

This allows you to write D extensions to python, as inline 
cells in an ipython/jupyter notebook.


I am doing a talk on using C++, D and Chapel in Python to the 
PyData London 2015 people, many I am sure will want to use 
IPython/Jupyter. If you can guide me to a short demo that has 
maximum impact for the least risk [*], I can certainly give it 
an airing.


[*] As ever with my live demos, the demo gods always want a say 
;-)


Ok, see examples/test.ipynb it's not amazing but it's a start.


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-14 Thread Russel Winder via Digitalmars-d-announce
On Sun, 2015-06-14 at 15:51 +, John Colvin via Digitalmars-d
-announce wrote:
 Bare-bones at the moment, but hopefully of interest to some people
 
 https://github.com/DlangScience/PydMagic
 
 This allows you to write D extensions to python, as inline cells 
 in an ipython/jupyter notebook.

I am doing a talk on using C++, D and Chapel in Python to the PyData
London 2015 people, many I am sure will want to use IPython/Jupyter. If
you can guide me to a short demo that has maximum impact for the least
risk [*], I can certainly give it an airing.

[*] As ever with my live demos, the demo gods always want a say ;-)

-- 
Russel.
=
Dr Russel Winder  t: +44 20 7585 2200   voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077   xmpp: rus...@winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder


signature.asc
Description: This is a digitally signed message part


Re: D extensions to python, inline in an ipython/jupyter notebook

2015-06-14 Thread Laeeth Isharc via Digitalmars-d-announce

On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote:
Bare-bones at the moment, but hopefully of interest to some 
people


https://github.com/DlangScience/PydMagic

This allows you to write D extensions to python, as inline 
cells in an ipython/jupyter notebook.


This was just what I was looking for - great work, and I think 
this will be very useful.




Laeeth