Re: D extensions to python, inline in an ipython/jupyter notebook
On Friday, 3 July 2015 at 10:52:38 UTC, John Colvin wrote: Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data? Woops, I means milliseconds.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Friday, 3 July 2015 at 10:52:38 UTC, John Colvin wrote:
On Friday, 3 July 2015 at 10:23:14 UTC, Laeeth Isharc wrote:
On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
It would be v helpful to have a Datetime conversion from D.
Looks like there is a macro for converting from ymd in
datetime.h, so I guess one could just write some code
against this API in C, D, or Cython and link it in with D
so one can transfer data structures over more easily.
I know just enough about that topic to be very scared.
I should be able to call these from D without changing PyD, I
think, for D to Python. Not sure about Python to D, but
maybe.
https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d
Doh!
It's already in deimos, but you need to call
PyDateTime_IMPORT(); before calling the conversion function.
So all that is needed is the following:
module example;
import std.datetime;
import std.stdio;
import pyd.pyd;
import std.conv;
import deimos.python.datetime;
DateTime foo()
{
return DateTime(1999,1,1);
}
extern(C) void PydMain() {
ex_d_to_python((DateTime dt) =
PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0));
def!foo();
module_init();
PyDateTime_IMPORT();
}
Ideally, shouldn't PyD do this for you ? I think the code is
already there in make_object.d, but it doesn't work - maybe
because PyDateTime_IMPORT() has not been called if it needs to
be.
Thanks.
Laeeth.
Aren't there time-zone concerns? Or is this just a mapping
between D's std.datetime.DateTime and python's
datetime.datetime with tzinfo==None, i.e. a naive date?
Also, there would have to be some serious warning signs about
translating from python to D, in that the micro-seconds will be
truncated. Doesn't a lack of microseconds make it unusable for
tick data?
I just put 0 in the microseconds field as I was too tired to look
up how D does that for now and it wasn't so important for my
proof of concept. But python has microseconds (though not
smaller, I think). Numpy and pandas have their own datetime64
concept - I have not so nice memories of using these, but I am
sure it must be possible to convert at C API level. Worst case
if there are no system calls to do the conversion, just hack up
the current binary representation of the struct and deal with the
mess when they change the version. But I wouldn't suggest you
put that in PyD unless we can do it via official call! I do
think regular datetime.datetime should be in there, though. I
think it would probably only need a handful of lines added to the
file I mentioned - I thought they were already there, but that
was my own change!
I am not sure where the init date call should go, and I am wary
of tinkering with PyD when I don't yet know the codebase, nor the
Python C API. I think someone who knows it could do it in a few
minutes.
Is it a problem not having sub micro second resolution? It might
be for some people, but then it's a core problem, and they
probably have the budget to do it themselves and might have their
own date representation anyway (which nonetheless might most
conveniently be mapped to datetime64 given all the python
libraries).
However, it's not my use case. (I guess those who need are
scientists and investors who analyze data at a high-frequency
level, which is not me - 1 minute bars are enough for now, and my
holding period will be much longer).
It's incrementally a big improvement to start by adding
datetime.datetime...
Laeeth.
Re: D extensions to python, inline in an ipython/jupyter notebook
Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date? Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data? https://docs.python.org/3/c-api/datetime.html The Python API has microseconds passed as an int in last argument. You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better. On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free. There are libraries that implement datetimes with a timezone. So sensible options are: 1. datetime.datetime-std.datetime.DateTime throwing away fractions of a second for python - D. 2. datetime.datetime-std.datetime.SysTime presuming python time is UTC and 2a) throwing away SysTime timezone and assuming it is UTC 2b) converting SysTime to a UTC time (and presuming python time is UTC). I would go for 1 as default and allowing 2b as an option. The problem is that if you implement the mapping in the pyd main code then user cannot override it in custom conversion code because it is only called if the inbuilt conversion fails. So maybe 2b is the only sensible feasible option.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Saturday, 4 July 2015 at 00:14:51 UTC, Laeeth Isharc wrote: On Friday, 3 July 2015 at 21:46:09 UTC, John Colvin wrote: On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote: Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date? Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data? https://docs.python.org/3/c-api/datetime.html The Python API has microseconds passed as an int in last argument. You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better. On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free. both datetime.datetime and numpy.datetime64 have timezone support. Great - in that case 2b makes sense. or rather, not 2b, but just mapping python + numpy to a SysTime
Re: D extensions to python, inline in an ipython/jupyter notebook
On Friday, 3 July 2015 at 21:46:09 UTC, John Colvin wrote: On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote: Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date? Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data? https://docs.python.org/3/c-api/datetime.html The Python API has microseconds passed as an int in last argument. You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better. On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free. both datetime.datetime and numpy.datetime64 have timezone support. Great - in that case 2b makes sense.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Friday, 3 July 2015 at 15:56:24 UTC, Laeeth Isharc wrote: Aren't there time-zone concerns? Or is this just a mapping between D's std.datetime.DateTime and python's datetime.datetime with tzinfo==None, i.e. a naive date? Also, there would have to be some serious warning signs about translating from python to D, in that the micro-seconds will be truncated. Doesn't a lack of microseconds make it unusable for tick data? https://docs.python.org/3/c-api/datetime.html The Python API has microseconds passed as an int in last argument. You're right that I hadn't considered that D DateTime doesn't have fractions of a second (I think), so a SysTime would be better. On the other hand, I think datetime.datetime and numpy datetime64 are timezone-free. both datetime.datetime and numpy.datetime64 have timezone support.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Friday, 3 July 2015 at 10:23:14 UTC, Laeeth Isharc wrote:
On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
It would be v helpful to have a Datetime conversion from D.
Looks like there is a macro for converting from ymd in
datetime.h, so I guess one could just write some code
against this API in C, D, or Cython and link it in with D so
one can transfer data structures over more easily.
I know just enough about that topic to be very scared.
I should be able to call these from D without changing PyD, I
think, for D to Python. Not sure about Python to D, but maybe.
https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d
Doh!
It's already in deimos, but you need to call
PyDateTime_IMPORT(); before calling the conversion function.
So all that is needed is the following:
module example;
import std.datetime;
import std.stdio;
import pyd.pyd;
import std.conv;
import deimos.python.datetime;
DateTime foo()
{
return DateTime(1999,1,1);
}
extern(C) void PydMain() {
ex_d_to_python((DateTime dt) =
PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0));
def!foo();
module_init();
PyDateTime_IMPORT();
}
Ideally, shouldn't PyD do this for you ? I think the code is
already there in make_object.d, but it doesn't work - maybe
because PyDateTime_IMPORT() has not been called if it needs to
be.
Thanks.
Laeeth.
Aren't there time-zone concerns? Or is this just a mapping
between D's std.datetime.DateTime and python's datetime.datetime
with tzinfo==None, i.e. a naive date?
Also, there would have to be some serious warning signs about
translating from python to D, in that the micro-seconds will be
truncated. Doesn't a lack of microseconds make it unusable for
tick data?
Re: D extensions to python, inline in an ipython/jupyter notebook
On Thursday, 2 July 2015 at 22:55:51 UTC, Laeeth Isharc wrote:
It would be v helpful to have a Datetime conversion from D.
Looks like there is a macro for converting from ymd in
datetime.h, so I guess one could just write some code against
this API in C, D, or Cython and link it in with D so one can
transfer data structures over more easily.
I know just enough about that topic to be very scared.
I should be able to call these from D without changing PyD, I
think, for D to Python. Not sure about Python to D, but maybe.
https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d
Doh!
It's already in deimos, but you need to call PyDateTime_IMPORT();
before calling the conversion function.
So all that is needed is the following:
module example;
import std.datetime;
import std.stdio;
import pyd.pyd;
import std.conv;
import deimos.python.datetime;
DateTime foo()
{
return DateTime(1999,1,1);
}
extern(C) void PydMain() {
ex_d_to_python((DateTime dt) =
PyDateTime_FromDateAndTime(dt.year,dt.month,dt.day,dt.hour,dt.minute,cast(int)dt.second,0));
def!foo();
module_init();
PyDateTime_IMPORT();
}
Ideally, shouldn't PyD do this for you ? I think the code is
already there in make_object.d, but it doesn't work - maybe
because PyDateTime_IMPORT() has not been called if it needs to be.
Thanks.
Laeeth.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Thursday, 2 July 2015 at 19:51:19 UTC, Laeeth Isharc wrote: What is the benefit from using distutils for working with D in a notebook? There are two standards - the Python one, and the D one. The advantage of using dub is that it becomes wonderfully easy to pull in D projects from code.dlang.org and to compile your own work developed under dub. (And dub itself continues to improve). Linking to a D project of decent complexity via distutils is not my idea of fun. This was my train of thought too. I do agree that for pyd itself it would be nice to retain the option of distutils (although I would love to see dub added also), pyd does have rudimentary dub support, but it's only for embedding python in D, not the other way around. Extension support is on my TODO list, it's not complicated. One should look at this as an alpha, extremely promising project. It is not any rough edges that are important at this stage, but that it has been done at all (in a form that is already very valuable). The set of people that want to use computers to explore larger data sets than lately considered comfortable in an iterative manner is not small, even confining It just to finance. It's of real value to be able to hook in to a proper back end that manages market and static data, with also any data processing and analytics also in D, and then to have a pretty and friendly front end that can just talk to this code on the back with minimal messing around to get there. It's also very nice to have both Jupyter and an Excel spreadsheet at windows onto the data on your local machine or in the enterprise cloud. The frictions to starting to play with D in a notebook are much lower than going via the command line, so I really am not sure if we should be worried about making it marketable at this stage rather than useful for doing real work. It's potentially a nice debugging tool where you are trying to make sense of things that don't tidily fit in a debugging window, and where there is just too much stuff to do it via writefln or logging without putting lots of effort into the infrastructure to find events first. I do like the simpler syntax (than PyD). One question is whether you need to wrap every member of a struct. @pdefRecursive!() or @pdef!(Recursive.yes) or @pdef!(Recursive) ? It's totally doable. But it's a very useful start as it stands. As an addendum: not many people prefer to do non system type things in C than a higher level language when time is money. So the mindset is you build a C extension to address the worst bits where Python's true colours shine through (ie when you are in a tight loop within Python interpreter and not spending most of the time in its C libraries. Or writing C glue to connect Python to another library. In my view, D is different. I would rather write in D than Python, and to me it's much better Tor doing serious work. Even for parsing a CSV I prefer it (although one could debate the point). In any case D is not especially slower to write in than Python (particularly when you include time spent getting the bugs out), and in a decent number of cases it may be more productive. So why bother with Python at all ? Better ecosystem for charting and exploring data, and an interpreter is better suited generally for certain kinds of tasks. Also in some areas more libraries, which can be helpful to get a quick result, so that one can go back and do it properly later. Libraries, libraries, libraries. In a perfect world we can reimplement everything in D and it'll be awesome, but in the getting shit done world that normal people inhabit they need it to work, today. Bridges to other languages gives us that. E.g. I have a 3D plotting library in D (still in stealth mode on that one) that uses matplotlib via pyd to generate tick labels using either the builtin TeX parser renderer (OK, fast) or actually calling out to latex (perfect, slow). Could I have done that all myself in D? Sure. Would I? No, i would still have terribly rendered tick labels with no TeX support. I don't have time to write a TeX implementation, even one as rudimentary and limited as matplotlib's. In short, you're right. It would be v helpful to have a Datetime conversion from D. Looks like there is a macro for converting from ymd in datetime.h, so I guess one could just write some code against this API in C, D, or Cython and link it in with D so one can transfer data structures over more easily. I know just enough about that topic to be very scared.
Re: D extensions to python, inline in an ipython/jupyter notebook
It would be v helpful to have a Datetime conversion from D.
Looks like there is a macro for converting from ymd in
datetime.h, so I guess one could just write some code against
this API in C, D, or Cython and link it in with D so one can
transfer data structures over more easily.
I know just enough about that topic to be very scared.
I should be able to call these from D without changing PyD, I
think, for D to Python. Not sure about Python to D, but maybe.
https://github.com/ariovistus/pyd/blob/master/infrastructure/python/python.d
From D to Python datetime.datetime
===
PyObject *PyDate_FromDate()(int year, int month, int day) {
return PyDateTimeAPI.Date_FromDate(year, month, day,
PyDateTimeAPI.DateType);
}
PyObject *PyDateTime_FromDateAndTime()(int year, int month, int
day, int hour, int min, int sec, int usec) {
return PyDateTimeAPI.DateTime_FromDateAndTime(year, month, day,
hour,
min, sec, usec, Py_None, PyDateTimeAPI.DateTimeType);
}
From Python datetime.datetime to D
===
// D translations of C macros:
int PyDateTime_GET_YEAR()(PyObject *o) {
PyDateTime_Date *ot = cast(PyDateTime_Date *) o;
return (ot.data[0] 8) | ot.data[1];
}
int PyDateTime_GET_MONTH()(PyObject *o) {
PyDateTime_Date *ot = cast(PyDateTime_Date *) o;
return ot.data[2];
}
int PyDateTime_GET_DAY()(PyObject *o) {
PyDateTime_Date *ot = cast(PyDateTime_Date *) o;
return ot.data[3];
}
int PyDateTime_DATE_GET_HOUR()(PyObject *o) {
PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
return ot.data[4];
}
int PyDateTime_DATE_GET_MINUTE()(PyObject *o) {
PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
return ot.data[5];
}
int PyDateTime_DATE_GET_SECOND()(PyObject *o) {
PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
return ot.data[6];
}
int PyDateTime_DATE_GET_MICROSECOND()(PyObject *o) {
PyDateTime_DateTime *ot = cast(PyDateTime_DateTime *) o;
return (ot.data[7] 16) | (ot.data[8] 8) | ot.data[9];
}
Re: D extensions to python, inline in an ipython/jupyter notebook
On Tue, 2015-06-30 at 18:20 +, John Colvin via Digitalmars-d -announce […] Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem I am not convinced by this, though cleary my feeling carry no weight in decision making :-) For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for: python3 setup.py build python3 setup.py install SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: D extensions to python, inline in an ipython/jupyter notebook
On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote: On Tue, 2015-06-30 at 18:20 +, John Colvin via Digitalmars-d -announce […] Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem I am not convinced by this, though cleary my feeling carry no weight in decision making :-) For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for: python3 setup.py build python3 setup.py install SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one. There are 2 seperate points where a build/install system is needed: 1) Installing the extension. It's just one python file, which makes it a good fit for %install_ext but of course I could wrap it in a trivial setup.py to make people feel more comfortable / make it fit in with easy_install/pip or related tools. 2) building the inline D code and sorting out linking to other libraries etc. The basics of this can be done with distutils (it's how pyd traditionally does it), but seeing as it's all D code from this point, dub is a better fit, otherwise people have to have manually install any D libraries they want to use. I originally modelled PydMagic on how the %%cython inline magic works. %%cython uses distutils under the scenes with minor leakage for the compilation options and PydMagic uses dub, with similar levels of leakage. The recently-spun-off-from-PydMagic work of making a nicer API for pyd ( github.com/John-Colvin/ppyd which PydMagic pulls in via dub) is independent of the build system, so you can use pyd's distutils extensions or dub. This is the project you would use if you want to write a proper python extension in D, as opposed to little snippets in a Jupyter notebook.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote: On Tue, 2015-06-30 at 18:20 +, John Colvin via Digitalmars-d -announce […] Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem I am not convinced by this, though cleary my feeling carry no weight in decision making :-) For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for: python3 setup.py build python3 setup.py install SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one. What is the benefit from using distutils for working with D in a notebook? There are two standards - the Python one, and the D one. The advantage of using dub is that it becomes wonderfully easy to pull in D projects from code.dlang.org and to compile your own work developed under dub. (And dub itself continues to improve). Linking to a D project of decent complexity via distutils is not my idea of fun. I do agree that for pyd itself it would be nice to retain the option of distutils (although I would love to see dub added also), One should look at this as an alpha, extremely promising project. It is not any rough edges that are important at this stage, but that it has been done at all (in a form that is already very valuable). The set of people that want to use computers to explore larger data sets than lately considered comfortable in an iterative manner is not small, even confining It just to finance. It's of real value to be able to hook in to a proper back end that manages market and static data, with also any data processing and analytics also in D, and then to have a pretty and friendly front end that can just talk to this code on the back with minimal messing around to get there. It's also very nice to have both Jupyter and an Excel spreadsheet at windows onto the data on your local machine or in the enterprise cloud. The frictions to starting to play with D in a notebook are much lower than going via the command line, so I really am not sure if we should be worried about making it marketable at this stage rather than useful for doing real work. It's potentially a nice debugging tool where you are trying to make sense of things that don't tidily fit in a debugging window, and where there is just too much stuff to do it via writefln or logging without putting lots of effort into the infrastructure to find events first. I do like the simpler syntax (than PyD). One question is whether you need to wrap every member of a struct. But it's a very useful start as it stands.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Thursday, 2 July 2015 at 18:28:50 UTC, Russel Winder wrote: On Tue, 2015-06-30 at 18:20 +, John Colvin via Digitalmars-d -announce […] Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem I am not convinced by this, though cleary my feeling carry no weight in decision making :-) For building C, and C++ extension (and indeed Chapel ones) it is assumed distutils will be used, allowing for: python3 setup.py build python3 setup.py install SCons can do this but every one demands distutils. Can dub really replace distutils for installing extensions as well as building them? Will people installing extensions be prepared to switch to a non -standard system? Whilst perhaps they should, they won't. I fear that without a distutils installation, the extension will never be installed outside the development team. It's not the D community that must be convinced, it is the Python one. As an addendum: not many people prefer to do non system type things in C than a higher level language when time is money. So the mindset is you build a C extension to address the worst bits where Python's true colours shine through (ie when you are in a tight loop within Python interpreter and not spending most of the time in its C libraries. Or writing C glue to connect Python to another library. In my view, D is different. I would rather write in D than Python, and to me it's much better Tor doing serious work. Even for parsing a CSV I prefer it (although one could debate the point). In any case D is not especially slower to write in than Python (particularly when you include time spent getting the bugs out), and in a decent number of cases it may be more productive. So why bother with Python at all ? Better ecosystem for charting and exploring data, and an interpreter is better suited generally for certain kinds of tasks. Also in some areas more libraries, which can be helpful to get a quick result, so that one can go back and do it properly later. It would be v helpful to have a Datetime conversion from D. Looks like there is a macro for converting from ymd in datetime.h, so I guess one could just write some code against this API in C, D, or Cython and link it in with D so one can transfer data structures over more easily.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Tuesday, 30 June 2015 at 18:20:21 UTC, John Colvin wrote: On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote: [...] v0.1.0 is released, with substantial improvements. This is more like a usable tool and less like a proof of concept now, although I would definitely say it is still alpha quality due to lack of testing. Highlights: Ditched distutils in favour of dub. This is easier for me to maintain and fits much better with the rest of the D ecosystem As a result of the above, the %%pyd magic interface has changed, see the README.md or %%pyd? for info on the new system Extended the @pdef!() UDA mechanism to work for not only functions, but pretty much everything else that pyd supports. You can now use it to automatically expose types, members, properties and more. Many bugs fixed. Please report any bugs, suggestions, problems, anything. It all helps.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Monday, 15 June 2015 at 19:35:16 UTC, Laeeth Isharc wrote: On Monday, 15 June 2015 at 17:28:52 UTC, John Colvin wrote: On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote: On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote: [...] Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen. I haven't looked at torch much, is it up to the hype? To be direct: I don't know, as I haven't used it. I was just interested in seeing how it fitted with ipython and bokeh in particular. I liked your talk at dconf. Thank you :) Looks like progress is being made on planting the seeds of a matrix implementation and computation library on top. Ilya Yaroshenko is doing great work here. Having a way to use D code to explore data within an approach of rapid iteration will be useful too, and I think your work is a great start on this (seems like its value as it is is very much greater than you modestly suggest, and also I am amazed by how short the code is). Well, as i briefly mentioned in my talk, it's really very easy to do, Jupyter/IPython makes it simple. Of course, behind the scenes Pyd is doing the hard work. At the moment I am using a combination of the python interface to Bokeh and D code to do the work. It would be great to port the bokeh server side stuff to D - I have made a start, but I haven't the space for a concerted effort. The funny thing is there is nothing to the server side - all you do is have a shallow layer that translates the cumulative state from API calls to JSON; the hard work of drawing stuff is done in Javascript on the client. Whether it's Bokeh or not, having some kind of visualisation solution that fits well with iteratively exploring data in D will be nice. (What you have done means it's already enough to be very usable, but there may be benefits to moving the charting itself to D). Visualisation is one of those things that no-one seems to have got right, despite the huge efforts that have gone in to it. I have ideas about D-based visualisation and already my own basic 3-D plotting library for huge datasets, but for now the link to python is key to get quick and easy access to the latest tools that the scientific and finance world are creating. Perhaps not for scientific applications, but for finance: reporting is also something that is an important and rising theme in use cases. Not just for accounting, but for example to understand portfolio risk in a richer way. You have for a long time been able to hook up reportlab with D code using PyD, but this element of rapid iteration was not so easy - small frictions have large consequences when you don't have a big team. So in any case, I really appreciate your work. One more thing - any thoughts on best spreadsheet like interface either purely on browser or within ipython to display results and allow user to enter parameters? My friend, Giles Thomas, wrote Resolver but that's too big and aimed at something different, and I don't know if it is being maintained properly. There is http://nbviewer.ipython.org/gist/rossant/9463955 (see https://www.youtube.com/watch?v=_JZFSFR6Yeo for demo) which looks like a good starting point, but it's very basic. If https://github.com/quantopian/qgrid supported editing (perhaps it does already, but I can't see any mention of it) then that would probably be better.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Tuesday, 16 June 2015 at 13:11:51 UTC, John Colvin wrote: Looks like progress is being made on planting the seeds of a matrix implementation and computation library on top. Ilya Yaroshenko is doing great work here. Yes - Ilya's work looks great, and Vlad Levenfeld has some interesting ideas too (less specifically on matrices, but relating to them). Well, as i briefly mentioned in my talk, it's really very easy to do, Jupyter/IPython makes it simple. Of course, behind the scenes Pyd is doing the hard work. Yes - so often it's the last bit of integration that isn't so hard but removes frictions that makes it possible to realize the value created by all the foundational meta work. And solving one's own problems is often a route to completing that integration. Visualisation is one of those things that no-one seems to have got right, despite the huge efforts that have gone in to it. I have ideas about D-based visualisation and already my own basic 3-D plotting library for huge datasets, but for now the link to python is key to get quick and easy access to the latest tools that the scientific and finance world are creating. Would love to see it when it is ready (although I haven't used 3d plots much myself). Agree about python, for better or for worse. There is http://nbviewer.ipython.org/gist/rossant/9463955 (see https://www.youtube.com/watch?v=_JZFSFR6Yeo for demo) which looks like a good starting point, but it's very basic. If https://github.com/quantopian/qgrid supported editing (perhaps it does already, but I can't see any mention of it) then that would probably be better. Thank you for these - I will take a look. Laeeth.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote: On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote: [...] Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen. I haven't looked at torch much, is it up to the hype? [...] Have some other constraints but will get in touch later if I can. I hope this wasn't premature to submit: http://www.reddit.com/r/programming/comments/39xmpj/run_inline_d_dlang_code_for_python_extensions_in/ Perhaps a little early but I'm sure it won't do any harm.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote: It's really only a beginning. Pyd's API is pretty clunky feeling by modern D standards, I hope to extend @pdef to automatically work for most language constructs, amongst other sugar. Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen. Any contributions are very much welcome, it's a fun thing to work on but I'm crazily busy at the moment. Anyone who's interested, get in touch :) Have some other constraints but will get in touch later if I can. I hope this wasn't premature to submit: http://www.reddit.com/r/programming/comments/39xmpj/run_inline_d_dlang_code_for_python_extensions_in/
Re: D extensions to python, inline in an ipython/jupyter notebook
On Monday, 15 June 2015 at 17:28:52 UTC, John Colvin wrote: On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote: On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote: [...] Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen. I haven't looked at torch much, is it up to the hype? To be direct: I don't know, as I haven't used it. I was just interested in seeing how it fitted with ipython and bokeh in particular. I liked your talk at dconf. Looks like progress is being made on planting the seeds of a matrix implementation and computation library on top. Having a way to use D code to explore data within an approach of rapid iteration will be useful too, and I think your work is a great start on this (seems like its value as it is is very much greater than you modestly suggest, and also I am amazed by how short the code is). At the moment I am using a combination of the python interface to Bokeh and D code to do the work. It would be great to port the bokeh server side stuff to D - I have made a start, but I haven't the space for a concerted effort. The funny thing is there is nothing to the server side - all you do is have a shallow layer that translates the cumulative state from API calls to JSON; the hard work of drawing stuff is done in Javascript on the client. Whether it's Bokeh or not, having some kind of visualisation solution that fits well with iteratively exploring data in D will be nice. (What you have done means it's already enough to be very usable, but there may be benefits to moving the charting itself to D). Perhaps not for scientific applications, but for finance: reporting is also something that is an important and rising theme in use cases. Not just for accounting, but for example to understand portfolio risk in a richer way. You have for a long time been able to hook up reportlab with D code using PyD, but this element of rapid iteration was not so easy - small frictions have large consequences when you don't have a big team. So in any case, I really appreciate your work. One more thing - any thoughts on best spreadsheet like interface either purely on browser or within ipython to display results and allow user to enter parameters? My friend, Giles Thomas, wrote Resolver but that's too big and aimed at something different, and I don't know if it is being maintained properly. Laeeth
Re: D extensions to python, inline in an ipython/jupyter notebook
On 6/15/15 10:28 AM, John Colvin wrote: On Monday, 15 June 2015 at 17:11:26 UTC, Laeeth Isharc wrote: On Monday, 15 June 2015 at 06:51:44 UTC, John Colvin wrote: [...] Yes - I had noticed same, but don't yet have the experience (and don't have available time for now) to do much about it. Was looking at Facebook torch to see how that fitted with bokeh and inotebook, but glad you actually did something to make it happen. I haven't looked at torch much, is it up to the hype? Yann LeCun recommended it; I think that's a good endorsement. -- Andrei
Re: D extensions to python, inline in an ipython/jupyter notebook
On Monday, 15 June 2015 at 05:11:17 UTC, Laeeth Isharc wrote: On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote: Bare-bones at the moment, but hopefully of interest to some people https://github.com/DlangScience/PydMagic This allows you to write D extensions to python, as inline cells in an ipython/jupyter notebook. This was just what I was looking for - great work, and I think this will be very useful. Laeeth It's really only a beginning. Pyd's API is pretty clunky feeling by modern D standards, I hope to extend @pdef to automatically work for most language constructs, amongst other sugar. I also aim to factor it out in to a library that doesn't depend on ipython, so people can just get a nicer Pyd. Ultimately maybe it could replace the upstream API. Any contributions are very much welcome, it's a fun thing to work on but I'm crazily busy at the moment. Anyone who's interested, get in touch :)
D extensions to python, inline in an ipython/jupyter notebook
Bare-bones at the moment, but hopefully of interest to some people https://github.com/DlangScience/PydMagic This allows you to write D extensions to python, as inline cells in an ipython/jupyter notebook.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Sunday, 14 June 2015 at 16:14:58 UTC, Russel Winder wrote: On Sun, 2015-06-14 at 15:51 +, John Colvin via Digitalmars-d -announce wrote: Bare-bones at the moment, but hopefully of interest to some people https://github.com/DlangScience/PydMagic This allows you to write D extensions to python, as inline cells in an ipython/jupyter notebook. I am doing a talk on using C++, D and Chapel in Python to the PyData London 2015 people, many I am sure will want to use IPython/Jupyter. If you can guide me to a short demo that has maximum impact for the least risk [*], I can certainly give it an airing. [*] As ever with my live demos, the demo gods always want a say ;-) Ok, see examples/test.ipynb it's not amazing but it's a start.
Re: D extensions to python, inline in an ipython/jupyter notebook
On Sun, 2015-06-14 at 15:51 +, John Colvin via Digitalmars-d -announce wrote: Bare-bones at the moment, but hopefully of interest to some people https://github.com/DlangScience/PydMagic This allows you to write D extensions to python, as inline cells in an ipython/jupyter notebook. I am doing a talk on using C++, D and Chapel in Python to the PyData London 2015 people, many I am sure will want to use IPython/Jupyter. If you can guide me to a short demo that has maximum impact for the least risk [*], I can certainly give it an airing. [*] As ever with my live demos, the demo gods always want a say ;-) -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: D extensions to python, inline in an ipython/jupyter notebook
On Sunday, 14 June 2015 at 15:51:37 UTC, John Colvin wrote: Bare-bones at the moment, but hopefully of interest to some people https://github.com/DlangScience/PydMagic This allows you to write D extensions to python, as inline cells in an ipython/jupyter notebook. This was just what I was looking for - great work, and I think this will be very useful. Laeeth
