On Monday, 22 December 2014 at 19:25:51 UTC, aldanor wrote:
On Monday, 22 December 2014 at 17:28:39 UTC, Daniel Davidson
wrote:
I don't see D attempting to tackle that at this point.
If the bulk of the work for the "data sciences" piece is the
maths, which I believe it is, then the attraction of D as a
"data sciences" platform is muted. If the bulk of the work is
preprocessing data to get to an all numbers world, then in
that space D might shine.
That is one of my points exactly -- the "bulk of the work", as
you put it, is quite often the data processing/preprocessing
pipeline (all the way from raw data parsing, aggregation,
validation and storage to data retrieval, feature extraction,
and then serialization, various persistency models, etc). One
thing is fitting some model on a pandas dataframe on your lap
in an ipython notebook, another thing is running the whole
pipeline on massive datasets in production on a daily basis,
which often involves very low-level technical stuff, whether
you like it or not. Coming up with cool algorithms and doing
fancy maths is fun and all, but it doesn't take nearly as much
effort as integrating that same thing into an existing
production system (or developing one from scratch). (and again,
production != execution in this context)
On Monday, 22 December 2014 at 17:28:39 UTC, Daniel Davidson
wrote:
What is a backtesting system in the context of Winton Capital?
Is it primarily a mathematical backtesting system? If so it
still may be better suited to platforms focusing on maths.
Disclaimer: I don't work for Winton :) Backtesting in trading
is usually a very CPU-intensive (and sometimes RAM-intensive)
task that can be potentially re-run millions of times to
fine-tune some parameters or explore some sensitivities.
Another common task is reconciling with how the actual trading
system works which is a very low-level task as well.
From what I have learned in Skills Matter presentations, for that
type of use cases, D has to fight against Scala/F# code running
in Hadoop/Spark/Azure clusters, backed up by big data databases.
--
Paulo
