On Saturday, 21 November 2015 at 14:16:26 UTC, Laeeth Isharc
wrote:
Not sure it is a great idea to use a variant as the basic
option when very often you will know that every cell in a
particular column will be of the same type.
I'm reading today about an n-dim extension to pandas named xray.
Maybe should try to understand how that fits. They support io
from netCDF, and are making extensions to support blocked input
using dask, so they can process data larger than in-memory limits.
http://xray.readthedocs.org/en/stable/data-structures.html
https://www.continuum.io/content/xray-dask-out-core-labeled-arrays-python
In general, pandas and xray are supporting with the requirement
of pulling in data from storage of initially unknown column and
index names and data types. Julia throws in support of jit
compilation and specialized operations for different data types.
It seems to me that D's strength would be in a quick compile,
which would then allow you to replace the dictionary tag
implementations and variants with something that used compile
time symbol names and data types. Seems like that would provide
more efficient processing, as well as better tab completion
support when creating expressions.
