Hi Charles,
Thanks for your suggestion. I found a way to load rows on demand, so
I'll add that to our next release (i.e. 2.3.2). The thing is much more
snappier for DataFrames with more than 100,000 rows, but it's not that
good for 1,000,000 ones. I don't think we can do better though :-)
Cheers,
Carlos
El 03/11/14 a las 11:23, Charles Vellutini escribió:
Thanks, interesting.
The thing is that pandas DataFrames have become a sort of de facto
standard in manipulating data, at least in my area
(econometrics/statistics). For those who have not used them yet, DFs
are awesome in terms of preparing data. Still, I will look into what
you suggested - always good to know there are other approaches.
Le mercredi 29 octobre 2014 10:50:26 UTC+1, AndroidHiramash a écrit :
Hi,
A suggestion, even if it is not directly related : generally the
"io" module is used, with streams, as high level file objects.
There you have two solutions :
- either use paraview. I heard they had a Python API. Why not
blend Spyder and Paraview, then...
- Or use HDF5 format and Python bindings. PyTables, as told on
their site, has the reputation to be "blind fast" and efficient at
loading while having metadata looking like pandas' dataframes. But
there it is up to you at the moment to refill your data into hdf
stores, I suppose ? Personnally, the hdf5 format reminds me both
hyperspy library and TDMS Labview's technical data format...
Le 28 oct. 2014 15:20, "Charles Vellutini" <[email protected]
<javascript:>> a écrit :
Hi,
The addition of DataFrames as objects that can be viewed (and
edited) in Spyder's Variable Explorer is a fantastic
development. Viewing data is extremely important in serious
data analysis and related debugging. With this addition Spyder
approaches the convenience and workability of dedicated,
mature statistical packages such as Stata -- all with the
performance and malleability of python. In my view, a true
game changer.
Now, I have noticed that the feature does not work well (yet)
on large data sets. On my system (python 3.4, 8 Go RAM),
attempting to use the Variable Explorer with a df with more
than say 100,000 rows freezes Spyder altogether. More work is
needed is optimize viewing (load rows/columns only as they are
viewed, or a similar strategy?). Also, I would like to suggest
that viewing is much more important than editing -- in case it
helps to optimize the feature? Editing data through a browser
is not something you normally do - viewing data on the other
hand, you do all the time.
Again congratulations on this, I truly believe that this is
important for the python data analysis community.
--
You received this message because you are subscribed to the
Google Groups "spyder" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to [email protected] <javascript:>.
To post to this group, send email to [email protected]
<javascript:>.
Visit this group at http://groups.google.com/group/spyderlib
<http://groups.google.com/group/spyderlib>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "spyder" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
To post to this group, send email to [email protected]
<mailto:[email protected]>.
Visit this group at http://groups.google.com/group/spyderlib.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"spyder" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/spyderlib.
For more options, visit https://groups.google.com/d/optout.
