Yes, I use HDF5 for more than a year now.
It's great. The only drawback is the lack of elegant Java tools.
I upload those files and store their obfuscated names in a database, but
the actual file in uploads.
Having said that it's not very elegant to construct the URL to locate the
file:
def show():
csv = db.csv(request.args(0, cast=int))
import pandas
import os
fff = os.path.join("applications", "cda", "uploads", csv.csvfile)
session.dataframe = pandas.read_csv(fff, parse_dates=True, index_col=0)
return dict(title=csv.title, body=csv.body, author=csv.author,
keys=session.dataframe.keys())
Note that sesssion,dataframe is now a global variable. And note the ugly
os.path.join with the name of the application hardcoded.... UGLY!
Here's a link:
https://tschmelz.**pythonanywhere.com/cda<https://tschmelz.pythonanywhere.com/cda>
and
https://github.com/tschm/cda
Thomas
On Tuesday, 22 October 2013 15:16:25 UTC+2, Richard wrote:
>
> I heard a lot of good about HDF5 file format to hande important volume of
> data hierachical (mean you can query what ever data you need without load
> the full data set into a json for instance) :
> http://en.wikipedia.org/wiki/Hierarchical_Data_Format
>
> It very much faster then postgres (sure postgres is not the faster backend
> but it scale gracefully)...
>
> The intend of this file format is to be used in conjonction with a DB.
>
> If I remember Pandas can write HDF5, not sure which lib it uses, there is
> two major lib in python which have different set of feature, one is more
> fancy but not support all the HDF5 feature and the other is supporting
> "all" the feature but is less sexy...
>
> Richard
>
>
> On Tue, Oct 22, 2013 at 6:54 AM, Cliff Kachinske
> <[email protected]<javascript:>
> > wrote:
>
>> use the rows field in auth_permission as described here.
>>
>> http://web2py.com/books/default/chapter/29/09/access-control#Authorization
>>
>>
>> On Tuesday, October 22, 2013 5:40:49 AM UTC-4, Thomas S wrote:
>>>
>>> Ok, I am making progress. I guess all those JavaScript tools are not
>>> great when it comes to plotting millions of points but I am happy to
>>> downsample on the server side and send less points
>>>
>>> - I am using flot instead of highcharts
>>>
>>> - Currently, the user is uploading a csv file. I don't do any parsing at
>>> this stage. However, I rather keep the file (under uploads) and parse it on
>>> request. Having said that this will become a lot more slick soon. This is
>>> my first application.
>>>
>>> One thing that puzzles me for now...
>>>
>>> A user has to login to upload a file (that's good), but he can then also
>>> modify or delete entries in the SQL database created by others. How can I
>>> make sure he/she only deletes rows he/she has created in the first place.
>>> All users should be able to see all files though.
>>>
>>> Here's a link:
>>> https://tschmelz.**pythonanywhere.com/csv<https://tschmelz.pythonanywhere.com/csv>
>>>
>>> I will soon post it to my Github (username tschm)
>>> thomas
>>>
>>>
>>> On Sunday, 20 October 2013 15:38:07 UTC+2, Niphlod wrote:
>>>>
>>>> first things first: are you sure that highcharts can handle 10*100k
>>>> points to draw a graph ?
>>>> As for the storage, you can do anything you like: if the data doesn't
>>>> change that much, storing into the database will be a long process only on
>>>> the first time.
>>>> On the other end, if you need to fetch 100k records and transform them
>>>> to json, it's going to take some time.
>>>> Not sure on how much you'll gain from parsing i.e. a csv file instead
>>>> of a querying a db....
>>>> if the returning json object is , let's say, 10 mb, it's always gonna
>>>> feel heavy.
>>>>
>>>> On Sunday, October 20, 2013 9:11:07 AM UTC+2, Thomas S wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have created a standard application relying on Pandas and PyQt4 to
>>>>> browse through a Pandas Dataframe.
>>>>> A dataframe is essentially a dictionary of time series data.
>>>>>
>>>>> I am new to web2py but I have experience with Pandas and matplotlib.
>>>>> I am also tempted to embed www.highcharts.com into my application.
>>>>>
>>>>> Before I dig into web2py I would like to know which route is probably
>>>>> most promising.
>>>>>
>>>>> Should I parse the dataframe on the webserver and write it into a SQL
>>>>> database? I guess that could be slow?
>>>>> Such a dataframe may consist of a dictionary with 100 elements each
>>>>> several 100k points.
>>>>>
>>>>> Should I parse a time series onto request into a json format and
>>>>> export to javaScript?
>>>>> In this case how could I provide a way to generate a menu from the
>>>>> keys in the dictionary.
>>>>> E.g. user clicks on a key, python does all the computations for some
>>>>> stats.
>>>>>
>>>>> The plan is to upload the data using csv files.
>>>>>
>>>>> So, I am a bit lost by the wide range of possibilities in web2py. I
>>>>> would be delighted if you would like to get involved in this open source
>>>>> project.
>>>>> The main goal for now is to learn web2py :-)
>>>>>
>>>>> Please find the Github of the original application here
>>>>> https://github.com/tschm/**PandasMonitor<https://github.com/tschm/PandasMonitor>
>>>>>
>>>>> Sorry for being so unprecise in my questions but it just reflects that
>>>>> I don't have a very precise plan at this stage.
>>>>>
>>>>> Kind regards
>>>>> Thomas
>>>>>
>>>> --
>> Resources:
>> - http://web2py.com
>> - http://web2py.com/book (Documentation)
>> - http://github.com/web2py/web2py (Source code)
>> - https://code.google.com/p/web2py/issues/list (Report Issues)
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "web2py-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
