Hello Nikola,
In general, larger chunk sizes will increase read speed.
Additionally, your problem sounds like a perfect place to use compression,
which can both decrease storage space and increase read speed (use blosc
compression this). Please refer to [1] for more information.
In general, if you know *a priori* that you have a hard maximum table size
that you will never go over, you can simply set your chunksize to this
value. On the other hand, if you know a minimum size that you will be
removing and this size is "large enough" then it makes sense to use this as
the chunksize sometimes too.
Be Well
Anthony
1. http://pytables.github.com/usersguide/optimization.html
On Fri, May 11, 2012 at 5:15 AM, nikola stevanovic <nidzadra...@gmail.com>wrote:
> *Hi everyone, *
>
> I'm new member and it's nice to meet you all.
> I need some advices about my work with pytables. The problem is next. I'm
> working on some kind of database using pytables and of course hdf5 format.
> I created table with *six columns, row size 92B*. One column in table is
> Time32Col. This column will be *indexed*. Table *will be updated* every
> couple days (rows will be appended on existing table). *Between every
> update users can create queries on table and consume data*. My question
> is how efficiently balance chunksize between updates, because numbers of
> rows in table will be start from *0 to 10 000 000 000* during the time?
> After this number I will start archiving process, i.e. for example remove
> first five billions rows and store in some other table for archiving. Of
> course, I need this balance because *reading speed*. So, what is most
> efficient way for setting chunksize for my problem? Sorry for my english.
>
> *
> Thanks for advice guys.
> Cheers!
> Nikola*
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
