El dt 10 de 10 del 2006 a les 16:31 -0500, en/na Jun Li va escriure:
> I figured out another way to do the job. Note: my table has 2.6
> million rows with 7 columns. 
> 
> src_rows = tbl_T.read(start=0, stop=tbl_T.nrows) 
> 
> The above statement will read the data of the whole table to the
> memory? am I right? src_rows will be an recorded array or nested
> recorded array right? 

That's right, but it is not necessary to load completely the table in
memory for doing what you want. See later.

> Then I figured out the correct indices of the piece of data I need and
> slicing them out and then do query on that chunk. 
> index1 = sp + int(sdate.absdays -
> integertoDate(tbl_attr_sdate).absdays) 
> index2 = sp + int(edate.absdays -
> integertoDate(tbl_attr_sdate).absdays) 
> txd = src_rows.field('tmax')[index1:index2] 

in such a case, you can avoid reading completely your table by using:

tbl_T.read(start=index1, stop=index2, field="tmax")

or better yet:

tbl_T.cols.tmax[index1:index2]

which is a shorthand of the former. With this, PyTables will only read
the part of the table that you are interested in.

> In this way, the performance improved considerably (from 140 s to 60
> s). However, the test performance is not in consistency, I test a lot
> of times. The run-time just switched from roughly 60 s to 110 s back
> and forth like randomly! is there any buffer or memory issue with
> read()? what' going on here? 

I don't know for sure, but completely reading the table in-memory will
create memory contention issues (and possibly will make your system to
swap intensively). Please, try out my suggestion and see if it cures
this.

> Another questions, Is there any function or quick way to INSERT a new
> row (rows) in the middle of an existing table or anywhere in the
> table. I know append() only append row(s) at the end of table and
> modifyRow() only modify existing row(s). Is pytable has such a
> function or if not, it is really handy to have this kind of basic
> function (consider adding to the new version maybe?) 

No, because it is more oriented to access to mostly read-only files,
PyTables doesn't have a quick way to insert a row (the only way is to
insert it in the middle of a copy of the entire table). If you need this
capability, perhaps you should use a relational database better.

Cheers,

-- 
>0,0<   Francesc Altet     http://www.carabos.com/
V   V   Cárabos Coop. V.   Enjoy Data
 "-"



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Reply via email to