Hi:
I have a 300,000+ line Pytable having 25 columns of various data,
mostly floats.
I need to do a set of simultaneous searches on this data, based on
observation type, time, location...
Here is the code I am using to do the search (here I am just
searching for 5 columns of data, I would like to do all 25 and have
in other ways, not much diff):
--------------
for o in table.iterrows():
ob_date = datetime.datetime(o['year'], o['month'], o
['day'], o['hour'], o['min'], o['sec'])
if o['status'] == status_in:
if o['type'] == obtype_in
and \
tgrid[0] <= ob_date < tgrid[1] and \
xgrid[1] <= o['x'] < xgrid[2]
and \
ygrid[1] <= o['y'] < ygrid[2]
and \
zgrid[1] <= o['z'] < zgrid[2]:
value.append(o['value'])
Hxa.append(o['Hxa'])
Hxf.append(o['Hxf'])
sdHxa.append(o['sdHxa'])
sdHxf.append(o['sdHxf'])
------------------
the only "funny" deal here is that I am using the python datetime
module to compare the times, but from my tests and trials with and
without it I don't think that is the problem (?).
It takes about 7 sec on the Mac G5 (2.5 ghz) and 8.5 sec on an SGI
Altix. I think that is not very good, right?
I have also tried "inlining" each variable with a search (really
slow), or try using the table.where functions, again, slow.
Another problem I am having with this is that I need to search over
20-30 individual time bins (like above), and I cannot figure out a
way to bin up the data
in one pass, e.g., I am calling the above loop 20-30 times, so to
datamine this table I have a longer wait than I thought I would....
Ideas or comments?
Lou Wicker
------------------------------------------------------------------------
----
| Dr. Louis J. Wicker
| Research Scientist, National Severe Storms Lab
| 1313 Halley Cir, Norman, OK 73069
| E-mail: [EMAIL PROTECTED]
| HTTP: www.nssl.noaa.gov/~lwicker
| Phone: (405) 366-0416
| Fax: (405) 366-0472
|
|
| Two wrongs don't make a right, but three left's do...
------------------------------------------------------------------------
----
|
| "The contents of this message are mine personally and
| do not reflect any position of the Government or NOAA."
|
------------------------------------------------------------------------
----
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Pytables-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/pytables-users
