On 2014-08-19 10:34, Kurt wrote:
> I am trying to process the following calendar and data attributes
> in a file: Da Mo Yr AttrA AttrB AttrC...
> I need to average AttrA for each of 365 Da days across Yr years.
> Then do the same for 27K files. Repeat for AttrB, AttrC etc. Can I
> do the averaging with lists or do I need some Python Db module (if
> it exists)?
Unless these files are huge *AND* you plan to perform this operation
repeatedly (rather than once), just do it in plain Python. (though
Python does offer both the "anydbm" and "sqlite3" modules for
database'ish stuff)
You omit some key details, so the example below would need to be
tweaked depending on things like:
- does it have headers
- how are the columns separated? (tabs, commas, spaces?)
- can a single date appear multiple times in a single file? (and if
so, what would you intend to do with it?)
- do you need aggregate averages across all the files, or just
per-file?
- are those values you're averaging integers or floats?
It would look something like (untested)
import csv
from collections import defaultdict
def report(fname, names_avgs):
"Do something with the averages"
print(fname)
for name, avg in names_avgs:
print(" %s: %f" % (name, avg))
def no_data(fname):
"Do something if a file has no data"
pass
for fname in fname_iter():
sums = defaultdict(float)
row_count = 0
with open(fname) as f:
r = csv.DictReader(f):
for row in r:
row_count += 1
for data in ("AttrA", "AttrB", "AttrC"):
sums[data] += float(row[data])
if row_count:
report(fname, [
(fieldname, total/row_count)
for fieldname, total
in sums
])
else:
no_data(fname)
I leave the details of fname_iter() and tweaking it based on the
omitted details as an exercise to the reader. :-)
-tkc
--
https://mail.python.org/mailman/listinfo/python-list
