Uwe: Thanks for your help also! This process takes a messy flat file,
cleans it up and eliminates garbage, then using the algorithm Ed
provided, breaks the data into records.
The file ends up being over a thousand records with 17 fields per record.
Reading the file, processing it and writing it out takes a fraction of a
second. This is considerably faster than I was expecting.
Jeff
Jeff Johnson
[EMAIL PROTECTED]
SanDC, Inc.
623-582-0323
Fax 623-869-0675
Uwe Grauer wrote:
> Ed Leafe wrote:
> ...
>> There are probably several other ways to accomplish this task; I
>> chose this one because it's very flexible. Try changing the value of
>> 'flds' to any other value, and see how the output changes.
>>
>
> I did a quick test.
> ----------------------------------------------------------------
> # t_ed.py
> # Number of fields per record
> flds = 3
> data = ["%s" % i for i in range(3000000)]
>
> def test_ed(data):
> nthFields = [data[n::flds] for n in range(flds)]
> recTuples = zip(*nthFields)
> recs = [",".join(tup) for tup in recTuples]
> output = "\n".join(recs)
> file("data.txt", "w").write(output)
>
> test_ed(data)
> ----------------------------------------------------------------
> # t_ug.py
> # Number of fields per record
> #flds = 3
> data = ["%s" % i for i in range(3000000)]
>
> def test_ug(data):
> alen = len(data)
> i = 0
> f = open("data.txt", "w")
> while(i+3 <= alen):
> f.write("%s,%s,%s\n" % (data[i], data[i+1], data[i+2]))
> i = i+3
> f.close()
>
> test_ug(data)
> ----------------------------------------------------------------
>
> Times for the two versions:
>
> time python t_ed.py
> real 0m27.093s
> user 0m26.302s
> sys 0m0.792s
>
> time python t_ug.py
> real 0m9.525s
> user 0m8.985s
> sys 0m0.536s
>
> Plain loops are a lot faster and do not consume such a lot of memory.
>
> Uwe
>
>
_______________________________________________
Post Messages to: [email protected]
Subscription Maintenance: http://leafe.com/mailman/listinfo/dabo-users
Searchable Archives: http://leafe.com/archives/search/dabo-users
This message: http://leafe.com/archives/byMID/dabo-users/[EMAIL PROTECTED]
