Hi Siva,
On 11/10/2009 03:42 PM, Siva Subramanian wrote:
Hi Steve,
Thanks a lot. It works like a breeze when the files are small (< 100MB)
Umm <100MB is a /small/ file ? That is pretty large file as csv files go, IMHO.
However, it takes a jolly good 45 mins on a desktop with 2 GB of memory and
3.2 G P4
The code i am using is given below,
[...sinp...]
Well, I assume that the cids is relatively small compared to the number of
records in 'report_2_5.cvs'. Is that assumption correct ?
If it is, i think the bottle neck is the memory. I see that you delay writing
the output till after the processing of all the rows. In that case what do you
do with the information from the processed rows ? Does that remain in memory ?
If that is true, you might want to change the code to write out the output as
soon as it is available.
Anyways, that is just a guess. Could you please send over a sample dataset as
well as the code you've got till now so that I can take a look ? (I see Asokan
also has requested the same, maybe you'll benefit from both our approaches :) ).
btw, if your data-set is really larger than 100MB, you may want to consider
importing the data into a DB (like sqllite) before working on it. I already
mentioned this in my last mail, but no harm in saying it again. Here are the
links again:
http://code.activestate.com/recipes/498130/
http://docs.python.org/library/sqlite3.html
[...sinp...]
Aside, Steve, really appreciate the detailed email you had sent earlier. It
has really kick started me into python. :)
Excellent ! It is always a pleasure inducting new junkies to the python
addiction ;). You should consider joining one of the python user groups in
India. The bangpypers, which are the banglore python users group recently
conducted the first (very successful) Indian Python conference. I am sure the
people on this list who were involved/attended it would be able to provide some
links.
cheers,
- steve
--
random non tech spiel: http://lonetwin.blogspot.com/
tech randomness: http://lonehacks.blogspot.com/
what i'm stumbling into: http://lonetwin.stumbleupon.com/
_______________________________________________
To unsubscribe, email [email protected] with
"unsubscribe <password> <address>"
in the subject or body of the message.
http://www.ae.iitm.ac.in/mailman/listinfo/ilugc
