Hi Ulrich, Thanks for your idea. I resolved the issue by making use of integers instead of strings. Earlier I had many duplicate strings in different rows retrieved from database. I created a list to have unique strings and their indexes are used in actual computations. So lot of space is saved by having integers instead of strings.
I am trying differnt approach too, to get ids from database instead of gettign direct values. values will be retrived by using unique ids of different tables. I hope, I will be able to improve perforamnce with this approach. Regards, Rama On 7/6/11, Ulrich Eckhardt <ulrich.eckha...@dominolaser.com> wrote: > Rama Rao Polneni wrote: >> After storing 1.99 GB data in to the dictionary, python stopped to >> store the remaining data in to dictionary. > > Question here: > - Which Python? > - "stopped to store" (you mean "stopped storing", btw), how does it behave? > Hang? Throw exceptions? Crash right away? > > >> Memory utilization is 26 GB/34GB. That means, still lot memory is left >> as unutilized. > > 2GiB is typically the process limit for memory allocations on 32-bit > systems. So, if you are running a 32-bit system or running a 32-bit process > on a 64-bit system, you are probably hitting hard limits. With luck, you > could extend this to 3GiB on a 32-bit system. > > >> Is this proplem becasue of large memory utlization. > > I guess yes. > > >> Is there any alternate solution to resolve this issue. Like splitting >> the dictionaries or writing the data to hard disk instead of writing >> to memory. > > If you have lost of equal strings, interning them might help, both in size > and speed. Doing in-memory compression would be a good choice, too, like > e.g. if you have string fields in the DB that can only contain very few > possible values, converting them to an integer/enumeration. > > Otherwise, and this is a more general approach, prefer making a single sweep > over the data. This means that you read a chunk of data, perform whatever > operation you need on it, possibly write the results and then discard the > chunk. This keeps memory requirements low. At first, it doesn't look as > clean as reading the whole data in one step, calculations as a second and > writing results as a third, but with such amounts of data as yours, it is > the only viable step. > > Good luck, and I'd like to hear how you solved the issue! > > Uli > > -- > Domino Laser GmbH > Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
