Hi Antonio, Last question about this, from pytables point of view and based on your experience, is it better to manage a table with 3 million rows and multidimensional cells or a table with 300 million rows and plain cells?
Thank you, Juanma El Aug 5, 2012, a las 17:32, Antonio Valentino <antonio.valent...@tiscali.it> escribió: > Hi Juan Manuel, > > Il 05/08/2012 22:52, Juan Manuel Vázquez Tovar ha scritto: >> Hi Antonio, >> >> This is the piece of code I use to read the part of the table I need: >> >> data = [case[´loads´][i] for case in table] >> >> where i is the index of the row that I need to read from the matrix (133x6) >> stored in each cell of the column "loads". >> >> Juanma >> > > that looks perfectly fine to me. > No idea about what could be the issue :/ > > You can perfform patrial reads using Table.iterrows: > > data = [case[´loads´][i] for case in table.iterrows(start, stop)] > > Please also consider that using a single np.array with 1e8 rows instead > of a list of arrays will allows you to save the memory overhead of 1e8 > array objects. > Considering that 6 doubles are 48 bytes while an empty np.array takes 80 > bytes > > In [64]: sys.getsizeof(np.zeros((0,))) > Out[64]: 80 > > you should be able to reduce the memory footprint by far more than an half. > > > cheers > > >> 2012/8/5 Antonio Valentino <antonio.valent...@tiscali.it> >> >>> Hi Juan Manuel, >>> >>> Il 05/08/2012 22:28, Juan Manuel Vázquez Tovar ha scritto: >>>> Hi Antonio, >>>> >>>> You are right, I don´t need to load the entire table into memory. >>>> The fourth column has multidimensional cells and when I read a single row >>>> from every cell in the column, I almost fill the workstation memory. >>>> I didn´t expect that process to use so much memory, but the fact is that >>> it >>>> uses it. >>>> May be I didn´t explain very well last time. >>>> >>>> Thank you, >>>> >>>> Juanma >>>> >>> >>> Sorry, still don't understand. >>> Can you please post a short code snipped that shows how exactly do you >>> read data into your program? >>> >>> My impression is that somewhere you use some instruction that triggers >>> loading of unnecessary data into memory. > > > -- > Antonio Valentino > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Pytables-users mailing list > Pytables-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pytables-users ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
