Re: [R] Query about memory used in list and dataframe
Hello, Also, note that to rbind data frames, like the op says, takes less memory than to cbind. x - 1:6 list.1 - list(x,x,x,x,x,x,x,x,x,x,x,x) test.df3 - as.data.frame(do.call(cbind, list.2)) test.df4 - as.data.frame(do.call(rbind, list.2)) object.size(list.1) object.size(test.df3) object.size(test.df4) (More columns == more variables == more memory.) Rui Barradas jholtman wrote I think you really have to show use your exact code that you did along with an 'str' of each intermediate data structure since my quick test does not bear out what you were saying: test.df - data.frame(a1= 1:6, a2= 1:6, a3 = 1:6, a4 = 1:6, a5 = 1:6, a6 = 1:6 + , a7=1:6, a8 = 1:6, a9 = 1:6, a10 = 1:6, a11 = 1:6, a12 = 1:6) test.df a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 6 object.size(test.df) 2264 bytes x - 1:6 test.df1 - data.frame(x,x,x,x,x,x,x,x,x,x,x,x) object.size(test.df1) 2264 bytes list.1 - list(x,x,x,x,x,x,x,x,x,x,x,x) object.size(list.1) 1032 bytes list.2 - NULL for (i in 1:12) list.2[[i]] - x object.size(list.2) 1032 bytes test.df3 - as.data.frame(do.call(cbind, list.2)) object.size(test.df3) 2264 bytes On Sat, May 5, 2012 at 5:50 PM, Shivam lt;shivamsingh@gt; wrote: Hi, I had a query regarding which object, a list or a dataframe, consumes more R memory. Let me clarify this: For example, I have a df of 6 rows and 12 columns, say 'test'. I do object.size() and find it uses 3.3 KB of memory. I run a loop and make a list, say 'testlist', of 6 elements, each element being the above mentioned df 'test'. The size of this list is 19.9 KB, understandably. Now I combine this list into a dataframe using rbind. The df formed has 12 cols and 36 rows. The size of this df is just 5.8 KB, almost a 75% reduction in memory. I had to work with a much larger list, and I thought of using the same method to convert my bigger list (62 dataframes, each having 4 cols and close to 200,000 rows) into a single dataframe. The big list, sat LIST A, had a size of 571 MB. But when I convert it into a dataframe, say DF A, using rbind, the object size increases to 1.35 GB. This was in contradiction to the earlier result. What am I missing? Why a 75% reduction in size in one case and double size in other? Anyone with any explanation? Sorry for the verbose email, just wanted to make my case clear. Thanks in advance, Regards Shivam [[alternative HTML version deleted]] __ R-help@ mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. __ R-help@ mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- View this message in context: http://r.789695.n4.nabble.com/Query-about-memory-used-in-list-and-dataframe-tp4611896p4612876.html Sent from the R help mailing list archive at Nabble.com. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
[R] Query about memory used in list and dataframe
Hi, I had a query regarding which object, a list or a dataframe, consumes more R memory. Let me clarify this: For example, I have a df of 6 rows and 12 columns, say 'test'. I do object.size() and find it uses 3.3 KB of memory. I run a loop and make a list, say 'testlist', of 6 elements, each element being the above mentioned df 'test'. The size of this list is 19.9 KB, understandably. Now I combine this list into a dataframe using rbind. The df formed has 12 cols and 36 rows. The size of this df is just 5.8 KB, almost a 75% reduction in memory. I had to work with a much larger list, and I thought of using the same method to convert my bigger list (62 dataframes, each having 4 cols and close to 200,000 rows) into a single dataframe. The big list, sat LIST A, had a size of 571 MB. But when I convert it into a dataframe, say DF A, using rbind, the object size increases to 1.35 GB. This was in contradiction to the earlier result. What am I missing? Why a 75% reduction in size in one case and double size in other? Anyone with any explanation? Sorry for the verbose email, just wanted to make my case clear. Thanks in advance, Regards Shivam [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Query about memory used in list and dataframe
I think you really have to show use your exact code that you did along with an 'str' of each intermediate data structure since my quick test does not bear out what you were saying: test.df - data.frame(a1= 1:6, a2= 1:6, a3 = 1:6, a4 = 1:6, a5 = 1:6, a6 = 1:6 + , a7=1:6, a8 = 1:6, a9 = 1:6, a10 = 1:6, a11 = 1:6, a12 = 1:6) test.df a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 6 object.size(test.df) 2264 bytes x - 1:6 test.df1 - data.frame(x,x,x,x,x,x,x,x,x,x,x,x) object.size(test.df1) 2264 bytes list.1 - list(x,x,x,x,x,x,x,x,x,x,x,x) object.size(list.1) 1032 bytes list.2 - NULL for (i in 1:12) list.2[[i]] - x object.size(list.2) 1032 bytes test.df3 - as.data.frame(do.call(cbind, list.2)) object.size(test.df3) 2264 bytes On Sat, May 5, 2012 at 5:50 PM, Shivam shivamsi...@gmail.com wrote: Hi, I had a query regarding which object, a list or a dataframe, consumes more R memory. Let me clarify this: For example, I have a df of 6 rows and 12 columns, say 'test'. I do object.size() and find it uses 3.3 KB of memory. I run a loop and make a list, say 'testlist', of 6 elements, each element being the above mentioned df 'test'. The size of this list is 19.9 KB, understandably. Now I combine this list into a dataframe using rbind. The df formed has 12 cols and 36 rows. The size of this df is just 5.8 KB, almost a 75% reduction in memory. I had to work with a much larger list, and I thought of using the same method to convert my bigger list (62 dataframes, each having 4 cols and close to 200,000 rows) into a single dataframe. The big list, sat LIST A, had a size of 571 MB. But when I convert it into a dataframe, say DF A, using rbind, the object size increases to 1.35 GB. This was in contradiction to the earlier result. What am I missing? Why a 75% reduction in size in one case and double size in other? Anyone with any explanation? Sorry for the verbose email, just wanted to make my case clear. Thanks in advance, Regards Shivam [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Jim Holtman Data Munger Guru What is the problem that you are trying to solve? Tell me what you want to do, not how you want to do it. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
