[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 --- Comment #17 from John Nagle --- Update: saving as .ods took about 1.5 hours, but finished. Opening file thus saved took about 5 minutes to open. -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 --- Comment #16 from John Nagle --- File load time for 233MB file is only 30 seconds. Scrolling around in the file works fine. It's only "Save" that's slow. Save as .ods is also extremely slow. It's not just .csv. So it's not the CSV exporter. About 5 minutes into that now. As before, 100% of 1 CPU in use, very little I/O, no thrashing, not out of memory. -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 --- Comment #15 from John Nagle --- Update: after over 2 hours, the writing of the CSV file finished. The output file is valid. Calc is back to its normal operating state. Strongly suggest profiling and looking for something that's O(N^2) in file exporting. -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 --- Comment #14 from John Nagle --- I'm seeing a similar bug. Read a 233MB CSV file into Calc without problems. Attempted to write it out as CSV with slightly different options ("quote everything"), and it's half done after an hour. About 2/3 done after an hour and a half. Progress continues. Calc is in the "greyed out" state, not processing mouse clicks. Since this is on Linux, it doesn't affect other processes much, and I'm letting this run to see if it finishes. It's not hung; the progress bar advances slowly. CPU utilization 100% of 1 CPU. Very little disk I/O. Memory not full. Not thrashing. Very little I/O, maybe one disk write every 10 seconds. This looks like there's something in writing out a CSV file that's worse than O(1), so it chokes on large files. Somebody with a debug build should profile this while writing out a large CSV file and see where it's spending all the CPU time. Ubuntu 16.04LTS x64 8GB. Calc version: libreoffice-calc 1:5.1.6-rc2.0ubuntu1-xenial2 -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Telesto changed: What|Removed |Added See Also||https://bugs.documentfounda ||tion.org/show_bug.cgi?id=95 ||346 -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 --- Comment #13 from Jean-Baptiste Faure --- I tried to export the attached csv file to .ods with LO 5.4.1.0.0+ built at home under Ubuntu 16.04 x86-64 but it failed with error I/O message. No problem to load the csv file, it was long but it worked. The PC has 8 Go of RAM and a SSD 512 Go. System monitor showed that process soffice.bin used 1,1 Go of RAM. Best regards. JBF -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Telesto changed: What|Removed |Added See Also||https://bugs.documentfounda ||tion.org/show_bug.cgi?id=10 ||8631 -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Bartosz changed: What|Removed |Added Blocks|88175 | Referenced Bugs: https://bugs.documentfoundation.org/show_bug.cgi?id=88175 [Bug 88175] [Meta] Tracking bug for limitations of XLSX (OOXML) format -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Telesto changed: What|Removed |Added Keywords||filter:xlsx Blocks||88175 Referenced Bugs: https://bugs.documentfoundation.org/show_bug.cgi?id=88175 [Bug 88175] [Meta] Tracking bug for limitations of XLSX (OOXML) format -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Bartosz changed: What|Removed |Added CC||gan...@poczta.onet.pl --- Comment #12 from Bartosz --- Created attachment 128996 --> https://bugs.documentfoundation.org/attachment.cgi?id=128996&action=edit Random value CSV file ~237MB (unpacked) I have tested attached CSV export to .ODS and .XLSX. In both format there are needed about 25GB of memory to save to desired formats. The difference is that during exporting to .ODS, the hard disc is used (you could noticed that free disc space is decreasing). The RAM usage for LibreOffice during exporting to .ods is constant (about 1200MB) Generally exporting into .ods is successful. Unforunately during exporting into .xlsx, only RAM is used. There is no disc usage, during exporting into .xlsx It means that during exporting to .xlsx, the amount of free RAM memory is dramatically decreasing until it crash. In my opinion desired solution for exporting to .xlsx, is saving data on disc, as it is already done for .ods. -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Bartosz changed: What|Removed |Added Status|REOPENED|NEW -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 --- Comment #11 from Holli Bleßmann --- This IS still a Bug. Have same issue with a 237MB large CSV file on xubuntu 16.04LTS 64bit running on an intel core i7 with 8GB RAM. Trying to save that file as a M$ Excel 2010 .xlsx ended up in getting out of memory (full use of RAM AND 5GB SWAP PARTITION!) and -after 30 minutes- a crashed LibreOffice Calc (v5.1.2.x). Btw: xubuntu with loaded LibreOffice Calc with an empty sheet needs aprox. 250MB of RAM and no use of the swap partition. As Markus Mohrhard mentioned above, I expected a normal RAM consumption of (temporary) 2x document size + needed additional spreadsheet data. In my case something about 500MB, NOT 12.5GB! So, this bug still exists. -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Holli Bleßmann changed: What|Removed |Added Status|RESOLVED|REOPENED Resolution|NOTABUG |--- --- Comment #10 from Holli Bleßmann --- This IS still a Bug. Have same issue with a 237MB large CSV file on xubuntu 16.04LTS 64bit running on an intel core i7 with 8GB RAM. Trying to save that file as a M$ Excel 2010 .xlsx ended up in getting out of memory (full use of RAM AND 5GB SWAP PARTITION!) and -after 30 minutes- a crashed LibreOffice Calc (v5.1.2.x). Btw: xubuntu with loaded LibreOffice Calc with an empty sheet needs aprox. 250MB of RAM and no use of the swap partition. As Markus Mohrhard mentioned above, I expected a normal RAM consumption of (temporary) 2x document size + needed additional spreadsheet data. In my case something about 500MB, NOT 12.5GB! So, this bug still exists. -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Markus Mohrhard changed: What|Removed |Added Status|ASSIGNED|RESOLVED Resolution|--- |NOTABUG --- Comment #9 from Markus Mohrhard --- I'm not sure what people expect. Just storing the 500 MB string would require us to use 500 MB of memory. There is not yet anything additional that we have allocated then for this data like cell information, ... If you need to process 500 MB csv files in a spreadsheet you need quite some memory. There is no way around it. -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Bartosz changed: What|Removed |Added Status|NEW |ASSIGNED -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 Holli Bleßmann changed: What|Removed |Added CC||holli.blessm...@gmx.de --- Comment #8 from Holli Bleßmann --- *** Bug 103267 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug.___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.documentfoundation.org/show_bug.cgi?id=84246 --- Comment #7 from QA Administrators --- ** Please read this message in its entirety before responding ** To make sure we're focusing on the bugs that affect our users today, LibreOffice QA is asking bug reporters and confirmers to retest open, confirmed bugs which have not been touched for over a year. There have been thousands of bug fixes and commits since anyone checked on this bug report. During that time, it's possible that the bug has been fixed, or the details of the problem have changed. We'd really appreciate your help in getting confirmation that the bug is still present. If you have time, please do the following: Test to see if the bug is still present on a currently supported version of LibreOffice (5.0.1 or preferably 5.0.2.2 or later) https://www.libreoffice.org/download/ If the bug is present, please leave a comment that includes the version of LibreOffice and your operating system, and any changes you see in the bug behavior If the bug is NOT present, please set the bug's Status field to RESOLVED-WORKSFORME and leave a short comment that includes your version of LibreOffice and Operating System Please DO NOT Update the version field Reply via email (please reply directly on the bug tracker) Set the bug's Status field to RESOLVED - FIXED (this status has a particular meaning that is not appropriate in this case) If you want to do more to help you can test to see if your issue is a REGRESSION. To do so: 1. Download and install oldest version of LibreOffice (usually 3.3 unless your bug pertains to a feature added after 3.3) http://downloadarchive.documentfoundation.org/libreoffice/old/ 2. Test your bug 3. Leave a comment with your results. 4a. If the bug was present with 3.3 - set version to "inherited from OOo"; 4b. If the bug was not present in 3.3 - add "regression" to keyword Feel free to come ask questions or to say hello in our QA chat: http://webchat.freenode.net/?channels=libreoffice-qa Thank you for your help! -- The LibreOffice QA Team This NEW Message was generated on: 2015-10-14 -- You are receiving this mail because: You are the assignee for the bug. ___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.freedesktop.org/show_bug.cgi?id=84246 --- Comment #6 from Owen Genat --- (In reply to Owen Genat from comment #2) > Around 2.5GB of RAM is used by LO during this process. This was a generalisation. (In reply to Marcelo from comment #5) > I have 4GB RAM, I think the machine is likely running out of RAM. Further test results using the provided script under GNU/Linux with v4.2.6.3: On a system with 3708MB RAM, no swap. CSV records/MB XLSX MB Peak RAM VIRT/RES[1] -- --- 3/~157 ~119 2046/1.1 4/~210 ~159 2378/1.6 5/~262 ~200 2871/2.0 6/~315 ~238 3268/2.3 7/~367 ~278 3563/2.6 75000/~394 ~298 3750/2.8 8/~420 N/A 3943/2.9[2] On a system with 7941MB RAM, no swap. CSV records/MB XLSX MB Peak RAM VIRT/RES[1] -- --- 10/~525 ~397 4816/3.7 [1] Values of virtual (MB) and resident (GB) usage displayed by the top command. [2] At this point Calc crashes. -- You are receiving this mail because: You are the assignee for the bug. ___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.freedesktop.org/show_bug.cgi?id=84246 --- Comment #5 from Marcelo --- I have 4GB RAM, processor Intel Core i5-3230M 2.60GHz, running elementary OS 0.2.1 64-bit -- You are receiving this mail because: You are the assignee for the bug. ___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.freedesktop.org/show_bug.cgi?id=84246 Jean-Baptiste Faure changed: What|Removed |Added CC||jbfa...@libreoffice.org --- Comment #4 from Jean-Baptiste Faure --- @Marcello: How much RAM do you have on your machine? Best regards. JBF -- You are receiving this mail because: You are the assignee for the bug. ___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs
[Libreoffice-bugs] [Bug 84246] Calc crashes on loading / saving large (~500MB) CSV files
https://bugs.freedesktop.org/show_bug.cgi?id=84246 Owen Genat changed: What|Removed |Added Status|UNCONFIRMED |NEW Component|Libreoffice |Spreadsheet Summary|Calc hanging|Calc crashes on loading / ||saving large (~500MB) CSV ||files Ever confirmed|0 |1 --- Comment #3 from Owen Genat --- I am not sure whether there is anything the developers can do about handling CSV files of this magnitude, but for now confirmed that there is an issue. Status set to NEW. Component set to Spreadsheet. Summary amended for clarity. -- You are receiving this mail because: You are the assignee for the bug. ___ Libreoffice-bugs mailing list Libreoffice-bugs@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/libreoffice-bugs