Sorry for many posts, but, I think, I found something interesting (or I just think so :)). I digged a *stat*() usage in ParaView at official GitHub source repository. If I get it right, there are two types of this function: *stat*() and *stat64*(). My only program that deals with large files is a disk editor & NTFS data recovery for Windows (it's free and was made at home), this program uses WinAPI, not C/C++ calls for file operations. So I don't know much about fopen/fclose/stat/e.t.c. on large files. So we have *stat* and *stat64* + corresponding data structures. We can see an example in this source: https://github.com/Kitware/VTK/blob/eae7f4446c867ee9d29b466676fa11282d56dafc/IO/LSDyna/private/LSDynaFamily.cxx For 64-bit configurations this code uses *stat64()* and *stat64* structure.
But if we look at the problematic place in *EnSight reader* we will only see the "usual" *stat*() that is, to my current knowledge, is 32 bit: https://github.com/Kitware/VTK/blob/master/IO/EnSight/vtkEnSightGoldBinaryReader.cxx (Please see lines 100 and 101) May it be the cause of the subj? Sorry if I'm wrong, I'm not a professional programmer. 2016-03-01 9:38 GMT+03:00 Andrew <[email protected]>: > I took a look at the code. Te problematic fragment (line 114 in > vtkEnSightGoldBinaryReader) just checks the file's existance and opens it > in standard way. The problem occures in *stat*() function that is fed > with the full file name and the reference to structure to receive file > params. I tried to read the file from Python shell and the result is the > same. I guessed that the reason is an additional slash before the short > file name (see error message) but small files are processed OK so the only > assumption I can make is that the problem occures in the *stat*() itself > on large (> ~2GiB) files due to some strange 32-bit relicts (although > itshould work perfectly in 64-bit builds). > Can anybody test the opening of large (3...4 GiB or more) EnSight Gold > binary files in ParaView 5.0.0 or 5.0.1? I cannot check any assumptions > because I only have Linux compilators set up and cannot do the > "hello-world"-like stat() test under Windows on a large file... > > 2016-03-01 8:53 GMT+03:00 Andrew <[email protected]>: > >> Hello Samuel. >> >> Sorry, but I can't undersatand how can the file fragmentation affect it's >> readability and parsing. Even if the file has thousands of fragments and >> "brain-damaging" :) runlist in many extended FILE records on NTFS >> partition, your program, if it uses Win API, will see it like a solid file >> (althougt there will be very low performance of sequental access). So there >> should be no any faults on reading highly fragmented files. >> I have checked out the "direct" reading of EnSight case from the cluster >> via samba with ParaView 5.0.0 on Windows machine. The same error appears as >> in my first message, although there are *different files and filesystems* >> . >> I think the problem is not a fragmentation but a *file size*. Some >> 32-bit-like relicts or so in EnSight reader (it's just a guess, sorry if >> I'm wrong). But my hypothesis does not asks the question why does the bug >> affect only Windows versions... Maybe there are differences between >> versions concerning data types. Do you have any >4 GB binary EnSight 6 or >> Gold case to test? I can't (not allowed) to publish my case and it's also >> big enough for upload (although theoretically I can do it from my home PC). >> >> >> 2016-02-29 20:27 GMT+03:00 Samuel Key <[email protected]>: >> >>> Andrew, >>> >>> Let us focus on MS Windows for a moment. The NTS file system and the use >>> of the file system by the O/S is not quite as it should be. >>> >>> In my experience, the above combination routinely creates an >>> unbelievable number of file fragments. Think in terms of confetti. As time >>> goes by, the fragmentation builds. Even though you have a respectable >>> amount of RAM, the free space itself on the disk driver can get rather >>> disorganized. >>> >>> You can start by using the MS-supplied defrager. Run this utility five >>> to seven times in a row and things might start looking better. However the >>> free-space can still remain disjointed. Personally, I prefer a 3rd party >>> utility for organizing and defraging the NTS file system (for example, >>> PerfectDisk from Raxco). Even this utility has to run a couple of times >>> plus once during the O/S re-boot phase to get a decent amount of >>> consolidated free space and a somewhat consolidate O/S paging file. All of >>> this has to be done on the O/S partition (usually C:) and the partition >>> where you read in the 5+ GB Ensight Geometry file. >>> >>> On my machine, the MS Win7 O/S has its own partition (100GB \C:) plus >>> another smaller dedicated partition for a second paging file (28GB \F:). >>> All of my simulation work including the EnSight results file-sets reside on >>> an entirely separate disk drive (280GB \G: that now needs to be even >>> larger). The 3rd party file consolidation software runs "day and night" on >>> the \G: partition. >>> >>> Just to emphasize the point, copying the 5+ GB EnSight geometry file on >>> to workstation can result in many scattered fragments even when the >>> free-space starts out consolidated. If the free space is not consolidated, >>> the file fragmentation will be even more. >>> >>> The attached screen capture, shows the O/S managed paging-file (brown) >>> in the \C: partition is beginning to fragment. >>> >>> If you go down this path and it helps, please let me know the outcome. >>> >>> Regards, >>> >>> Sam Key >>> >>> On 2/29/2016 1:04 AM, Andrew wrote: >>> >>> Another "experimental data". I was able to load this big case in >>> ParaView viat the CGNS format. It was neccesary to select CGNS 3.0 and HDF5 >>> options in CFX Converter. But when I apply the Extract Block filter I see >>> no any boundaries, only fluid domain (that is displayed OK). So CGNS format >>> is not a wayout... >>> >>> 2016-02-29 9:00 GMT+03:00 Andrew <[email protected]>: >>> >>>> I performed another test with *EnSi.ght 6* (not Gold) format. >>>> Conversion was made on Win-7 x64 machine. The same results: PV 4.3 crashed, >>>> PV 5.0 "cannot stat" (full error message is in my first post on this >>>> topic). Sorry, now I need to work on with my report... >>>> >>>> >>>> >>>> >>>> >>>> 2016-02-29 8:47 GMT+03:00 Andrew < <[email protected]> >>>> [email protected]>: >>>> >>>>> Hello. Sorry for silence, I'm usually offline on holydays, although I >>>>> have internet connection and read your mail. >>>>> >>>>> Samuel Key >>>>> >>>>> Thanks for your attention. >>>>> >>>>> About transferring files from Linux to Windows. I use binary EnSight >>>>> files because CFX converter fails to write so large files in ASCII format. >>>>> But I tried to convert from CFX to EnSight (binary) on the same Win-7 x64 >>>>> machine with ParaView. Both installed versions (4.3 and 5.0) failed to >>>>> open >>>>> the geometry (I unchecked all fields). ParaView 4.3 crashed, ParaView 5.0 >>>>> just cannot open file... >>>>> >>>>> I think that the root cause may be a geometry file size. It is 4.9 GiB >>>>> (5.3 GB), mesh have about 150 millions of cells, not 600 thusands like in >>>>> your case. Its possible that ParaView for Windows cannot process so large >>>>> EnSight files, although the Linux version reads in this geometry and >>>>> displays outline (swapping is so intensive so I cannot displaye something >>>>> else). >>>>> >>>>> 2016-02-26 19:31 GMT+03:00 Samuel Key < <[email protected]> >>>>> [email protected]>: >>>>> >>>>>> Andrew, >>>>>> >>>>>> For what it is worth, I use ParaView >>>>>> (ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit) on Windows-7 64bit. I just now >>>>>> tested two medium-sized, EnSight Gold-formatted simulation results files >>>>>> (600,000 finite elements) and both worked. >>>>>> >>>>>> In the past, I have had difficulty with the EnSight ASCII-formatted >>>>>> *.case file when moving the simulation results (*.case file and friends) >>>>>> from MS Windows to EnSight running on a linux platform. It was necessary >>>>>> to >>>>>> pass the ASCII *.case file through a 'dos2unix' filter. Transferring >>>>>> files >>>>>> the other way might have the same problem??? >>>>>> >>>>>> Hope this helps. >>>>>> >>>>>> Samuel Key >>>>>> FMA Development, LLC >>>>>> 1005 39th Ave NE >>>>>> Great Falls, Montana 59404 >>>>>> USA >>>>>> >>>>>> On 2/25/2016 11:26 PM, Andrew wrote: >>>>>> >>>>>> Hello. We use Ansys CFX in our work but the standard post-processor >>>>>> CFD-Post consumes one of Ansys licenses while Paraview is free. So I >>>>>> often >>>>>> make pictures for reports with ParaView to preserve Ansys license for >>>>>> another users/needs (because other users are not familiar with ParaView >>>>>> and >>>>>> CFD-Post may consume general "acfd" license feature that may be used for >>>>>> calculations). I convert CFX results to EnSight Gold format in CFX Solver >>>>>> Manager (Export Results) and then open in ParaView. It was working well >>>>>> on >>>>>> Linux (CentOS 6.7 x64, Ubuntu 12.04 x64 with updates) but now I work with >>>>>> large files and I need to use another machine that has 128 GB of RAM and >>>>>> runs Windows-7 x64. >>>>>> I downloaded and installed the following version: >>>>>> ParaView-5.0.0-Qt4-OpenGL2-Windows-64bit.exe >>>>>> When I open my EnSight-format file I receive the error message that >>>>>> geometry file cannot be read. I tried to move EnSight files to a simple >>>>>> location (D:\Temp) and renamed them (test.case, test.geom + edited the >>>>>> case >>>>>> to change geometry file name to "simple" one). The same result, it cannot >>>>>> find geometry file... Full error message is as follows: >>>>>> >>>>>> ERROR: In >>>>>> C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx, >>>>>> line 114 >>>>>> vtkEnSightGoldBinaryReader (0000000009451230): stat failed. >>>>>> >>>>>> >>>>>> ERROR: In >>>>>> C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightGoldBinaryReader.cxx, >>>>>> line 220 >>>>>> vtkEnSightGoldBinaryReader (0000000009451230): Unable to open file: >>>>>> D:\Storage\Media\Work\Gradirni\CFX\EnSight\C-G002L3S-R020-S000-14.5\/C-G002L3S-R020-S000.geom >>>>>> >>>>>> >>>>>> ERROR: In >>>>>> C:\bbd\df0abce0\source-paraview\VTK\IO\EnSight\vtkEnSightReader.cxx, line >>>>>> 306 >>>>>> vtkEnSightGoldBinaryReader (0000000009451230): error reading geometry >>>>>> file >>>>>> >>>>>> >>>>>> ERROR: In >>>>>> C:\bbd\df0abce0\source-paraview\VTK\Common\ExecutionModel\vtkExecutive.cxx, >>>>>> line 784 >>>>>> vtkCompositeDataPipeline (0000000008E1A640): Algorithm >>>>>> vtkEnSightGoldBinaryReader(0000000009451230) returned failure for >>>>>> request: >>>>>> vtkInformation (00000000093C3920) >>>>>> Debug: Off >>>>>> Modified Time: 214873 >>>>>> Reference Count: 1 >>>>>> Registered Events: (none) >>>>>> Request: REQUEST_DATA >>>>>> ALGORITHM_AFTER_FORWARD: 1 >>>>>> FORWARD_DIRECTION: 0 >>>>>> FROM_OUTPUT_PORT: 0 >>>>>> >>>>>> OK, I tried to open it in ParaView 4.3. It says nothing (only border >>>>>> of the message window appears) and crashes. At that moment I see in the >>>>>> status bar that EnSight reader progress is 100%. >>>>>> >>>>>> Both ParaView 5.0.0 and 4.3 was tested with EnSight files exported on >>>>>> Linux and Windows machine. It didn't help. >>>>>> >>>>>> OK, I tried to export CFX results to CGNS format. ParaView 5.0.0 >>>>>> (without MPI) cannot read it, so I fed it to ParaView 4.3. It says that >>>>>> its >>>>>> "unable to filed any meshes". >>>>>> >>>>>> I also performed an experiment on my Linux laptop. It has only 16 GB >>>>>> of RAM so its unreal to work with such a big file on this machine but it >>>>>> reads the same EnSight case and displays the domain! No arrays selected >>>>>> (mesh only), swap file grows up to 8 GB but the file opens without errors >>>>>> and crashes. It's only a test because I have not any Linux machine with >>>>>> memory large enough (our Linux cluster has 128 GB of RAM on every node >>>>>> including master but it has very poor video). >>>>>> >>>>>> I switched to CFD Post for this case but I want to know are there any >>>>>> workarounds to open my EnSight-format results in ParaView for Windows. >>>>>> >>>>>> Thanks for your attention. >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Powered by www.kitware.com >>>>>> >>>>>> Visit other Kitware open-source projects at >>>>>> http://www.kitware.com/opensource/opensource.html >>>>>> >>>>>> Please keep messages on-topic and check the ParaView Wiki at: >>>>>> http://paraview.org/Wiki/ParaView >>>>>> >>>>>> Search the list archives at: http://markmail.org/search/?q=ParaView >>>>>> >>>>>> Follow this link to >>>>>> subscribe/unsubscribe:http://public.kitware.com/mailman/listinfo/paraview >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Powered by <http://www.kitware.com>www.kitware.com >>>>>> >>>>>> Visit other Kitware open-source projects at >>>>>> <http://www.kitware.com/opensource/opensource.html> >>>>>> http://www.kitware.com/opensource/opensource.html >>>>>> >>>>>> Please keep messages on-topic and check the ParaView Wiki at: >>>>>> <http://paraview.org/Wiki/ParaView>http://paraview.org/Wiki/ParaView >>>>>> >>>>>> Search the list archives at: <http://markmail.org/search/?q=ParaView> >>>>>> http://markmail.org/search/?q=ParaView >>>>>> >>>>>> Follow this link to subscribe/unsubscribe: >>>>>> http://public.kitware.com/mailman/listinfo/paraview >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> _______________________________________________ >>> Powered by www.kitware.com >>> >>> Visit other Kitware open-source projects at >>> http://www.kitware.com/opensource/opensource.html >>> >>> Please keep messages on-topic and check the ParaView Wiki at: >>> http://paraview.org/Wiki/ParaView >>> >>> Search the list archives at: http://markmail.org/search/?q=ParaView >>> >>> Follow this link to >>> subscribe/unsubscribe:http://public.kitware.com/mailman/listinfo/paraview >>> >>> >>> >>> _______________________________________________ >>> Powered by www.kitware.com >>> >>> Visit other Kitware open-source projects at >>> http://www.kitware.com/opensource/opensource.html >>> >>> Please keep messages on-topic and check the ParaView Wiki at: >>> http://paraview.org/Wiki/ParaView >>> >>> Search the list archives at: http://markmail.org/search/?q=ParaView >>> >>> Follow this link to subscribe/unsubscribe: >>> http://public.kitware.com/mailman/listinfo/paraview >>> >>> >> >
_______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView Search the list archives at: http://markmail.org/search/?q=ParaView Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/paraview
