On 05/19/2012 06:30 PM, Mohammad Mirzadeh wrote:
Hi guys,
I need a little bit of advice on how I'm outputting data. So far I have
been using a simple legacy VTK format and dumping my grid and all
sclaar/vectors into that single file. Basically I start with writing the
grid and then append as many as variables that I like to the file. This
has the problem that it was ascii and slow for large data sets,
especially fro a parallel code where the solver was taking only 5 sec
but the IO was around 20!!
So I want to turn into binary and I have learned how to output binary
vtu files. The issue is, since the cluster I'm working with has kill
times (Max 24 hr) I also need to be able to save and resume my
simulation. So, what I do right now is dump my variables in binary form
into a single file that does not have any xml tag or anything -- just
raw binary data. This is a preferred method since when I want to resume,
I do not need to parse the xml code; I just read the binary file and
that's it. Besides, I save on the grid information I'm writing to disk.
For a 23M grid point, the grid information (which basically is a
quadtree saved as a general unstructured grid with
cell connectivity information etc) file is about 1GB whereas my actual
data is only about 150MB. In most cases my grid does not change with
time, so I really only need to save it once and that's enough.
Now, the problem is, I want to be able to use or refer to these binary
files using the xml tags. Here are the questions:
1) Is this supported in .vtu format? How can I refer to a binary file
instead of dumping all the binary information in the file?
2) If this is not supported in .vtu, is it something xdmf can do? If so
how? Can you please point me to a tutorial?
3) Is there any plan or strategy that would be better than what I have
in mind right now? Please remember that retaining the raw binary format
of the files is a must for me (well I do not see convincing reasons to
give up on it, but if there is one, please let me know)
Thanks
Mohammad
Just this week, I wrote a Python program that uses the numpy.fromfile
function and ParaView's paraview.vtk.dataset_adapter.numpyTovtkDataArray
and paraview.vtk.io.vtkXMLPolyDataWriter functions to convert a large
binary file to .vtp format. It is reasonably fast. Maybe you can use
something like that as a post-processing step.
This separates the tasks of producing data and converting it to VTK format.
--
Hal Canary
_______________________________________________
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the ParaView Wiki at:
http://paraview.org/Wiki/ParaView
Follow this link to subscribe/unsubscribe:
http://www.paraview.org/mailman/listinfo/paraview
