FYI: I will take care of the wrong documentation. Not today and probably not tomorrow but soon :)
Andy On Tue, Mar 22, 2011 at 3:07 PM, Simon Su <[email protected]> wrote: > Hi Ken, > > This is great. I will work with the scientific programming team that came > up with the file to see if we can go anywhere with it. Thank you so much > for taking the time to look into the file. > > best > -simon > > > On Tue, Mar 22, 2011 at 2:22 PM, Moreland, Kenneth <[email protected]>wrote: > >> Simon, >> >> After taking a look at your zos_*.nc file, I believe the problem is that >> the bounds variables (specifically lon_vertices and lat_vertices) are >> incorrect and causing malformed cells. The proper format for these >> variables according to the CF convention is documented here – >> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/ch07.html#cell-boundaries >> – >> under "Bounds for 2-D coordinate variables with 4-sided cells". These >> variables are referred to as lonbnd and latbnd in the CF documentation, but >> otherwise the name is inconsequential. >> >> These bounds variables are supposed to contain a 4-tuple for each cell >> in the grid. The 4 values in each tuple specify the longitude or latitude >> coordinates in counterclockwise order starting with the "lower left" value. >> Adjacent cells should have duplicate longitude and latitude coordinates >> where appropriate. >> >> So, assuming that the bounds of the first cell (which I am extrapolating >> from other coordinate variables in the zos file) is –280 to –279 in the >> longitude direction and –82 to –81 in the latitude direction, then the first >> 4-tuple in lon_vertices should be [–280, –279, –279, –280] and the first >> tuple in lat_vertices should be [–82, –82, –81, –81]. The next entries in >> the lon_vertices and lat_vertices arrays are for the adjacent cell in the >> horizontal direction and should be [–279, –278, –278, –279] >> and [–82, –82, –81, –81], respectively. >> >> In this regard, the values stored in zon*.nc's lon_vertices and >> lat_vertices are nonsensical. ncdump shows the first two entries of each >> as: >> >> lon_vertices = >> -280, -279, -278, -277, >> -276, -275, -274, -273, >> >> lat_vertices = >> -82, -82, -82, -82, >> -82, -82, -82, -82, >> >> >> As you can see, these bounds do not form proper quadrilaterals. They >> are all degenerate polygons on the –82 latitude coordinate. My guess is >> that whatever wrote out this file mistook the format for boundary variables >> for coordinate variables. >> >> -Ken >> >> **** Kenneth Moreland >> *** Sandia National Laboratories >> *********** >> *** *** *** email: [email protected] >> ** *** ** phone: (505) 844-8919 >> *** web: http://www.cs.unm.edu/~kmorel >> >> From: Simon Su <[email protected]> >> Date: Tue, 22 Mar 2011 11:40:46 -0400 >> >> To: Kenneth Moreland <[email protected]> >> Cc: Andy Bauer <[email protected]>, "[email protected]" < >> [email protected]> >> Subject: Re: [Paraview] loading nc formatted data >> >> Hi Ken, >> >> Thank you again for your help. >> >> Is there another wiki page that also describe all the xml used syntax in >> the http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? It may help for me to >> have an overall picture of the xml syntax used? thanks. And is there a >> complete example out there that show the whole process on how one can create >> a custom data loader plugin into ParaView? Not sure if I am saying this >> correctly but something like, say I have this data file in my own format >> that has grid information and a scalar variable. How do I write a data >> loader plugin so that I can use ParaView to visualize the data. Then if I >> have to guess, first you write this xml file to describe the data loader >> plugin to ParaView and then use use sort of utility to generate the plugin >> standard code, and then implement the function in the code to load and >> create your vtk data structure (grid data type and scalar variable data >> type) and then pass it to ParaView. something like that? thanks... >> >> yes. it make sense that it is a part of VTK. ParaView is to expose VTK >> functionality after all. >> >> good thing is, I think we settle on a single large nc file that has all >> the information needed to plot the visualization, including tripolar grid >> information. Hopefully that will simplify things for me. >> >> I am sorry that the ftp site is not behaving. I have placed the files at >> http://www.cs.uh.edu/~ssu/4ParaView together with the nc file I used to >> generate the image 4.png (what looks like a double grid to me) and 5.png >> (coloring by the variable zos which I don't thin it should look like that >> ) Hopefully vtkNetCDFCFReader will do the job for me. >> >> thanks >> -simon >> >> >> >> On Mon, Mar 21, 2011 at 7:12 PM, Moreland, Kenneth <[email protected]>wrote: >> >>> It looks like the documentation for a reader plugin is slightly messed >>> up. I'll leave it to Andy (or anyone who is not me) to fix. The example in >>> the ParaView source does not list a file in SERVER_MANAGER_SOURCES because >>> it uses a source (vtkPNGReader.cxx) that is already compiled as part of VTK. >>> The Wiki is trying to capture the fact that usually you are building a >>> plugin with your own reader, and you will have to list your source code for >>> the reader there. vtkMyPNGReader.cxx is just a stand in for some reader >>> that you wrote. It probably does not really exist. I noticed, however, >>> that the Wiki documentation is also wrong in that the XML file is pointing >>> to the vtkPNGReader class when the CMake configuration is implying that it >>> should be pointing to the (imaginary) vtkMyPNGReader class. >>> >>> The vtkNetCDFCFReader.cxx file is located >>> in ParaView/VTK/IO/vtkNetCDFCFReader.cxx. Don't get too wrapped up around >>> the complexity of the NetCDF reader definition in readers.xml. It's >>> complicated because it supports defining a time series as a collection of >>> files ( >>> http://www.vtk.org/Wiki/Animating_legacy_VTK_file_series#Making_custom_readers_work_with_file_series). >>> If you collect all times in a single netCDF file (or don't have time >>> steps), you can skip all that and just define a new object much like that of >>> netCDFReaderCore in reader.xml (in the sources ProxyGroup of course). >>> >>> That said, I think this should all be unnecessary since the CF >>> convention (and ParaView reader) already contains a means of defining cells >>> that bridge across these seems. I suspect it has nothing to do with the CF >>> convention version. It is probably that your data files with openings at >>> the seems are not defining a bounds attribute, which is optional in CF. If >>> you could post some example data (or even the listing of the file header >>> from ncdump) I might be able to verify that. >>> >>> I'm not sure what is going on with the new data set (4.png). I can't >>> comment on it much because your ftp server seems to be broken for me and I >>> don't remember the image too well. At any rate, I would probably say >>> something like I couldn't tell you what was going on without seeing the nc >>> file. >>> >>> -Ken >>> >>> **** Kenneth Moreland >>> *** Sandia National Laboratories >>> *********** >>> *** *** *** email: [email protected] >>> ** *** ** phone: (505) 844-8919 >>> *** web: http://www.cs.unm.edu/~kmorel >>> >>> From: Simon Su <[email protected]> >>> Date: Mon, 21 Mar 2011 15:19:24 -0400 >>> To: Kenneth Moreland <[email protected]> >>> Cc: Andy Bauer <[email protected]>, "[email protected]" < >>> [email protected]> >>> >>> Subject: Re: [Paraview] loading nc formatted data >>> >>> Hi Ken, >>> >>> Thank you for explaining. >>> >>> The reader that is being used is vtkNetCDFCFReader. It is defined in >>>> ParaView/Servers/ServerManager/Resources/readers.xml, although it is not >>>> obvious. >>>> >>>> >>> Can you (or Andy) point me to documentation more documentation beside >>> http://www.vtk.org/Wiki/ParaView/Plugin_HowTo ? Andy also metioned >>> readers.xml file you mentioned. But after going through ParaView/Servers/ >>> ServerManager/Resources/readers.xml, I am still fuzzy about where the >>> file vtkNetCDFCFReader.cxx ( if any - I would like to know how it is >>> implemented and maybe start by modifying it in my learning process). The >>> plugin wiki page on "Adding a Reader" section mentioned CMakeList.txt file >>> >>> FIND_PACKAGE(ParaView REQUIRED) >>> INCLUDE(${PARAVIEW_USE_FILE}) >>> ADD_PARAVIEW_PLUGIN(MyReader "1.0" >>> >>> >>> >>> SERVER_MANAGER_XML MyPNGReader.xml >>> SERVER_MANAGER_SOURCES vtkMyPNGReader.cxx >>> GUI_RESOURCE_FILES MyReaderGUI.xml) >>> >>> >>> but the one on my source tree >>> ParaView-3.10.0/Examples/Plugins/Reader/CMakeList.txt looks like >>> >>> ADD_PARAVIEW_PLUGIN(MyPNGReader "1.0" >>> SERVER_MANAGER_XML readers.xml >>> GUI_RESOURCE_FILES pqReader.xml >>> ) >>> >>> note no vtkMyPNGReader.cxx file mention and the file is also not in the >>> directory. I also did a search on source tree for vtkNetCDFCFReader.cxx >>> file but can't find it. I think I am missing a big piece of something since >>> I am not getting the point that I should be getting after looking at >>> readers.xml. In fact, I can't find any of the netcdf file readers (other >>> than CF reader). It is also not listed under my Plugin Manager GUI on the >>> ParaView that I compiled but it is loading the CF convention nc file. Are >>> they (the *.cxx files) generated on the fly during compile time? >>> >>> Another thing, besides >>> http://www.paraview.org/Wiki/VisIt_Database_Bridge , is there a place >>> that I can find how to use that VisIt bridge in ParaView. Maybe an example >>> of how to load silo file? I actually have a plugin of my own in VisIt that >>> can load the climate modeling data that I have. I would like to see if I can >>> use my VisIt plugin in ParaView and not write another plugin for ParaView? >>> :) >>> >>> >>> >>>> As the name implies, this reader reads netCDF files using the CF >>>> convention. (As the default netCDF reader, it also gracefully handles >>>> files >>>> that do not follow this convention.) With this assumption, I will try to >>>> explain what it does. It reads arrays as regular 1, 2, or 3D arrays, >>>> possibly with time. The CF convention also provides a means to assign >>>> coordinates to each grid point and to identify the coordinates as longitude >>>> or latitude. >>>> >>>> >>> The published nc files we have do adhere to CF convention we also use >>> CMOR in one of the pipeline. I noticed Data_tos_O1_2001-2002.nc is at >>> CF-1.0 and since I am new at CF Convention, I can only speculate that what I >>> see at ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png with what looks >>> like double grid is due to difference between CF-1.0 and CF-1.4? Will >>> vtkNetCDFCFReader handle CF-1.4 data? >>> >>> >>> >>> >>>> Even though coordinates are defined as longitude and latitude, the >>>> topology of the grid itself is still a grid. Thus, the grid gets wrapped >>>> around, but still has these seems that you see because topologically the >>>> one >>>> end of the regular grid is not attached to the other. I can't think of any >>>> filter that will identify and close these seems. In fact, it's not >>>> straightforward to do at all. If you look at your topology, it is not lain >>>> out on a simple spherical grid. >>>> >>>> The "right" way to solve your problem, which may or may not be in your >>>> control, is to create netCDF files that specify cell boundaries for a >>>> closed >>>> topology. Your netCDF file must be following at least some parts of the CF >>>> convention; your data would not show up as a sphere if it were not. The CF >>>> convention provides a way of defining cells that are not constrained by a >>>> regular grid topology. It is done through a "bounds" attribute on the >>>> dimension descriptor variables. You need either 1D or 2D bounds. They are >>>> described in this section of the CF convention documentation: >>>> http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/cf-conventions.html#cell-boundaries >>>> . >>>> >>>> >>> Agreed. The hack we have in VisIt loader (replace the last grid value >>> with the 1st grid value to create a "closed topology" ) is not >>> "generalizable" and will blow up in my face if I hand it to the user. >>> >>> >>> thanks >>> -simon >>> >>> >>> >>> >>> >>>> The VTK test data has examples of both 1D bounds >>>> (tos_O1_2001-2002.nc) and 2D bounds (sampleCurveGrid4.nc). You can get the >>>> VTK test data from git (http://vtk.org/VTKData.git), or download them >>>> directly from the gitourious server: >>>> >>>> >>>> http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/tos_O1_2001-2002.nc;h=30aa4a9b5e08b9bdf64540f2b144d83b279cca6c;hb=HEAD >>>> >>>> >>>> http://vtk.org/gitweb?p=VTKData.git;a=blob;f=Data/sampleCurveGrid4.nc;h=0ab89c27a25f92c047b58dca8b3057ca8d4df017;hb=HEAD >>>> >>>> >>>> -Ken >>>> >>>> **** Kenneth Moreland >>>> *** Sandia National Laboratories >>>> *********** >>>> *** *** *** email: *[email protected] >>>> *** *** ** phone: (505) 844-8919 >>>> *** web: *http://www.cs.unm.edu/~kmorel* >>>> >>>> From: Simon Su <[email protected]> >>>> Date: Fri, 18 Mar 2011 16:43:42 -0400 >>>> To: Andy Bauer <[email protected]> >>>> Cc: <[email protected]> >>>> Subject: Re: [Paraview] loading nc formatted data >>>> >>>> Hi Andy, >>>> >>>> python trace gave me >>>> >>>> zos_Omon_GFDLESM2M_historical_r1i1p1_186101188012_nc = NetCDFReader( >>>> FileName=['/work/sms/data/cmor-20110128/mon/ocean/zos/r1i1p1/zos_Omon_GFDL-ESM2M_historical_r1i1p1_186101-188012.nc'] >>>> ) >>>> >>>> and there are tons of netcdf reader in ParaView and it is not in the >>>> plugin directory >>>> >>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> pwd >>>> /local/home/build/paraview/ParaView-3.10.0/Plugins >>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> ll >>>> total 84 >>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 AdiosReader/ >>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 AnalyzeNIfTIReaderWriter/ >>>> -rw-r--r-- 1 sms t 3261 Mar 9 13:31 CMakeLists.txt >>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 CoProcessingScriptGenerator/ >>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 CosmoFilters/ >>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 EyeDomeLighting/ >>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 ForceTime/ >>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 H5PartReader/ >>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 Manta/ >>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Moments/ >>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 NetDMFReader/ >>>> drwxr-xr-x 6 sms t 4096 Mar 10 12:37 PointSprite/ >>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 PrismPlugins/ >>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 pvblot/ >>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SierraPlotTools/ >>>> drwxr-xr-x 3 sms t 4096 Mar 10 12:37 SLACTools/ >>>> drwxr-xr-x 4 sms t 4096 Mar 10 12:37 StreamingView/ >>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 SurfaceLIC/ >>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 Vapor/ >>>> drwxr-xr-x 2 sms t 4096 Mar 10 12:37 VRPN/ >>>> drwxr-xr-x 2 sms t 4096 Mar 9 13:31 VRUI/ >>>> sms:/local/home/build/paraview/ParaView-3.10.0/Plugins> >>>> >>>> >>>> Can you help describe how Netcdf files plugins are done in ParaView. Do >>>> they have a super class of Netcdf that they all derive from to write the >>>> different flavors of netcdf readers? If so, where can the code be found? >>>> >>>> thanks >>>> -simon >>>> >>>> >>>> >>>> On Fri, Mar 18, 2011 at 10:30 AM, Andy Bauer <[email protected]>wrote: >>>> >>>>> Hi Simon, >>>>> >>>>> Replies below... >>>>> >>>>> On Thu, Mar 17, 2011 at 12:41 PM, Simon Su >>>>> <[email protected]>wrote: >>>>> >>>>>> Hi All, >>>>>> >>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/1.png >>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/2.png >>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/3.png >>>>>> >>>>>> the above are screen shot from older version of *.nc files that I have >>>>>> that I loaded up with Paraview 3.10.0 64-bit which I compiled myself. As >>>>>> you >>>>>> can see, the grid is correctly loaded. But in the data, there is a >>>>>> crack. Is >>>>>> there a filter that can fix this in ParaView? :) >>>>>> >>>>> >>>>> I'm not aware of any filter that will fix this automatically. Can you >>>>> describe the grid a bit more? It kind of looks like a multiblock of >>>>> structured grids. >>>>> >>>>>> >>>>>> ftp://ftp.gfdl.noaa.gov/pub/sms/4ParaView/4.png is the latest nc file >>>>>> that I have of similar simulation preprocessed output. When I loaded it >>>>>> up, >>>>>> it is clearly doing making assumption on the grid that is not correct and >>>>>> hence, the double looking grid. >>>>>> >>>>>> The questions now are: >>>>>> >>>>>> 1. ParaView has lots of *.nc file loader. How do I know which loader >>>>>> is ParaView using to load the data? If I pick a type for the Files of >>>>>> type >>>>>> option in the open File window, will that gurantees that ParaView will be >>>>>> using that particular file loader? >>>>>> >>>>> >>>>> If there is an ambiguity for which file loader to use (i.e. multiple >>>>> readers assume the same extension), then the GUI should pop up a dialog >>>>> for >>>>> you to specify which one to use. You can use the python trace to figure >>>>> out >>>>> exactly what reader is being used to load the file. >>>>> >>>>> >>>>>> >>>>>> 2. If I know which loader is used, I would like to see if I can modify >>>>>> the existing loader to create a new type of *.nc file loader to fix the >>>>>> grid >>>>>> of my data. Where is the code in the source tree of ParaView is the >>>>>> loader >>>>>> plugin placed? >>>>>> >>>>> Based on the name of the name of the reader from the python script, >>>>> you can look up the actual class name in the >>>>> ParaView/Servers/ServerManager/Resources/readers.xml file. >>>>> >>>>> >>>>>> >>>>>> or if there is an easier solution to fix my grid problem that doesn't >>>>>> involve developing a new data loader plugin that would be better.... :) >>>>>> please let me know... >>>>>> >>>>>> thanks >>>>>> -simon >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Powered by www.kitware.com >>>>>> >>>>>> Visit other Kitware open-source projects at >>>>>> http://www.kitware.com/opensource/opensource.html >>>>>> >>>>>> Please keep messages on-topic and check the ParaView Wiki at: >>>>>> http://paraview.org/Wiki/ParaView >>>>>> >>>>>> Follow this link to subscribe/unsubscribe: >>>>>> http://www.paraview.org/mailman/listinfo/paraview >>>>>> >>>>>> >>>>> >>>> _______________________________________________ Powered by >>>> www.kitware.com Visit other Kitware open-source projects at >>>> http://www.kitware.com/opensource/opensource.html Please keep messages >>>> on-topic and check the ParaView Wiki at: >>>> http://paraview.org/Wiki/ParaView Follow this link to >>>> subscribe/unsubscribe: >>>> http://www.paraview.org/mailman/listinfo/paraview >>>> >>> >>> >> >
_______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView Follow this link to subscribe/unsubscribe: http://www.paraview.org/mailman/listinfo/paraview
