Let me try to get back to your responses to my earlier post. 1. There's no magic here. Essentially netCDF is only an API for self-describing multi-dimensional (scalar) arrays. (I know something about this idea because it's derived from work I did over 15 years ago.) As a data model and an API this is both good and bad. The good is its simplicity. (The concept is easy to use and flexible. How are the arrays are used is an exercise left to the programmer.) The bad is its simplicity. (No inherit support for more complex data. How are the arrays are used is an exercise left to the programmer.) But the descriptions as global and local attributes are sufficiently flexible to define semantics via conventions for to interpret the arrays in an application. The netCDF web site discusses several conventions, including those that the DX importer currently understands as well as others (e.g., COARDS), which are more aligned with the way people typically use netCDF. 2. Well, it certainly would be more convenient for a user to have (general) Import do this. If you look at the code for CDF import within Import, you can see that the concept has already been implemented to some extent, but not for netCDF. For data in netCDF that I use regularly, there are some very specialized conventions. So, it was easier and more effiicent for me to write custom code used in filters and macros, for example, to import the data in the way that I needed. I also wrote some code for importing CDF attributes without the data a long time ago (i.e. also to support user selection prior to import). But again, it's the same idea. 3. I saw your earlier post on this. 4. That's good. I tend to use ncdf for historical reasons or with some codes that use their own naming conventions (i.e., with date/time tags). So, I never checked this. 5. That bug was reported in 1997 with version 3.1.4B (commercial DX). I don't think it was ever fixed. It probably dates from much earlier than that.
I hope this helps. Nils Smeds <[EMAIL PROTECTED]>@opendx.watson.ibm.com on 07/18/2001 07:58:20 PM Please respond to [email protected] Sent by: [EMAIL PROTECTED] To: [email protected] cc: Subject: Re: [opendx-dev] netCDF import of time series data What would be nice is to write it so that the module gets independent of fixed sizes (other than DX limitations). This would require a more thorough rewrite to make sure that memory is freed and allocated consistently and pointers kept sane. What I did was a minimal change to the module to have it work correctly for a set of netCDF files that the documentation says the module can handle. It would be great if the last version (#3) of the patch I wrote made it into the "official" OpenDX as it is really only a bug fix. I'd appreciate if anyone who is a OpenDX maintainer commented on my suggestions and pointed out any problems they have with it so that I can make necessary modifications. [EMAIL PROTECTED] said: > DX supports an arbitrary number of almost arbitrarily named > attributes per object (like a series group, or each field, etc.). > ("almost" as there are a few keyword attributes). So you might > consider creating a metadata set instead of truncating. The first > attribute would declare the number of attributes to follow, then as > many as you need to transfer the needed info from netCDF to DX. You > can retrieve these within DX with the Attribute module for display. > Your decision on how to cut the long string up. My intention at this point was to make the minimal amount of changes needed to get the module working more correct. The only thing that gets truncated at this point (I think, I don't yet understand all the code in the module) is name of attributes and single word attribute values. These are now truncated to 128 characters, which is the same as the netCDF limit of NC_MAX_NAME. Another limitation is MAXATTRSTR(=16) in import_ncdf.c, which I have not yet quite understood what implications it has. With my patch I have successfully read in a netCDF file with three series each with 200 series positions and each field consisting of a scalar float field at 2145 points. And that is on a 300 MHz PC with 128 MB of RAM. That makes me _very_ happy. The limitation with MAXATTRSTR is that if there are more than MAXATTRSTR tokens between two ";" in a "multiple word" attribute value they will not all be parsed, but the latter one gets discarded. I don't know if that is a problem. The "multiple word" attributes are pre-defined in the User's Reference and are the global attributes that connects variable names to a series. I still now too little about DX in general and DX internals in particular to fully understand how to create metadata and what consequences it might have on backward compatibility of the netCDF import module. What I mainly want is a to get the module to work as described in the documentation ;-) while not breaking any exisiting user's networks. [EMAIL PROTECTED] said: > These conventions are essentially semantics for higher-order > constructs beyond multi-dimensional arrays, which are handled natively > within DX. Thus, it's a clear match. Essentially, there is a > mechanism to describe coordinates/mesh as a product specification > (e.g., lat, lon, pressure), fill and missing values, array ordering, > time, units, etc. Currently, Import can read such a netCDF, but > cannot associate the positions information with the data, and the > attributes are not Import'd at all. I've only read the HTML documentation and not tested it, but it seems to me that the import module does honour product specifications. A problem I can see is if a "Convention" does not name the position attribute as positions. I guess it could be possible to write some code that does the right thing according to (at least some of) the conventions in use. [EMAIL PROTECTED] said: > 2. Along the lines of 1., being able to import netCDF attributes > without the data would be useful to create metadata-driven > applications with the overhead of reading data. This, I guess, is more in the line of a dedicated netCDF import module? Or is it possible to include this in the general import module without making it to behave "unpredictably" for inexperienced users as me for netCDF files as compared to other formats? Maybe a netCDFAttributes module? [EMAIL PROTECTED] said: > 3. netCDF import was designed around a nearly-decade-old version. > There has been some enhancement since from Unidata. Yes I saw that the calls to the netCDF library uses old conventions. It is probably a good thing to update these so that the module does not get obsoleted it the old calls are withdrawn. [EMAIL PROTECTED] said: > 4. netCDFs often go by the extension of .nc or occasionally .ncdf, > but the former is the preferred one. Import doesn't know about that > extension. The current OpenDX version knows that .nc is a netCDF file. If that is what you said it didn't in the past. It still also recognizes .ncdf as well it seems. [EMAIL PROTECTED] said: > 5. There was a bug reported in netCDF import that it duplicates > positions, connections and neighbors in groups. I don't know if it > was ever fixed, but it sounds like the problem that was reported > earlier this week. I didn't see that post. Could you send that one to me off-list? I have just subscribed so it might have arrived before me joining the list. /Nils
