What would be nice is to write it so that the module gets independent of
fixed sizes (other than DX limitations). This would require a more thorough
rewrite to make sure that memory is freed and allocated consistently and
pointers kept sane. What I did was a minimal change to the module to have
it work correctly for a set of netCDF files that the documentation says the
module can handle.
It would be great if the last version (#3) of the patch I wrote made it into
the "official" OpenDX as it is really only a bug fix. I'd appreciate if anyone
who is a OpenDX maintainer commented on my suggestions and pointed out any
problems they have with it so that I can make necessary modifications.
[EMAIL PROTECTED] said:
> DX supports an arbitrary number of almost arbitrarily named
> attributes per object (like a series group, or each field, etc.).
> ("almost" as there are a few keyword attributes). So you might
> consider creating a metadata set instead of truncating. The first
> attribute would declare the number of attributes to follow, then as
> many as you need to transfer the needed info from netCDF to DX. You
> can retrieve these within DX with the Attribute module for display.
> Your decision on how to cut the long string up.
My intention at this point was to make the minimal amount of changes
needed to get the module working more correct. The only thing that gets
truncated at this point (I think, I don't yet understand all the code in
the module) is name of attributes and single word attribute values.
These are now truncated to 128 characters, which is the same as the
netCDF limit of NC_MAX_NAME.
Another limitation is MAXATTRSTR(=16) in import_ncdf.c, which I have not
yet quite understood what implications it has. With my patch I have
successfully read in a netCDF file with three series each with 200
series positions and each field consisting of a scalar float field at
2145 points. And that is on a 300 MHz PC with 128 MB of RAM. That makes
me _very_ happy. The limitation with MAXATTRSTR is that if there are
more than MAXATTRSTR tokens between two ";" in a "multiple word" attribute
value they will not all be parsed, but the latter one gets discarded. I
don't know if that is a problem. The "multiple word" attributes are
pre-defined in the User's Reference and are the global attributes that
connects variable names to a series.
I still now too little about DX in general and DX internals in particular
to fully understand how to create metadata and what consequences it might
have on backward compatibility of the netCDF import module. What I mainly
want is a to get the module to work as described in the documentation ;-)
while not breaking any exisiting user's networks.
[EMAIL PROTECTED] said:
> These conventions are essentially semantics for higher-order
> constructs beyond multi-dimensional arrays, which are handled natively
> within DX. Thus, it's a clear match. Essentially, there is a
> mechanism to describe coordinates/mesh as a product specification
> (e.g., lat, lon, pressure), fill and missing values, array ordering,
> time, units, etc. Currently, Import can read such a netCDF, but
> cannot associate the positions information with the data, and the
> attributes are not Import'd at all.
I've only read the HTML documentation and not tested it, but it seems to me
that the import module does honour product specifications. A problem I can
see is if a "Convention" does not name the position attribute as positions.
I guess it could be possible to write some code that does the right thing
according to (at least some of) the conventions in use.
[EMAIL PROTECTED] said:
> 2. Along the lines of 1., being able to import netCDF attributes
> without the data would be useful to create metadata-driven
> applications with the overhead of reading data.
This, I guess, is more in the line of a dedicated netCDF import module? Or is
it possible to include this in the general import module without making it
to behave "unpredictably" for inexperienced users as me for netCDF files as
compared to other formats? Maybe a netCDFAttributes module?
[EMAIL PROTECTED] said:
> 3. netCDF import was designed around a nearly-decade-old version.
> There has been some enhancement since from Unidata.
Yes I saw that the calls to the netCDF library uses old conventions. It is
probably a good thing to update these so that the module does not get
obsoleted it the old calls are withdrawn.
[EMAIL PROTECTED] said:
> 4. netCDFs often go by the extension of .nc or occasionally .ncdf,
> but the former is the preferred one. Import doesn't know about that
> extension.
The current OpenDX version knows that .nc is a netCDF file. If that is what
you said it didn't in the past. It still also recognizes .ncdf as well it
seems.
[EMAIL PROTECTED] said:
> 5. There was a bug reported in netCDF import that it duplicates
> positions, connections and neighbors in groups. I don't know if it
> was ever fixed, but it sounds like the problem that was reported
> earlier this week.
I didn't see that post. Could you send that one to me off-list? I have just
subscribed so it might have arrived before me joining the list.
/Nils
