On 26/03/18 16:57, V. Balaji wrote: > Groups can have quite significant performance issues when the datasets are > large... > something to keep in mind while designing standards or conventions that > _require_ the > use of netCDF4 groups. It's a last resort, in my opinion... to be used only if > there's simply no other way to express what you want.
That is a bug in the netCDF4 library. When you use the HDF-5 library directly (and still set the appropriate attributes), then groups have no performance impact at all. On the other hand you'll still have a significant performance degradation if the number of variables is large. As far as I understand the issue is that a linear search is performed in the netCDF library, where an appropriate algorithm (likely a hash) would have been more appropriate. Kind regards, Maarten > Erik Quaeghebeur writes: > >> Dear list, >> >> >> I have not come across any mention of netCDF4 groups (and other netCDF4 >> features) >> in the CF conventions. I was wondering if there are nevertheless standard >> ways to >> use groups. >> >> For example, I have repackaged some statistics data (csv files) from met >> masts in >> netCDF4. I've been trying to apply the CF conventions, but bump into issues, >> e.g., >> of duplication. >> >> Currently, the structure I use is: >> >> root level: >> * time dimension/variable >> * per-instrument-type groups >> >> instrument level: >> * height (of instrument) dimension/variable >> * instrument metadata attributes (not covered by CF conventions) >> * signal groups (some instruments measure more than one signal) signal level: >> * signal specific metadata, e.g., units (covered by CF conventions, but for >> variables) >> * signal statistics variables >> >> statistics variables: >> * data >> * some statistic-specific attributes like cell_methods >> * should I duplicate the signal metadata such as unit here? >> >> The group structure is helpful, because its structure improves >> self-descriptiveness, I feel. I guess the structure is also a form of >> metadata. But >> perhaps it is considered out-of-scope for the CF conventions? >> >> >> Best, >> >> Erik >> >> > Maarten Sneep -- KNMI T: 030 2206747 E: [email protected] R: A2.14 _______________________________________________ CF-metadata mailing list [email protected] http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
