The same issue occurs with World Ocean Database which consists of
mainly profile data. Each profile typically consists of several
variables measured along the depth. The quality flags used for all the
variable are same. I have attached a snippet below of how a CF
compliant netCDF would look. The over-head due to repeating the
quality flags is not much when the data is stored is a multi-profile
format, but when each profile is stored in an individual file, the
overhead becomes significant, since these flags have to be present in
each file.
Upendra
float temperature(z) ;
temperature:standard_name = "sea_water_temperature" ;
temperature:long_name = "sea water temperature" ;
temperature:units = "degree_C" ;
temperature:coordinates = "time lon lat z" ;
temperature:WODprofile_flag = 0 ;
temperature:_FillValue = -1.e+10f ;
temperature:grid_mapping = "crs" ;
temperature:cell_methods = "time:point longitude:point
latitude:
point z:point" ;
temperature:ancillary_variables = "temperatu
re_WODflag" ;
int temperature_WODflag(z) ;
temperature_WODflag:standard_name =
"sea_water_temperature statu
s_flag" ;
temperature_WODflag:flag_values = "0, 1, 2, 3, 4, 5,
6, 7, 8, 9"
;
temperature_WODflag:flag_meanings = "accepted_value
range_outlie
r failed_inversion_check failed_gradient_check
observed_level_bullseye_flag_and_
zero_gradient_check combined_gradient_and_inversion_checks
failed_range_and_inve
rsion_checks failed_range_and_gradient_checks
failed_range_and_questionable_data
_checks failed_range_and_combined_gradient_and_inversion_checks" ;
temperature_WODflag:reference = "" ;
float salinity(z) ;
salinity:standard_name = "sea_water_salinity" ;
salinity:long_name = "sea water salinity" ;
salinity:units = "" ;
salinity:coordinates = "time lon lat z" ;
salinity:WODprofile_flag = 0 ;
salinity:_FillValue = -1.e+10f ;
salinity:grid_mapping = "crs" ;
salinity:cell_methods = "time:point longitude:point
latitude:poi
nt z:point" ;
salinity:ancillary_variables = "salinity_WODfla
g" ;
int salinity_WODflag(z) ;
salinity_WODflag:standard_name = "sea_water_salinity
status_flag
" ;
salinity_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6,
7, 8, 9" ;
salinity_WODflag:flag_meanings = "accepted_value
range_outlier f
ailed_inversion_check failed_gradient_check
observed_level_bullseye_flag_and_zer
o_gradient_check combined_gradient_and_inversion_checks
failed_range_and_inversi
on_checks failed_range_and_gradient_checks
failed_range_and_questionable_data_ch
ecks failed_range_and_combined_gradient_and_inversion_checks" ;
salinity_WODflag:comment = "" ;
float oxygen(z) ;
oxygen:standard_name =
"volume_fraction_of_oxygen_in_sea_water"
;
oxygen:long_name = "Oxygen" ;
oxygen:units = "ml/l" ;
oxygen:coordinates = "time lon lat z" ;
oxygen:WODprofile_flag = 0 ;
oxygen:_FillValue = -1.e+10f ;
oxygen:grid_mapping = "crs" ;
oxygen:cell_methods = "time:point longitude:point
latitude:point
z:point" ;
oxygen:ancillary_variables = "oxygen_WODflag" ;
int oxygen_WODflag(z) ;
oxygen_WODflag:standard_name =
"volume_fraction_of_oxygen_in_sea
_water status_flag" ;
oxygen_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7,
8, 9" ;
oxygen_WODflag:flag_meanings = "accepted_value
range_outlier fai
led_inversion_check failed_gradient_check
observed_level_bullseye_flag_and_zero_
gradient_check combined_gradient_and_inversion_checks
failed_range_and_inversion
_checks failed_range_and_gradient_checks
failed_range_and_questionable_data_chec
ks failed_range_and_combined_gradient_and_inversion_checks" ;
oxygen_WODflag:comment = "" ;
float phosphate(z) ;
phosphate:standard_name =
"mole_concentration_of_phosphate_in_se
a_water " ;
phosphate:long_name = "Phosphate" ;
phosphate:units = "umol/l" ;
phosphate:coordinates = "time lon lat z" ;
phosphate:WODprofile_flag = 0 ;
phosphate:_FillValue = -1.e+10f ;
phosphate:grid_mapping = "crs" ;
phosphate:cell_methods = "time:point longitude:point
latitude:po
int z:point" ;
phosphate:ancillary_variables = "phosphate_WOD
flag" ;
int phosphate_WODflag(z) ;
phosphate_WODflag:standard_name =
"mole_concentration_of_phospha
te_in_sea_water status_flag" ;
phosphate_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6,
7, 8, 9" ;
phosphate_WODflag:flag_meanings = "accepted_value
range_outlier
failed_inversion_check failed_gradient_check
observed_level_bullseye_flag_and_ze
ro_gradient_check combined_gradient_and_inversion_checks
failed_range_and_invers
ion_checks failed_range_and_gradient_checks
failed_range_and_questionable_data_c
hecks failed_range_and_combined_gradient_and_inversion_checks" ;
phosphate_WODflag:comment = "" ;
float silicate(z) ;
silicate:standard_name =
"mole_concentration_of_silicate_in_sea_
water" ;
silicate:long_name = "Silicate" ;
silicate:units = "umol/l" ;
silicate:coordinates = "time lon lat z" ;
silicate:WODprofile_flag = 0 ;
silicate:_FillValue = -1.e+10f ;
silicate:grid_mapping = "crs" ;
silicate:cell_methods = "time:point longitude:point
latitude:point z:point" ;
silicate:ancillary_variables = "silicate_sigfigs
silicate_WODflag" ;
int silicate_WODflag(z) ;
silicate_WODflag:standard_name =
"mole_concentration_of_silicate_in_sea_water status_flag" ;
silicate_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6,
7, 8, 9" ;
silicate_WODflag:flag_meanings = "accepted_value
range_outlier failed_inversion_check failed_gradient_check
observed_level_bullseye_flag_and_zero_gradient_check combined_gradient_an
d_inversion_checks failed_range_and_inversion_checks
failed_range_and_gradient_checks
failed_range_and_questionable_data_checks
failed_range_and_combined_gradient_and_inversion_checks" ;
silicate_WODflag:comment = "" ;
float nitrate(z) ;
nitrate:standard_name =
"mole_concentration_of_nitrate_in_sea_water" ;
nitrate:long_name = "Nitrate" ;
nitrate:units = "umol/l" ;
nitrate:coordinates = "time lon lat z" ;
nitrate:WODprofile_flag = 0 ;
nitrate:_FillValue = -1.e+10f ;
nitrate:grid_mapping = "crs" ;
nitrate:cell_methods = "time:point longitude:point
latitude:point z:point" ;
nitrate:ancillary_variables = "nitrate_WODflag" ;
int nitrate_WODflag(z) ;
nitrate_WODflag:standard_name =
"mole_concentration_of_nitrate_in_sea_water status_flag" ;
nitrate_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7,
8, 9" ;
nitrate_WODflag:flag_meanings = "accepted_value
range_outlier failed_inversion_check failed_gradient_check
observed_level_bullseye_flag_and_zero_gradient_check
combined_gradient_and
_inversion_checks failed_range_and_inversion_checks
failed_range_and_gradient_checks
failed_range_and_questionable_data_checks
failed_range_and_combined_gradient_and_inversion_checks" ;
nitrate_WODflag:comment = "" ;
float nitrite(z) ;
nitrite:long_name = "Nitrite" ;
nitrite:units = "umol/l" ;
nitrite:coordinates = "time lon lat z" ;
nitrite:WODprofile_flag = 0 ;
nitrite:_FillValue = -1.e+10f ;
nitrite:grid_mapping = "crs" ;
nitrite:cell_methods = "time:point longitude:point
latitude:point z:point" ;
nitrite:ancillary_variables = "nitrite_WODflag" ;
int nitrite_WODflag(z) ;
nitrite_WODflag:flag_values = "0, 1, 2, 3, 4, 5, 6, 7,
8, 9" ;
nitrite_WODflag:flag_meanings = "accepted_value
range_outlier failed_inversion_check failed_gradient_check
observed_level_bullseye_flag_and_zero_gradient_check
combined_gradient_and
_inversion_checks failed_range_and_inversion_checks
failed_range_and_gradient_checks
failed_range_and_questionable_data_checks
failed_range_and_combined_gradient_and_inversion_checks" ;
nitrite_WODflag:comment = "" ;
On 10/31/2011 12:12 PM, Randy Horne wrote:
In the application I am working on, there are specific types of
products where where the same quality flags apply to multiple
variables. These products will be contained in a single NetCDF file.
The current CF conventions dictate that quality flags are attached to
specific variables. The implication is that comforming with CF
conventions would require the same quality flags to be stored
multiple times in our NetCDF product files. This is potentially
problematic due to the resulting increase in product file size. Our
variables have on the order of 10s of millions of data points where
each has a distinct quality flag.
This issue has come up before, and any design to optimize flags has
been dismissed due to the overriding objective of having variables
that are self-describing.
Product file compression is a potential answer, but this presents its
own set of performance issues.
Maybe flags could be handled in a similar manner as coordinates to
allow sharing among variables ?
..............End of Message ...............................-->
_______________________________________________
cf-satellite mailing list
[email protected]
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
cf-satellite mailing list
[email protected]
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
