Dear Martin and Jim I prefer the proposal to use flag_values and flag_meanings to indicate special values of a coordinate axis. In Martin's use-case, the coordinate variable contains physical values of height for all the elements of the axis apart from the special "retrieval error" bin. To me it feels less satisfactory to convert this coordinate variable to an auxiliary coordinate variable with a different standard name just because of this bin. Instead, I feel we should keep the standard name unchanged, with the special bin in the right place in monotonic order. That means generic software which doesn't recognise the special coordinate value will treat it as a physical value. This is a hazard because it could lead to peculiar plots, or to meaningless results if, for example, the coordinate axis is differentiated. I think we can tolerate this hazard. What do you think?
The flag_values mechanism could be used to indicate to up-to-date CF-aware software that the value is special (not missing), as Jim suggested. As I said before, this would need a change to the convention, but only a small one, just to permit flag_values and flag_meanings for coordinate variables as well as data variables. This requires a change to Appendix A, and it would be sensible to have a few words and a new example in Section 3.5. I suspect that if we add this possibility there would be other applications for it. For example, it could be used to indicate that the bottom or top element of a coordinate axis has bounds which are open on one side. Best wishes Jonathan ----- Forwarded message from Jim Biard <jbi...@cicsnc.org> ----- > Date: Fri, 10 May 2019 10:58:48 -0400 > From: Jim Biard <jbi...@cicsnc.org> > To: "cf-metadata@cgd.ucar.edu" <cf-metadata@cgd.ucar.edu> > Subject: Re: [CF-metadata] Missing data bins in histograms > User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) > Gecko/20100101 Thunderbird/60.6.1 > > Hi. > > Sorry I have been so quiet lately. I've been caught up in other activities. > > I have a strong aversion to the proposal to overload the > missing_value attribute with a wholly different meaning. Using > missing_value in this way will produce unexpected results in a > number of existing software packages. If the minor modification to > CF to designate flag attributes to be used on coordinate variables > doesn't seem like an acceptable solution for one reason or another, > I think we should define a new convention that doesn't add > contradictory interpretations of existing attributes. > > Grace and peace, > > Jim > > On 5/2/19 11:49 AM, Martin Juckes - UKRI STFC wrote: > >Dear Jonathan, Jim, > > > > > > > >I’m sorry to have dropped this conversation after starting it three years > >ago. We ended up not fixing the problem for CMIP6, but I think it is worth > >taking another look. > > > > > > > >Coming back to it again, I think that a variation on Jim’s suggestion could > >work: rather than using flags it should be possible to use a coordinate > >variable, as is done for some CMIP variables that have region names along > >one axis. The NetCDF dimension would be an index, and the array of values > >defining the bins would be an auxiliary coordinate which, I believe, is not > >subject to the rules on monotonicity and missing values which apply to > >NetCDF dimensions. There may be a need for some clarifications, but I think > >this approach would be much closer to the current convention that any change > >in the specification for non-auxiliary coordinate variables. > > > > > >We have a specific use case in CMIP6 for which the bins are height bins > >(height of detected cloud), with one bin reserved for "retrieval error". > > > > > >This might not need a change in the convention rules, but it would help, I > >think, to at least add an example and a standard name for the coordinate > >variable. For example: > > > > > >float data(time,lat,lon,zindex); > > > > data: standard_name = > > "histogram_of_equivalent_reflectivity_factor_over_height_above_reference_ellipsoid"; > > > > data: coordinates="zbins"; > > > >float zbins(zindex); > > > > zbins: long_name="Height ranges (with bin for missing data at first > > element)"; > > > > zbins:missing_value= -9999.; > > > > zbins: units="m"; > > > > zbins: bounds="zbin_bnds"; > > > > zbins: standard_name = "????"; > > > >float zbin_bnds(zindex,2); > > > > zbin_bnds:missing_value= -9999.; > > > >float lat(lat); > > > >float lon(lon); > > > > > >data: > > > > zbins = -9999., 25., 100., ....; > > > > zbin_bnds = -9999.,-9999., 0., 50., 50., 150., ... > > > > > >The use of missing_value in the bounds variable appears to conflict with > >conformance rules, but I'm not sure if this is really banned by the > >convention in this context. > > > > > >Using missing_value in this way appears to be acceptable to the convention, > >but I think it conflicts with the spirit of the convention: it is not > >indicating that a value of "zbins" is missing, but indicating that this > >index of the array relates to a count of missing values. For this reason I > >have omitted _FillValue. > > > > > >The "zbins" auxiliary coordinate here is a height-like variable, but I don't > >think we can use a standard name "height": is it worth adding a standard > >name "height_bins" defined to be "Height ranges, as used, for example in a > >histogram or frequency distribution. A variable with this standard name may > >include a special bin for the count or frequency of missing data. This > >should be indicated by setting the value of that bin and its bounds to equal > >the missing_value of the variable. If there is no missing value bin, it is > >recommended that the term 'height' be used instead." > > > > > >regards, > > > >Martin > > > > > >CF-metadata] Missing data bins in histograms > > > >Jonathan Gregoryj.m.gregory at reading.ac.uk > ><mailto:cf-metadata%40cgd.ucar.edu?Subject=Re%3A%20%5BCF-metadata%5D%20Missing%20data%20bins%20in%20histograms&In-Reply-To=%3C20161013094247.GF6219%40met.reading.ac.uk%3E> > >Thu Oct 13 03:42:47 MDT 2016 > > > > * Previous message (by thread): [CF-metadata] Missing data bins in > > histograms > > <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html> > > * Next message (by thread): [CF-metadata] Usage of > > histogram_of_X_over_Z > > <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html> > > * Messages sorted by: [ date > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984> [ > > thread > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984> > > [ subject > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984> > > [ author > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984> > > > >________________________________ > > > >Dear Jim > > > > > > > >In Appendix A it does not say that the flag attributes are allowed for > > > >coordinate variables - it has just "D" in the "Use" column. This is not an > > > >argument why they shouldn't be if there is a need, but they weren't > >introduced > > > >with that in mind. The use which you suggested for Martin's case is a good > > > >idea, but I think it would need a change to the convention. > > > > > > > >Best wishes > > > > > > > >Jonathan > > > > > > > >----- Forwarded message from Jim Biard <jbiard at > >cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> ----- > > > > > > > >>Date: Wed, 12 Oct 2016 14:58:11 -0400 > >>From: Jim Biard <jbiard at > >>cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > >>To: cf-metadata at > >>cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > >>Subject: Re: [CF-metadata] Missing data bins in histograms > >>User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) > >> Gecko/20100101 Thunderbird/45.4.0 > >>Jonathan, > >>Missing/fill values are not allowed, but I don't see any language > >>prohibiting flags. I'd appreciate it if you could expand on your > >>thoughts about why they aren't allowed. > >>Grace and peace, > >>Jim > >>On 10/12/16 1:30 PM, Jonathan Gregory wrote: > >>>Dear Jim > >>>That is an ingenious idea. I don't think the flag atts are currently > >>>allowed > >>>for coord variables, but they could be, I agree. > >>>Best wishes > >>>Jonathan > >>>----- Forwarded message from Jim Biard <jbiard at > >>>cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> ----- > >>>>Date: Tue, 11 Oct 2016 14:39:56 -0400 > >>>>From: Jim Biard <jbiard at > >>>>cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > >>>>To: cf-metadata at > >>>>cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > >>>>Subject: Re: [CF-metadata] Missing data bins in histograms > >>>>User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) > >>>> Gecko/20100101 Thunderbird/45.4.0 > >>>>Hi. > >>>>Another approach could be to use flag_values and flag_meanings on > >>>>the coordinate variable to indicate one or more special coordinate > >>>>values that correspond to any number of "missing data" or "out of > >>>>bounds" bins. These attributes aren't forbidden by CF, and > >>>>everything should be fine as long as the coordinate variable remains > >>>>monotonic. > >>>>Grace and peace, > >>>>Jim > >>>>On 10/11/16 8:41 AM, martin.juckes at > >>>>stfc.ac.uk<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > >>>>wrote: > >>>>>Hello, > >>>>>the CF standard name list has two "histogram_.... " entries, and in the > >>>>>CMIP6 data request we may need to add a third, a > >>>>>histogram_of_cloud_top_height. Besides the standard name, we also need, > >>>>>for this new variable, a method of encoding the "missing data" bin in > >>>>>the histogram. That is, the histogram should record frequency in 16 data > >>>>>bins and one additional bin for the frequency of missing data. > >>>>>Can we define a "missing_data_index" attribute for histogram variables, > >>>>>and use this to indicate that the first bin in the array has this > >>>>>special purpose. It might be more pythonic to put the _FillValue in the > >>>>>coordinate value for the missing data bin, but I suspect that this would > >>>>>cause substantial problems for many software packages. > >>>>>regards, > >>>>>Martin > >>>>>_______________________________________________ > >>>>>CF-metadata mailing list > >>>>>CF-metadata at > >>>>>cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > >>>>>http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > >>>>-- > >>>>CICS-NC <http://www.cicsnc.org/> Visit us on > >>>>Facebook <http://www.facebook.com/cicsnc> *Jim Biard* > >>>>*Research Scholar* > >>>>Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/> > >>>>North Carolina State University <http://ncsu.edu/> > >>>>NOAA National Centers for Environmental Information > >>>><http://ncdc.noaa.gov/> > >>>>/formerly NOAA’s National Climatic Data Center/ > >>>>151 Patton Ave, Asheville, NC 28801 > >>>>e: jbiard at > >>>>cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > >>>><mailto:jbiard at > >>>>cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > >>>>o: +1 828 271 4900 > >>>>/Connect with us on Facebook for climate > >>>><https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics > >>>><https://www.facebook.com/NOAANCEIoceangeo> information, and follow > >>>>us on Twitter at @NOAANCEIclimate > >>>><https://twitter.com/NOAANCEIclimate> and @NOAANCEIocngeo > >>>><https://twitter.com/NOAANCEIocngeo>. / > >>>>_______________________________________________ > >>>>CF-metadata mailing list > >>>>CF-metadata at > >>>>cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > >>>>http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > >>>----- End forwarded message ----- > >>>_______________________________________________ > >>>CF-metadata mailing list > >>>CF-metadata at > >>>cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > >>>http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > >>-- > >>CICS-NC <http://www.cicsnc.org/> Visit us on > >>Facebook <http://www.facebook.com/cicsnc> *Jim Biard* > >>*Research Scholar* > >>Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/> > >>North Carolina State University <http://ncsu.edu/> > >>NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/> > >>/formerly NOAA’s National Climatic Data Center/ > >>151 Patton Ave, Asheville, NC 28801 > >>e: jbiard at > >>cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > >><mailto:jbiard at > >>cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > >>o: +1 828 271 4900 > >>/Connect with us on Facebook for climate > >><https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics > >><https://www.facebook.com/NOAANCEIoceangeo> information, and follow > >>us on Twitter at @NOAANCEIclimate > >><https://twitter.com/NOAANCEIclimate> and @NOAANCEIocngeo > >><https://twitter.com/NOAANCEIocngeo>. / > > > > > >>_______________________________________________ > >>CF-metadata mailing list > >>CF-metadata at > >>cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > >>http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > > > > > > > > > >----- End forwarded message ----- > > > > > > > >________________________________ > > > > * Previous message (by thread): [CF-metadata] Missing data bins in > > histograms > > <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html> > > * Next message (by thread): [CF-metadata] Usage of > > histogram_of_X_over_Z > > <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html> > > * Messages sorted by: [ date > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984> [ > > thread > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984> > > [ subject > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984> > > [ author > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984> > > > >________________________________ > > > >More information about the CF-metadata mailing > >list<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > -- > CICS-NC <http://www.cicsnc.org/> Visit us on > Facebook <http://www.facebook.com/cicsnc> *Jim Biard* > *Research Scholar* > Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/> > North Carolina State University <http://ncsu.edu/> > NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/> > /formerly NOAA’s National Climatic Data Center/ > 151 Patton Ave, Asheville, NC 28801 > e: jbi...@cicsnc.org <mailto:jbi...@cicsnc.org> > o: +1 828 271 4900 > > /Connect with us on Facebook for climate > <https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics > <https://www.facebook.com/NOAANCEIoceangeo> information, and follow > us on Twitter at @NOAANCEIclimate > <https://twitter.com/NOAANCEIclimate> and @NOAANCEIocngeo > <https://twitter.com/NOAANCEIocngeo>. / > > > _______________________________________________ > CF-metadata mailing list > CF-metadata@cgd.ucar.edu > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ----- End forwarded message ----- _______________________________________________ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
