Dear Jim and Martin I think the minimum is to change App A, but it would be helpful to add some words and a new example to Section 3.5, because the flag_* attributes are introduced and discussed only for data variables there.
Cheers Jonathan ----- Forwarded message from Jim Biard <jbi...@cicsnc.org> ----- > Date: Mon, 13 May 2019 10:46:30 -0400 > From: Jim Biard <jbi...@cicsnc.org> > To: "cf-metadata@cgd.ucar.edu" <cf-metadata@cgd.ucar.edu> > Subject: Re: [CF-metadata] Missing data bins in histograms > User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) > Gecko/20100101 Thunderbird/60.6.1 > > Martin, > > I believe that the convention change is as simple as changing the > table cells where it declares the flag_* attributes to be for data > variables only to declare that the flag_* attributes can be used on > coordinate variables as well. > > Grace and peace, > > Jim > > On 5/13/19 5:39 AM, Martin Juckes - UKRI STFC wrote: > >Dear Jim, Jonathan, > > > > > >OK, I accept your point that this would be a new meaning of "missing_value", > >and that probably causes more serious problems than the one we are trying to > >solve. > > > > > >I believe that Jonathan is also correct in saying that we can't use > >flag_values for this purpose without a change in the convention ... but > >perhaps we should explore whether such a change would lead to a good > >solution. Do you have a specific proposal in mind? > > > > > >If not, here is an idea adapted from my last post. In this example I have > >used a real valued dimension coordinate to define the data bins, with an > >arbitrary value in the first bin. The new feature is an auxiliary coordinate > >which is constructed to label the first bin as "missing_data" and the > >remaining bins as "data". > > > > > >I believe that it would be mis-leading to use the existing standard name > >"height" for the variable "zbins" in this example, because the first bin is > >not a height range. Hence, I suggest introducing a new standard name > >"height_bins" which would allow one or more bins to have special meanings > >which should be indicated by a string valued auxiliary coordinate. > > > > > >"zbin_flags" could also be encoded as a character array, with values > >["missing_values", "data", "data", ......]. This could be done without > >changing the convention text, since character arrays are allowed auxiliary > >coordinate. However, I believe that it would be worth making an adjustment > >here, since the use of flags seems more natural. > > > > > >I've given the "zbin_flags" a new standard name "bin_status_flag", which is > >related to the existing term "status_flag", but refers to the status of the > >data being counted in the histogram rather than to the status of the > >histogram itself. This allows us to refer unambiguously to the fact that we > >are counting missing data in the first bin. > > > > > >Would this work? > > > > > >float data(time,lat,lon,zbins); > > > > data: standard_name = > > "histogram_of_equivalent_reflectivity_factor_over_height_above_reference_ellipsoid"; > > > > data: coordinates="zbin_flags status"; > > > >float zbins(zbina); > > > > zbins: long_name="Height ranges (with bin for missing data at first > > element)" > > > > zbins: units="m"; > > > > zbins: bounds="zbin_bnds"; > > > > zbins: standard_name = "height_bins"; > > > >float zbin_bnds(zindex,2); > > > >integer zbin_flags(zbins); > > > > zbin_flags: long_name = "Flags indicating the status of data which is > > counted in each bin"; > > > > zbin_flags:standard_name = "bin_status_flag"; > > > > zbin_flags:flag_values = 0,1; > > > > zbin_flags:flag_meanings = "missing_values data"; > > > >character status(char_len); > > > > status:standard_name = "status_flag"; > > > > status:long_name = "Flag indicating quality of histogram"; > > > >float lat(lat); > > > >float lon(lon); > > > > > >data: > > > > zbins = -9999., 25., 100., ....; > > > > zbin_bnds = -9999.,0., 0., 50., 50., 150., ... > > > > zbin_flags = 0,1,1,1,... > > > > > >Regards, > >Martin > > > >________________________________ > >From: CF-metadata <cf-metadata-boun...@cgd.ucar.edu> on behalf of Jim Biard > ><jbi...@cicsnc.org> > >Sent: 10 May 2019 15:58 > >To: cf-metadata@cgd.ucar.edu > >Subject: Re: [CF-metadata] Missing data bins in histograms > > > > > >Hi. > > > >Sorry I have been so quiet lately. I've been caught up in other activities. > > > >I have a strong aversion to the proposal to overload the missing_value > >attribute with a wholly different meaning. Using missing_value in this way > >will produce unexpected results in a number of existing software packages. > >If the minor modification to CF to designate flag attributes to be used on > >coordinate variables doesn't seem like an acceptable solution for one reason > >or another, I think we should define a new convention that doesn't add > >contradictory interpretations of existing attributes. > > > >Grace and peace, > > > >Jim > > > >On 5/2/19 11:49 AM, Martin Juckes - UKRI STFC wrote: > > > >Dear Jonathan, Jim, > > > > > > > >I’m sorry to have dropped this conversation after starting it three years > >ago. We ended up not fixing the problem for CMIP6, but I think it is worth > >taking another look. > > > > > > > >Coming back to it again, I think that a variation on Jim’s suggestion could > >work: rather than using flags it should be possible to use a coordinate > >variable, as is done for some CMIP variables that have region names along > >one axis. The NetCDF dimension would be an index, and the array of values > >defining the bins would be an auxiliary coordinate which, I believe, is not > >subject to the rules on monotonicity and missing values which apply to > >NetCDF dimensions. There may be a need for some clarifications, but I think > >this approach would be much closer to the current convention that any change > >in the specification for non-auxiliary coordinate variables. > > > > > >We have a specific use case in CMIP6 for which the bins are height bins > >(height of detected cloud), with one bin reserved for "retrieval error". > > > > > >This might not need a change in the convention rules, but it would help, I > >think, to at least add an example and a standard name for the coordinate > >variable. For example: > > > > > >float data(time,lat,lon,zindex); > > > > data: standard_name = > > "histogram_of_equivalent_reflectivity_factor_over_height_above_reference_ellipsoid"; > > > > data: coordinates="zbins"; > > > >float zbins(zindex); > > > > zbins: long_name="Height ranges (with bin for missing data at first > > element)"; > > > > zbins:missing_value= -9999.; > > > > zbins: units="m"; > > > > zbins: bounds="zbin_bnds"; > > > > zbins: standard_name = "????"; > > > >float zbin_bnds(zindex,2); > > > > zbin_bnds:missing_value= -9999.; > > > >float lat(lat); > > > >float lon(lon); > > > > > >data: > > > > zbins = -9999., 25., 100., ....; > > > > zbin_bnds = -9999.,-9999., 0., 50., 50., 150., ... > > > > > >The use of missing_value in the bounds variable appears to conflict with > >conformance rules, but I'm not sure if this is really banned by the > >convention in this context. > > > > > >Using missing_value in this way appears to be acceptable to the convention, > >but I think it conflicts with the spirit of the convention: it is not > >indicating that a value of "zbins" is missing, but indicating that this > >index of the array relates to a count of missing values. For this reason I > >have omitted _FillValue. > > > > > >The "zbins" auxiliary coordinate here is a height-like variable, but I don't > >think we can use a standard name "height": is it worth adding a standard > >name "height_bins" defined to be "Height ranges, as used, for example in a > >histogram or frequency distribution. A variable with this standard name may > >include a special bin for the count or frequency of missing data. This > >should be indicated by setting the value of that bin and its bounds to equal > >the missing_value of the variable. If there is no missing value bin, it is > >recommended that the term 'height' be used instead." > > > > > >regards, > > > >Martin > > > > > >CF-metadata] Missing data bins in histograms > > > >Jonathan Gregoryj.m.gregory at reading.ac.uk > ><mailto:cf-metadata%40cgd.ucar.edu?Subject=Re%3A%20%5BCF-metadata%5D%20Missing%20data%20bins%20in%20histograms&In-Reply-To=%3C20161013094247.GF6219%40met.reading.ac.uk%3E><mailto:cf-metadata%40cgd.ucar.edu?Subject=Re%3A%20%5BCF-metadata%5D%20Missing%20data%20bins%20in%20histograms&In-Reply-To=%3C20161013094247.GF6219%40met.reading.ac.uk%3E> > >Thu Oct 13 03:42:47 MDT 2016 > > > > * Previous message (by thread): [CF-metadata] Missing data bins in > > histograms > > <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html> > > * Next message (by thread): [CF-metadata] Usage of > > histogram_of_X_over_Z > > <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html> > > * Messages sorted by: [ date > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984> > > [ thread > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984> > > [ subject > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984> > > [ author > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984> > > > >________________________________ > > > >Dear Jim > > > > > > > >In Appendix A it does not say that the flag attributes are allowed for > > > >coordinate variables - it has just "D" in the "Use" column. This is not an > > > >argument why they shouldn't be if there is a need, but they weren't > >introduced > > > >with that in mind. The use which you suggested for Martin's case is a good > > > >idea, but I think it would need a change to the convention. > > > > > > > >Best wishes > > > > > > > >Jonathan > > > > > > > >----- Forwarded message from Jim Biard <jbiard at > >cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > > ----- > > > > > > > > > > > >Date: Wed, 12 Oct 2016 14:58:11 -0400 > > > > > >From: Jim Biard <jbiard at > >cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > > > > > >To: cf-metadata at > >cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > > > > >Subject: Re: [CF-metadata] Missing data bins in histograms > > > > > >User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) > > > > > > Gecko/20100101 Thunderbird/45.4.0 > > > > > >Jonathan, > > > > > >Missing/fill values are not allowed, but I don't see any language > > > > > >prohibiting flags. I'd appreciate it if you could expand on your > > > > > >thoughts about why they aren't allowed. > > > > > >Grace and peace, > > > > > >Jim > > > > > >On 10/12/16 1:30 PM, Jonathan Gregory wrote: > > > > > >Dear Jim > > > > > >That is an ingenious idea. I don't think the flag atts are currently allowed > > > > > >for coord variables, but they could be, I agree. > > > > > >Best wishes > > > > > >Jonathan > > > > > >----- Forwarded message from Jim Biard <jbiard at > >cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > > ----- > > > > > >Date: Tue, 11 Oct 2016 14:39:56 -0400 > > > > > >From: Jim Biard <jbiard at > >cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > > > > > >To: cf-metadata at > >cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > > > > >Subject: Re: [CF-metadata] Missing data bins in histograms > > > > > >User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) > > > > > > Gecko/20100101 Thunderbird/45.4.0 > > > > > >Hi. > > > > > >Another approach could be to use flag_values and flag_meanings on > > > > > >the coordinate variable to indicate one or more special coordinate > > > > > >values that correspond to any number of "missing data" or "out of > > > > > >bounds" bins. These attributes aren't forbidden by CF, and > > > > > >everything should be fine as long as the coordinate variable remains > > > > > >monotonic. > > > > > >Grace and peace, > > > > > >Jim > > > > > >On 10/11/16 8:41 AM, martin.juckes at > >stfc.ac.uk<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > wrote: > > > > > >Hello, > > > > > >the CF standard name list has two "histogram_.... " entries, and in the > >CMIP6 data request we may need to add a third, a > >histogram_of_cloud_top_height. Besides the standard name, we also need, for > >this new variable, a method of encoding the "missing data" bin in the > >histogram. That is, the histogram should record frequency in 16 data bins > >and one additional bin for the frequency of missing data. > > > > > >Can we define a "missing_data_index" attribute for histogram variables, and > >use this to indicate that the first bin in the array has this special > >purpose. It might be more pythonic to put the _FillValue in the coordinate > >value for the missing data bin, but I suspect that this would cause > >substantial problems for many software packages. > > > > > >regards, > > > > > >Martin > > > > > >_______________________________________________ > > > > > >CF-metadata mailing list > > > > > >CF-metadata at > >cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > > > > >http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > > > > > >-- > > > > > >CICS-NC <http://www.cicsnc.org/><http://www.cicsnc.org/> Visit us on > > > > > >Facebook <http://www.facebook.com/cicsnc><http://www.facebook.com/cicsnc> > > *Jim Biard* > > > > > >*Research Scholar* > > > > > >Cooperative Institute for Climate and Satellites NC > ><http://cicsnc.org/><http://cicsnc.org/> > > > > > >North Carolina State University <http://ncsu.edu/><http://ncsu.edu/> > > > > > >NOAA National Centers for Environmental Information > ><http://ncdc.noaa.gov/><http://ncdc.noaa.gov/> > > > > > >/formerly NOAA’s National Climatic Data Center/ > > > > > >151 Patton Ave, Asheville, NC 28801 > > > > > >e: jbiard at > >cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > <mailto:jbiard at > >cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > > > > > >o: +1 828 271 4900 > > > > > >/Connect with us on Facebook for climate > > > > > ><https://www.facebook.com/NOAANCEIclimate><https://www.facebook.com/NOAANCEIclimate> > > and ocean and geophysics > > > > > ><https://www.facebook.com/NOAANCEIoceangeo><https://www.facebook.com/NOAANCEIoceangeo> > > information, and follow > > > > > >us on Twitter at @NOAANCEIclimate > > > > > ><https://twitter.com/NOAANCEIclimate><https://twitter.com/NOAANCEIclimate> > >and @NOAANCEIocngeo > > > > > ><https://twitter.com/NOAANCEIocngeo><https://twitter.com/NOAANCEIocngeo>. / > > > > > >_______________________________________________ > > > > > >CF-metadata mailing list > > > > > >CF-metadata at > >cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > > > > >http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > > > > > >----- End forwarded message ----- > > > > > >_______________________________________________ > > > > > >CF-metadata mailing list > > > > > >CF-metadata at > >cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > > > > >http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > > > > > >-- > > > > > >CICS-NC <http://www.cicsnc.org/><http://www.cicsnc.org/> Visit us on > > > > > >Facebook <http://www.facebook.com/cicsnc><http://www.facebook.com/cicsnc> > >*Jim Biard* > > > > > >*Research Scholar* > > > > > >Cooperative Institute for Climate and Satellites NC > ><http://cicsnc.org/><http://cicsnc.org/> > > > > > >North Carolina State University <http://ncsu.edu/><http://ncsu.edu/> > > > > > >NOAA National Centers for Environmental Information > ><http://ncdc.noaa.gov/><http://ncdc.noaa.gov/> > > > > > >/formerly NOAA’s National Climatic Data Center/ > > > > > >151 Patton Ave, Asheville, NC 28801 > > > > > >e: jbiard at > >cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > <mailto:jbiard at > >cicsnc.org<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata>> > > > > > >o: +1 828 271 4900 > > > > > >/Connect with us on Facebook for climate > > > > > ><https://www.facebook.com/NOAANCEIclimate><https://www.facebook.com/NOAANCEIclimate> > > and ocean and geophysics > > > > > ><https://www.facebook.com/NOAANCEIoceangeo><https://www.facebook.com/NOAANCEIoceangeo> > > information, and follow > > > > > >us on Twitter at @NOAANCEIclimate > > > > > ><https://twitter.com/NOAANCEIclimate><https://twitter.com/NOAANCEIclimate> > >and @NOAANCEIocngeo > > > > > ><https://twitter.com/NOAANCEIocngeo><https://twitter.com/NOAANCEIocngeo>. / > > > > > >_______________________________________________ > > > > > >CF-metadata mailing list > > > > > >CF-metadata at > >cgd.ucar.edu<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > > > > >http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata > > > > > >----- End forwarded message ----- > > > > > > > >________________________________ > > > > * Previous message (by thread): [CF-metadata] Missing data bins in > > histograms > > <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/018983.html> > > * Next message (by thread): [CF-metadata] Usage of > > histogram_of_X_over_Z > > <http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/008836.html> > > * Messages sorted by: [ date > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/date.html#18984> > > [ thread > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/thread.html#18984> > > [ subject > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/subject.html#18984> > > [ author > > ]<http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984><http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2016/author.html#18984> > > > >________________________________ > > > >More information about the CF-metadata mailing > >list<http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata><http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata> > > > > > >-- > >[CICS-NC] <http://www.cicsnc.org/> Visit us on > >Facebook <http://www.facebook.com/cicsnc> Jim Biard > >Research Scholar > >Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/> > >North Carolina State University <http://ncsu.edu/> > >NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/> > >formerly NOAA’s National Climatic Data Center > >151 Patton Ave, Asheville, NC 28801 > >e: jbi...@cicsnc.org<mailto:jbi...@cicsnc.org> > >o: +1 828 271 4900 > > > >Connect with us on Facebook for > >climate<https://www.facebook.com/NOAANCEIclimate> and ocean and > >geophysics<https://www.facebook.com/NOAANCEIoceangeo> information, and > >follow us on Twitter at > >@NOAANCEIclimate<https://twitter.com/NOAANCEIclimate> and > >@NOAANCEIocngeo<https://twitter.com/NOAANCEIocngeo>. > > > -- > CICS-NC <http://www.cicsnc.org/> Visit us on > Facebook <http://www.facebook.com/cicsnc> *Jim Biard* > *Research Scholar* > Cooperative Institute for Climate and Satellites NC <http://cicsnc.org/> > North Carolina State University <http://ncsu.edu/> > NOAA National Centers for Environmental Information <http://ncdc.noaa.gov/> > /formerly NOAA’s National Climatic Data Center/ > 151 Patton Ave, Asheville, NC 28801 > e: jbi...@cicsnc.org <mailto:jbi...@cicsnc.org> > o: +1 828 271 4900 > > /Connect with us on Facebook for climate > <https://www.facebook.com/NOAANCEIclimate> and ocean and geophysics > <https://www.facebook.com/NOAANCEIoceangeo> information, and follow > us on Twitter at @NOAANCEIclimate > <https://twitter.com/NOAANCEIclimate> and @NOAANCEIocngeo > <https://twitter.com/NOAANCEIocngeo>. / > > > _______________________________________________ > CF-metadata mailing list > CF-metadata@cgd.ucar.edu > http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata ----- End forwarded message ----- _______________________________________________ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
