Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Congratulations and thanks to all who contributed to this successful piece of work. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-904724784__;Iw!!G2kpM7uM-TzIFchu!h7VoghGjMTn1rrd3O_N1QAOrLj190n7zp4pggJsXCyTbrQ2s7gFTncEWbj66tZlWIj_b7HFgLDw$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
@JonathanGregory @AndersMS et al., chanting is all done and the merge is complete. Thanks all for your many varied contributions - this was a lot of work on all sides and my hope is that it proves useful to both data producers and consumers moving forward! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-904601442__;Iw!!G2kpM7uM-TzIFchu!jxv4NRIdbU84GAVIV88fJ7CgJMx8tjsA2YeicKv0UGHJ1aU5IhjerqYcyvxeTLnILgVW2eLzEAA$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Closed #327 via #326. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*event-5200310874__;Iw!!G2kpM7uM-TzIFchu!msq58nKwzlkJWlCUNosIPase5AbocSw9nplikWLAQY5SigSwXARReQ6y9GbEJtTLz0s3z3bodng$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS @erget et al. I would be pleased to merge the pull request and close this issue, but I see that the PR has conflicts which have to be resolved. I expect there is some GitHub incantation which you can pronounce to resolve them. Best wishes Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-904572648__;Iw!!G2kpM7uM-TzIFchu!gZUt0MWRaeCx0b4h4xV_rBDWxze5PxRuOpWWGQL-9hAd-ijc18n7ocqd7PkzUCCc-rXUmYV5uSM$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS Thanks for the clarification. That's fine. The proposal will be approved this Friday 24th if no further concern is raised. Best wishes Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-899510310__;Iw!!G2kpM7uM-TzIFchu!iJBr3vUXMH645_xjjCkNr9P6cyYvQM4DDfPcFX7yz5ofkX_2ix1_BVD3ornVu1LKRTHFisMjyHg$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory , We have just discussed the matter of the cell bounds interpolation and the question you [raised ](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-895958409__;Iw!!G2kpM7uM-TzIFchu!gLEFMHJr1jaLv0o50ir8t01K17y2gyz2I31bLEGfwvRA87BLO-oQfsnAUqtt7ygHoywYEwBVj0M$ ). To make the conditions for bounds interpolation clearer, we have changed (b10fb67) the first part of the first paragraph in the section on bounds interpolation to: > Coordinates may have cell bounds. For the case that the reconstituted cells > are contiguous and have exactly two cell bounds along each interpolated > dimension, cell bounds of interpolated dimensions can be stored as bounds tie > points and reconstituted through interpolation. We hope you are fine with that change. Best regards, The interpolation team -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-896011153__;Iw!!G2kpM7uM-TzIFchu!gLEFMHJr1jaLv0o50ir8t01K17y2gyz2I31bLEGfwvRA87BLO-oQfsnAUqtt7ygHoywYWjRgFZ8$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory, Once again, thank you very much for your thorough review and valuable comments, which significantly improved the proposal. Cheers Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-895960636__;Iw!!G2kpM7uM-TzIFchu!nlJe9Cy-VdeL3NpxkKi9PSms4PSmRIEwCgS1uN1p387hD8qU_x9RFt_KP7z6mlCI8XWifZAv2vY$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory , Regarding the interpolation of bounds, [you asked](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-886687324__;Iw!!G2kpM7uM-TzIFchu!gwecS-gI9rNIndTP8rqMcB6mXYSV_fxclkyOlS4CO2hI_NE9xWHxmdM2h4s2mto-MGJkkxsi9oo$ ): > Are you restricting the consideration of 2D bounds to rectangular cells, or > are polygons of n vertices allowed? We are restricting the interpolation of bounds to contiguous cell bounds. I think that the consequence of this is that we are are restricting the consideration of 2D bounds to rectangular cells. Possibly @davidhassell can confirm. What we do support is interpolation of 1D, 2D, etc bounds. Hence the sentence: > One of the vertices of each coordinate tie point cell is chosen as the bounds > tie point for the cell. It is selected as the vertex of the tie point cell > that is the closest to the boundary of the interpolation subarea with respect > to each interpolated dimension. that applies for any number of interpolated dimensions. Cheers Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-895958409__;Iw!!G2kpM7uM-TzIFchu!gwecS-gI9rNIndTP8rqMcB6mXYSV_fxclkyOlS4CO2hI_NE9xWHxmdM2h4s2mto-MGJkbfpAcUM$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS @davidhassell @erget @oceandatalab and collaborators Thanks for the enormous amount of hard and thorough work you have put into this, and for answering all my questions and comments. I have no more concerns. Looking through the rendered PDF of App J, I see boxes, probably indicating some character which Chrome can't print, in "Common Conversions and Formulas", after sin and cos. If anyone else would like to review and comment, they are welcome to do so. If no further concerns are raised, the proposal will be accepted on 24th August. Cheers Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-892015911__;Iw!!G2kpM7uM-TzIFchu!lkaOxM7nuhzyPl13cR61ODihBZf0LdEhtPMkaOSWlI8wSZaYGFEi5_dwjoll274VMTZgKQiX2wk$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory et al., Due to the heroic contributions primarily of @AndersMS and @davidhassell as well as the expert review of @oceandatalab and friends we can present to you the now-finalised version of the pull request associated with this issue. To see all points listed and addressed one-by-one you can check https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-885811067__;Iw!!G2kpM7uM-TzIFchu!iuHGEKXnfmeYQ8ydb1UlV1etKqehmIGKeGsvche4pbqkdWoMo5-ZHhtAXlpivtB_wIcL_i4gnsQ$ hopefully that is traceable. We have completed our proposal, finalising the section regarding computational precision - this is now found at the end of chapter 8.3. https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/pull/326__;!!G2kpM7uM-TzIFchu!iuHGEKXnfmeYQ8ydb1UlV1etKqehmIGKeGsvche4pbqkdWoMo5-ZHhtAXlpivtB_wIcLeb7TFAc$ contains the documents in their latest state, which I have also attached in a compiled form for your perusal: - [cf-conventions.pdf](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/files/6924419/cf-conventions.pdf__;!!G2kpM7uM-TzIFchu!iuHGEKXnfmeYQ8ydb1UlV1etKqehmIGKeGsvche4pbqkdWoMo5-ZHhtAXlpivtB_wIcLEkUTLqE$ ) - [conformance.pdf](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/files/6924421/conformance.pdf__;!!G2kpM7uM-TzIFchu!iuHGEKXnfmeYQ8ydb1UlV1etKqehmIGKeGsvche4pbqkdWoMo5-ZHhtAXlpivtB_wIcL6j8oWHw$ ) Note before finalisation of this version of the Conventions the following items will need to be addressed; these are however of a purely editorial nature so in the interest of time we are not correcting them for the 3 week freeze: - Fix numbering and labelling of figures - Fix image scaling so that the PDF doesn't make you squint This will need to be fixed globally before publishing so that figures don't get squeezed in. A clever idea here would be to name e.g. the first figure in chapter 7 "Figure 7.1" so that the figures are always numbered correctly independently of previous chapters. I leave this to future minds to solve. I therefore thank all contributors again for the loads of precise and hard work, and motion that the 3 week period start for this proposal so that we are on time to get it adopted into CF-1.9. I look forward to hearing hopefully a resounding silence in response to the finalised proposal! - @erget -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-891853175__;Iw!!G2kpM7uM-TzIFchu!iuHGEKXnfmeYQ8ydb1UlV1etKqehmIGKeGsvche4pbqkdWoMo5-ZHhtAXlpivtB_wIcL5BDyBXY$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory, @AndersMS, and all, > Conformance > > For "Each tie_point_variable token specifies a tie point variable that must > exist in the file, and each interpolation_variable token specifies an > interpolation variable that must exist in the file," I think all you can say > is that there are variables of these names in the file, since a checker can't > tell they are definitely the "kind" of variable you intend. > > Regarding, "The legal values for the interpolation_name attribute are > contained in Appendix J," it would be helpful for the author of the checker > to say where they can be found in the appendix. Addressed in https://urldefense.us/v3/__https://github.com/AndersMS/cf-conventions/pull/21/commits/8b8c1850321077711e8d786c53625d1aae60a042__;!!G2kpM7uM-TzIFchu!irvVRQKnhtnc-JLaGpmnQ4ynjO-FpA3Cx--SpGojd__vz_1ZSMNqKFpFvnriI9MS_rFcAhIp19E$ I have also added some conformance requirements and recommendations for bounds tie point variables: https://urldefense.us/v3/__https://github.com/AndersMS/cf-conventions/pull/21/commits/bdac108def687b64db8693aa4f1e50d45c120ce0__;!!G2kpM7uM-TzIFchu!irvVRQKnhtnc-JLaGpmnQ4ynjO-FpA3Cx--SpGojd__vz_1ZSMNqKFpFvnriI9MS_rFcItwL9wM$ Thanks, David -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-887376988__;Iw!!G2kpM7uM-TzIFchu!irvVRQKnhtnc-JLaGpmnQ4ynjO-FpA3Cx--SpGojd__vz_1ZSMNqKFpFvnriI9MS_rFcsgMUFX0$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS Thanks for your detailed replies. I think there are only two outstanding points in those you have answered. **18**: Now I understand what you mean, thanks. To make this clearer to myself, I would say something like this: Bounds interpolation uses the same tie point index variables and therefore the same tie point cells as coordinate interpolation. One of the vertices of each coordinate tie point cell as chosen as the bounds tie point for the cell. For 1D bounds, the vertex chosen is the one which is on the side closer to the boundary of the interpolation subarea. For 2D bounds, the vertex chosen is the one which is closest to the boundary of the interpolation subarea, considering all the interpolated coordinates together, or in other words, the one closest to the corner of the interpolation subarea. Are you restricting the consideration of 2D bounds to rectangular cells, or are polygons of _n_ vertices allowed? **27**: I think the key point is that you mean _three-dimensional_ Cartesian interpolation. I didn't think of that. If you could clarify this, it would be fine. Cheers Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-886687324__;Iw!!G2kpM7uM-TzIFchu!iEy4S2WEt_kU5_OoPHE9-yyJcgP-l8nMW5sdOcoONKxT3tVZLZ5Q8jC-27FnuTaoo6nVAuGz0IE$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory, Just to let you know that I just updated my reply to **Reply to Comment/Proposed Change 23** [above](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-885811067__;Iw!!G2kpM7uM-TzIFchu!ka13PuCYIfBl_QKqXo21oEaMqWLa1EN27lGb6hy09dPP5UUgs7iV-0JpWNlHHlM1qchlDY7KIrM$ ). Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-886255154__;Iw!!G2kpM7uM-TzIFchu!ka13PuCYIfBl_QKqXo21oEaMqWLa1EN27lGb6hy09dPP5UUgs7iV-0JpWNlHHlM1qchlpS13IHo$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear All, Here are the links to the easy-to-read versions including all the above changes: - [Chapter 8](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/blob/303aaa440351b05a7ff52d4329426d81a5c1e7bc/ch08.adoc__;!!G2kpM7uM-TzIFchu!irpikmmVtMlQcObJWZC6fGNaurWQ87IykKVKcz8sbsz6QfKl2-g1LotjvMhH17Jgdy5YnVrB--0$ ) - [Appendix J](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/blob/303aaa440351b05a7ff52d4329426d81a5c1e7bc/appj.adoc__;!!G2kpM7uM-TzIFchu!irpikmmVtMlQcObJWZC6fGNaurWQ87IykKVKcz8sbsz6QfKl2-g1LotjvMhH17Jgdy5YdpR1rnM$ ) Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-885817089__;Iw!!G2kpM7uM-TzIFchu!irpikmmVtMlQcObJWZC6fGNaurWQ87IykKVKcz8sbsz6QfKl2-g1LotjvMhH17Jgdy5YW0OF6dU$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory , Thank you for your rich set of comments and suggestions. I have provided replies below, in the same format we used for the first set of comments. Several of the replies I have already implemented in the document and indicated the corresponding commit. For others, the reply is not conclusive and if you find time, your feedback on the reply would be valuable. The comments on the conformance chapter, I would prefer that @davidhassell look at when he is available again. Best regards, Anders **Comment/Proposed Change 17** > Chapter 8: "Tie point mapping attribute" mentions "target dimension", which > is not a phrase used elsewhere. Should this be "interpolated dimension"? **Reply to Comment/Proposed Change 17** You are right, it should be "interpolated dimension" in that section. I have updated the text. **Commit(s) related to Comment/Proposed Change 17** ca81618 **Comment/Proposed Change 18** > Chapter 8: You say, "For the purpose of bounds interpolation, a single bounds > tie point is created for each coordinate tie point, and is selected as the > vertex of the tie point cell that is the closest to the boundary of the > interpolation subarea with respect to each interpolated dimension." I don't > understand why there is a choice of bounds tie points, because there's no > index variable for them. Doesn't the tie point index variable dictate the > choice of tie points for bounds? **Reply to Comment/Proposed Change 18** In the compressed data set we only store one bounds tie points per coordinate tie point. However, in the existing boundary variable defined in section 7.1. Cell Boundaries, requires you to store, in the case of 2D bounds for example, fours bounds. The selection is between those four bounds, of which only one is the correct selection. Did that help? Let me know if you think we can improve the text in that respect. **Commit(s) related to Comment/Proposed Change 18** None **Comment/Proposed Change 19** > Appendix J: The title says Appendix A. Presumably that's something to do with > automatic numbering. **Reply to Comment/Proposed Change 19** Correct, that will be updated as part of the publishing magic. **Commit(s) related to Comment/Proposed Change 19** None. **Comment/Proposed Change 20** > Appendix J: All of the subsections listed at the start (Common Definitions > and Notation, Common conversions and formulas, Interpolation Methods, > Coordinate Compression Steps, Coordinate Uncompression Steps) should have > subsection headings, I think. They will be Sections J.1 etc. At the moment > the last two are labelled as Tables J.1 and J.2 rather than subsections, but > they're never referenced as tables. **Reply to Comment/Proposed Change 20** I agree and have introduced section numbering and removed table captions in Appendix J. **Commit(s) related to Comment/Proposed Change 20** f6f48fb **Comment/Proposed Change 21** > Appendix J: Fig 1ff. s is explained beneath the fig, but it would be useful > it explain it at the side of the fig as well, as you do for tp i and u. **Reply to Comment/Proposed Change 21** Done. Also, I have named s as the “interpolation argument”, which I think is what it is. **Commit(s) related to Comment/Proposed Change 21** 1002806 **Comment/Proposed Change 22** > Appendix J: Also, it would be useful to put the paragraph explaining > notation before Fig 1, because Fig 1 uses the notation. **Reply to Comment/Proposed Change 22** Agree. Done. **Commit(s) related to Comment/Proposed Change 22** 0fdc7e4 **Comment/Proposed Change 23** > You say, "When an interpolation method is referred to as linear or quadratic, > it means that the method is linear or quadratic in the indices of the > interpolated dimensions." Linear also means that the coordinates of the > interpolated points are evenly spaced, doesn't it; if so, that would be > helpful to state. > Appendix J: **Reply to Comment/Proposed Change 23** Good observation, I have added that. **Commit(s) related to Comment/Proposed Change 23** 5f9ad9a **Comment/Proposed Change 24** > Appendix J: You say, "In the case of two dimensional interpolation, the two > variables are equivalently computed as ...". I would say "similarly", not > "equivalently", which I would understand to mean that s1 and s2 are > equivalent. **Reply to Comment/Proposed Change 24** Done. **Commit(s) related to Comment/Proposed Change 24** 0116283 **Comment/Proposed Change 25** > Appendix J: It would be better not to use c for the coefficient, because it > can be confused with the point c. **Reply to Comment/Proposed Change 25** Agreed and renamed from “c” to “w”. Also renamed related function fc() to fw(). **Commit(s) related to Comment/Proposed Change 25** ea474a5 **Comment/Proposed Change 26** > Appendix J: Please put the "Common conversion and formulae" table before the > interpolation methods, or at least refer to it. Otherwise the reader > encounters fdot fcross
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear all @AndersMS and colleagues have proposed a large addition to Chapter 8 and an accompanying new appendix to the CF convention, defining methods for storing subsampled coordinate variables and the descriptions of the interpolation methods that should be used to reconstruct the entire (uncompressed) coordinate variables. I've reviewed this in detail and it makes sense and seems to clear to me, as someone who's never used these methods. Those who wrote this proposal are the experts. Enough support has been expressed for this proposal to be adopted, after allowing the time prescribed for the rules for further comments, and there are no objections expressed. Therefore this proposal is on course for adoption in the next release of the CF convention as things stand. If anyone else who wasn't involved in preparing it has the time and interest to review it, that would no doubt be helpful, and _now_ is the time to do that, in order not to delay its approval. It definitely requires careful reading and thinking, but it's logical and well-illustrated. Best wishes Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-885632022__;Iw!!G2kpM7uM-TzIFchu!hv6vIhhlKtmWg-m4ws3Q2yCvfdVTijT--7gElphaVliGuTAWYIstwxv86vSOrJsk5gOck91vdxE$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS and colleagues Thanks again for the new version. I find it very clear and comprehensive. I have a few comments. ## Chapter 8 "Tie point mapping attribute" mentions "target dimension", which is not a phrase used elsewhere. Should this be "interpolated dimension"? You say, "For the purpose of bounds interpolation, a single bounds tie point is created for each coordinate tie point, and is selected as the vertex of the tie point cell that is the closest to the boundary of the interpolation subarea with respect to each interpolated dimension." I don't understand why there is a choice of bounds tie points, because there's no index variable for them. Doesn't the tie point index variable dictate the choice of tie points for bounds? ## Appendix J The title says Appendix A. Presumably that's something to do with automatic numbering. All of the subsections listed at the start (Common Definitions and Notation, Common conversions and formulas, Interpolation Methods, Coordinate Compression Steps, Coordinate Uncompression Steps) should have subsection headings, I think. They will be Sections J.1 etc. At the moment the last two are labelled as Tables J.1 and J.2 rather than subsections, but they're never referenced as tables. Fig 1ff. `s` is explained beneath the fig, but it would be useful it explain it at the side of the fig as well, as you do for `tp` `i` and `u`. Also, it would be useful to put the paragraph explaining notation _before_ Fig 1, because Fig 1 uses the notation. You say, "When an interpolation method is referred to as linear or quadratic, it means that the method is linear or quadratic in the indices of the interpolated dimensions." Linear also means that the coordinates of the interpolated points are evenly spaced, doesn't it; if so, that would be helpful to state. You say, "In the case of two dimensional interpolation, the two variables are equivalently computed as ...". I would say "similarly", not "equivalently", which I would understand to mean that `s1` and `s2` are equivalent. `quadratic`. It would be better not to use `c` for the coefficient, because it can be confused with the point **c**. Please put the "Common conversion and formulae" table before the interpolation methods, or at least refer to it. Otherwise the reader encounters `fdot` `fcross` `fplus` `fminus` `fmultiply` etc. without having seen their definitions. Actually you list it before the interpolation methods in the preamble. [`bi_`]`quadratic_remote_sensing`. Why not call it [`bi_`]`quadratic_latitude_longitude`, which describes the method, rather than its typical application? What does it mean to treat them as Cartesian or not? I would describe bilinear interpolation in lat,lon as treating them as Cartesian coordinates, but you must mean something different. Is there a projection plane involved? Where is `latitude_limit` defined? A couple of times, you write, "For each of the interpolated dimension". There should be an -s. ## Conformance For "Each **`tie_point_variable`** token specifies a tie point variable that must exist in the file, and each **`interpolation_variable`** token specifies an interpolation variable that must exist in the file," I think all you can say is that there are variables of these names in the file, since a checker can't tell they are definitely the "kind" of variable you intend. Regarding, "The legal values for the **`interpolation_name`** attribute are contained in Appendix J," it would be helpful for the author of the checker to say _where_ they can be found in the appendix. Best wishes Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-885626643__;Iw!!G2kpM7uM-TzIFchu!l8ErbkKYKTUpxE4i-cyjD81MQrn8MvQQkK84MM_g-jMgwBcJx3tqKmBJl5HBHowuMe2UL-cwEI0$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Great, thanks, @AndersMS. I am still learning about GitHub. I was using the Diff, which doesn't show the diagrams, rather than Viewing the file, which works fine. Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-885594652__;Iw!!G2kpM7uM-TzIFchu!nlwgFwIXDMd2b_9Y78Q2mpVCdOcDDqkCV_kW2lOH0xbxLgJtWEFFWTJvzfqW1treqbgETW_f7V0$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory , I am still a bit new to documents on GtiHup, but these two links does the job in my browser: - [Chapter 8](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/blob/d033feeb798aa4c63e3d0d8ce04d27809dde2d0f/ch08.adoc__;!!G2kpM7uM-TzIFchu!nPmNHzbRWh73ZUUuMSyhan9-uga-SkgzoF66c0eWKNiFUljSypmhaNNUwaUml07t1-ATSZ8voBk$ ) - [Appendix J](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/blob/d033feeb798aa4c63e3d0d8ce04d27809dde2d0f/appj.adoc__;!!G2kpM7uM-TzIFchu!nPmNHzbRWh73ZUUuMSyhan9-uga-SkgzoF66c0eWKNiFUljSypmhaNNUwaUml07t1-ATvT6rAC4$ ) I got these links by going to #326, then selecting the _Files changed_ tab, then scrolling down to _ch08.adoc_ or _appj.adoc_ and then selecting _View File_ in the "... " pull down menu on the right hand side, opposite to the file name. Hope this will work at your end. We had a section on boundary interpolation in the first version you read, but it was short and didn`t do the job we would like it to do. For example, it did not guarantee to reconstitute contiguous bounds as contiguous bounds. The new section is our consolidated version, which does all what we wanted it to do. Best regards, Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-885099665__;Iw!!G2kpM7uM-TzIFchu!nPmNHzbRWh73ZUUuMSyhan9-uga-SkgzoF66c0eWKNiFUljSypmhaNNUwaUml07t1-ATyiTeTTE$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS et al. Thanks for the new version. Can you tell me where to find versions of Ch 8 and App J with the figures in place? That would make it easier to follow. I've just read the text of Ch 8, which I found much clearer than before. I don't recall reading about bounds last time. Is that new, or was I asleep? Best wishes Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-885067233__;Iw!!G2kpM7uM-TzIFchu!mphOawkadNGwafnEAo-0MbLTcK9sSubqx-lwFjAaOKcAiM_e9rzgic2zF32NRV3HAN83wJcBcLw$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory, Appendix J is now ready for your review. The only remaining open issues is now that we will do one more iteration on the section on Computational Precision for Chapter 8 - we will publish it here within the next days. Best regards, Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-884514418__;Iw!!G2kpM7uM-TzIFchu!kujy8_Q2ZkyKKZU8JoY6spnpm-TX8CwANNQxGnbkunyt6ataFH0JTxfS4bnqOUs_k3-aFNWrkbA$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear All, Just to let you know that as agreed during the discussion of the new "Interpolation of Cell Boundaries" section (f3de508) I have added a the following sentence in the "Interpolation Parameters" section (2ce5d66): > Interpolation parameters are not permitted to contain absolute coordinate > information, such as additional tie points, but may contain relative > coordinate information, for example an offset with respect to a tie point or > with respect to a combination of tie points. This is to ensure that > interpolation methods are equally applicable to both coordinate and bounds > interpolation. Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-884275934__;Iw!!G2kpM7uM-TzIFchu!k1DuYS60h9ZIZA2Ean9YfbrOXjEtoZtzW66cDro25WzqmX25qcDtkFv_L_4QHVF9RXi1dU6fB3A$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS Thanks for the update and your hard work on this. I will read the section again in conjunction with Appendix J, once you announce that the latter is ready. Best wishes Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-884128750__;Iw!!G2kpM7uM-TzIFchu!mTYSLA72tpB_bJ7V6iBXXBSNE5l6BAJgS2FYU5Sv6BamMTwyX1Arp6rw5NJtZYO50rk_Cgmm3pY$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear team, Following our meeting this afternoon, I propose the following new paragraph at the end of the section "Tie Points and Interpolation Subareas": > Tie point coordinate variables for both coordinate and auxiliary coordinate > variables must be defined as a numeric data type and are not allowed to have > missing values. Please let me know if you have comments. Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-883656064__;Iw!!G2kpM7uM-TzIFchu!iXED3weNs_vMTclHfLaag1U1oy-DQjLfQbts-al6vkdRdzU3R-K5-vJj74gtWiCdHkwONsohFMA$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
> Maybe we could append a new item at the end of "Coordinate Compression Steps" > in Appendix J recommending that data producers check the positional error by > comparing the reconstructed coordinates against the original data, and then > provide as many details as possible regarding the reconstruction process and > results (computational precision, positional error, etc...) in the `comment` > attribute of the subsampled coordinates variables. Thanks, Sylvain, I support this suggestion -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-876374013__;Iw!!G2kpM7uM-TzIFchu!kZszGue4OdMmpZ1gqyUUx7l4B062bw9hFfkvShkLgNOnhvlySBeut2HTEbEHPrFGIZFfGUVhzNo$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
We may be solving a problem here before it arises. From this arises the danger that we'll solve a problem that won't arise, or that we'll solve it in a way that's not as useful as it could be! It seems that computational precision is neither sufficient to describe the actual target, which is the positional error, nor is it necessary when considered in light of the rest of the Conventions, which also do not give such low-level details about numerical reconstruction - although this might be relevant for geophysical variables and the like! I propose therefore leaving it off. Data producers do have this field: `comment : Miscellaneous information about the data or methods used to produce it.` @AndersMS et al. FYI -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-874741086__;Iw!!G2kpM7uM-TzIFchu!lBKAneM_G9Y15zH8T37BTqvcYwVPIVzO7vA6qyv7cSdAjRTMBEAgnG8UB-ZKUsw6Ly89fTg-WWU$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear All,
Regarding the wording of the section on computational precision attribute, I
have reservations with respect to the direction it has taken and I suggest we
discuss the matter during our meting this afternoon.
It is essential to the value and usability of the of the _Lossy Compression by
Coordinate Sampling_ to reach a common understanding on this and get the
wording of the new section right.
Here are a couple of thoughts and comments for the further discussion.
@oceandatalab : You wrote:
> The "{...] using 64-bit floating-point arithmetic will reconstitute [...]" in
> the shorter version is misleading from my point of view because it eludes the
> software/hardware factor (though I agree it will not be an issue in most
> cases).
The full sentence
[here](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-873017814__;Iw!!G2kpM7uM-TzIFchu!mVh75PtEay7PlpX00xbIr2k6SEGL1_Y74IoiSZ-UXj9poMrXgzOOnSnvj4a7nXM2TDZ9PKaVr0c$
) was:
> As an example, a computational_precision = "64" would provide the guidance to
> the data user that using 64-bit floating-point arithmetic will reconstitute
> the coordinates with **an accuracy comparable** to the accuracy intended by
> the data creator.
and I think that with the wording **an accuracy comparable** in the sentence is
reasonable as I wrote it.
@davidhassell: You wrote:
> The accuracy will also depend, however, on how the interpolation method is
> implemented
and
> There are no restrictions on the choice of interpolation method
> implementation, for neither the data creator nor the data user,
I am uncertain about the meaning of this. As I see it, most of what we have
written is aimed at accurately and completely describe both the process of
compressing coordinates and uncompressing coordinates, so what scope do you see
for variations in implementations that would have an effect on the numerical
results of the uncompression process?
If we look at one of the interpolation methods, say the Biquadratic
Interpolation of geographic coordinates method, it will never fully reproduce
the coordinates, unless the the original coordinates were located in a perfect
biquadratic manner, which is typically not the case. This is true even if we
disregard limitations of the floating point arithmetic and differences of the
computing platform. One can think of this as the mathematical performance of
the method, excluding any affects from floating point arithmetic and
differences of the computing platform.
In the VIIRS example we have looked at, this mathematical performance is in the
order of 0.5 m when using 16x16 point size of the interpolation subarea for
VIIRS M-Band.
If we look at the capability of floating-point numbers to represent a position
on a global scale (Earth radius of 6371) then we have that:
- 32-bit floating-point, having 7.22 significant decimal digits, can represent
a global position to a precision of 0.38 m.
- 64-bit 3 floating-point, having 15.95 significant decimal digits, can
represent a global position to a precision of 0.7 10-9 m or 0.7 nano-meter, a
distance comparable to the diameter of our largest atoms.
The floating-point arithmetic operation that constitutes the interpolation
method will contribute to degrading the precision of the reconstituted
coordinates, compared to the precision of the floating point precision itself.
If we say that this degradation of precision is one to two orders of magnitude,
then we have that:
- Applying 32-bit floating-point arithmetic to the VIIRS example, the error
introduced by the floating-point arithmetic operations will be **one to two
orders of magnitude larger** than the pure mathematical performance of the
biquadratic method.
- Applying 64-bit floating-point arithmetic to the VIIRS example, the error
introduced by the floating-point arithmetic operations will be **seven to eight
orders of magnitude smaller** than the pure mathematical performance of the
biquadratic method.
As it can be seen, in this example, applying 32-bit floating-point arithmetic
may noticeably impact results in a negative way, whereas applying 64-bit
floating-point arithmetic will not noticeably impact results, when compared to
the overall mathematical performance of the method.
As I understand it, computing platform variations in floating-point arithmetic
implementations will mainly have the effect of introducing errors/deviations in
the last significant bits of the floating-point numbers. So if the
computational precision is chosen to have sufficient margin with respect to the
mathematical performance (in the example above, this would be 64-bit
floating-point arithmetic) the effect of computing platform variations are
unlikely to be noticeable for a well implemented interpolation method.
Cheers
Anders
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi @davidhassell and @oceandatalab, I also support the `computational precision` paragraph by @davidhassell presented here https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-873949026__;Iw!!G2kpM7uM-TzIFchu!ifSyuOkdJsHGTTtpVUg4RK3D8mVVpZqY5P3kLHxmFbskdHB9H3WmMq_LpGYgP_r397N7e4rH6t8$ Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-874183415__;Iw!!G2kpM7uM-TzIFchu!ifSyuOkdJsHGTTtpVUg4RK3D8mVVpZqY5P3kLHxmFbskdHB9H3WmMq_LpGYgP_r397N7IRLcMek$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi @davidhassell, I am in favor of your version of the "computational precision" paragraph: it conveys all the required information while remaining concise and yet clearly warns users about the limited scope of the `computational_precision` attribute. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-874143671__;Iw!!G2kpM7uM-TzIFchu!gxMJ-sRx0cjgF9WrPIDWvA6LxcXTtxMdQzbWHJCucX2ct4-WT8ePQV4BnEF1qx9I4mL5sesprKw$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear All, As proposed [above](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-872151596__;Iw!!G2kpM7uM-TzIFchu!i2lqlelli9HNPJ8B2iYj8zTfVVqAhReHwOARRaVVBHVvLhBGBN3dSzLqichBRORcq_I9wjOrnsg$ ) I will go ahead and change all occurrences and forms of **_sample_** with **_subsampled_** in the present PR #326, including the headings of chapter 8.3, Append J and chapter 8.3 of the conformance document, unless I receive reservations against the proposal by tomorrow end of business. The new title of Chapter 8.3 would then become **_Lossy Compression by Coordinate Subsampling_**. Best regards, Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-874092855__;Iw!!G2kpM7uM-TzIFchu!i2lqlelli9HNPJ8B2iYj8zTfVVqAhReHwOARRaVVBHVvLhBGBN3dSzLqichBRORcq_I9NBsQDpk$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi all, Sylvain's descriptions and rational are very good, I think. I am wondering, however, if we are making too bold claims about accuracy when we have no control over the interpolation method's implementation. A user's technique may differ from the creator's (that's OK), but if one technique was numerically ill-conditioned and the other not, even using the same precision could lead to inaccurate results. With that in mind, here's another suggestion (I think I prefer the one-paragraph approach, as the it helps connect the constituent points, but I don't have a strong opinion on that): --- The accuracy of the reconstituted coordinates depends mainly on the degree of subsampling and the choice of interpolation method, both of which are set by the creator of the dataset. The accuracy will also depend, however, on how the interpolation method is implemented and on the computer platform carrying out the computations .There are no restrictions on the choice of interpolation method implementation, for neither the data creator nor the data user, but the floating-point arithmetic precision used by the data creator during the preparation and validation of the compressed coordinates must be specified by setting the interpolation variable’s **`computational_precision`** attribute to one of the following values: (table) "32": 32-bit floating-point arithmetic, comparable to the binary32 standard in [IEEE_754] "64": 64-bit floating-point arithmetic, comparable to the binary64 standard in [IEEE_754] Using the given computational precision in the interpolation computations is a necessary, but not sufficient, condition for the data user to be able to reconstitute the coordinates to an accuracy comparable to that intended by the data creator. For instance, a **`computational_precision`** value of "`64`" would specify that, using the same software and hardware as the creator of the compressed dataset, sufficient accuracy could not be reached when using a floating-point precision lower than 64-bit floating-point arithmetic in the interpolation computations required to reconstitute the coordinates. --- -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-873949026__;Iw!!G2kpM7uM-TzIFchu!ladz7sdGh6BjP3nd9S2KrbHPI-fJb3_VKoQrkFgjZDNTUIFICvRylNi8OInJJhHg7H2LhAo84n0$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Thank you for the comments @AndersMS and @erget.
I like the concise version too, I would just keep my version of the "As an
example ..." paragraph even if it is more verbose because it states exactly
what the attribute means, hopefully leaving no room for misinterpretation.
The "{...] using 64-bit floating-point arithmetic **will** reconstitute [...]"
in the shorter version is misleading from my point of view because it eludes
the software/hardware factor (though I agree it will not be an issue in most
cases).
As for regrouping the 3 paragraphs into one, I think we should keep them
separated so that the content of paragraph 2 stands out: it is really important
to state that exact reproducibility is not what is offered here so that users
don't have unrealistic expectations.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-873101481__;Iw!!G2kpM7uM-TzIFchu!mR0NrSSMiY12UQZWKIZ0g91s2QsI-Fa_aa7cFmV3vS_lyT8vDFC5XLU-kNzho-OZSDjkRA3viok$
This list forwards relevant notifications from Github. It is distinct from
cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the
UCAR list will result in a subscription to this list.
To unsubscribe from this list only, send a message to
cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
@oceandatalab (Sylvain) & @AndersMS - I am in favour of the shorter text; in fact, perhaps one could combine these 3 paragraphs into 1: > The accuracy of the reconstituted coordinates will mainly depend on the > degree of subsampling, the choice of interpolation method and the choice of > the floating-point arithmetic precision used in the interpolation method > computations. > > The accuracy of the reconstituted coordinates may also depend on details of > the interpolation method implementation and on the computer platform, meaning > that the results of the coordinate reconstitution process may not be fully > reproducible. > > However, to enable the data user to reconstitute the coordinates to an > accuracy comparable to the accuracy intended by the data creator, the data > creator shall specify the floating-point arithmetic precision used during the > preparation and validation of the compressed coordinates by setting the > interpolation variable’s computational_precision attribute to one of the > following values: Please understand that as a suggestion from my side - if you feel the 3 paragraphs are better I wouldn't feel strongly enough to call the CF police. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-873066613__;Iw!!G2kpM7uM-TzIFchu!mph7ZtWT4IPNt5iXN5MYqhLj7CW3nMR6nAZvfhMvYvYRthrYCW7AXtyR7K2x5NTMNooEr9vWDOU$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear Sylvain (@oceandatalab) Thank you very much for your proposed wording of the Computational Precision text, which I think is a sound way to formulate the meaning and usage of the `computational_precision` attribute. I like the detailed rationale you have provided and support having the `computational_precision` attribute of the interpolation variable mandatory. Possibly we could shorten the text slightly and still convey the message? Would the following possibly do the job? **8.3.8 Computational Precision** The accuracy of the reconstituted coordinates will mainly depend on the degree of subsampling, the choice of interpolation method and the choice of the floating-point arithmetic precision used in the interpolation method computations. The accuracy of the reconstituted coordinates may also depend on details of the interpolation method implementation and on the computer platform, meaning that the results of the coordinate reconstitution process may not be fully reproducible. However, to enable the data user to reconstitute the coordinates to a accuracy comparable to the accuracy intended by the data creator, the data creator shall specify the floating-point arithmetic precision used during the preparation and validation of the compressed coordinates by setting the interpolation variable’s `computational_precision `attribute to one of the following values: (table) "32": 32-bit floating-point arithmetic, comparable to the binary32 standard in [IEEE_754] "64": 64-bit floating-point arithmetic, comparable to the binary64 standard in [IEEE_754] As an example, a `computational_precision` = "64" would provide the guidance to the data user that using 64-bit floating-point arithmetic will reconstitute the coordinates with an accuracy comparable to the accuracy intended by the data creator. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-873017814__;Iw!!G2kpM7uM-TzIFchu!mfqlpwce8MY_Ko0Jwe3q7zB4yHaQ_q_SMgTIVH0HO93jdABT07IZuPql32A9HGlhcZSMtfZ2-KU$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
@AndersMS: yes I think replacing "sample/sampled" with "subsample/subsampled" would make the text more consistent. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-872180730__;Iw!!G2kpM7uM-TzIFchu!hmExYad1E2EtKOBPK4kRHaiCFUQtV6xN0mKWNJOIVto5CipFkc4srlxWqVd6kdMyo8HqVdXfM5s$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi, Here is a new take on the computational precision paragraph: 8.3.8 Computational Precision The accuracy of the reconstituted coordinates will depend on the degree of subsampling, the choice of interpolation method and the choice of the floating-point arithmetic precision used in the interpolation method computations. Implementation details of the interpolation methods and hardware can also have an impact on the accuracy of the reconstituted coordinates. The creator of the compressed dataset must check that the coordinates reconstituted using the interpolation parameters specified in the file have sufficient accuracy compared to the coordinates at full resolution. Although it may depend on the software and hardware used by the creator, the floating-point arithmetic precision used during this validation step must be specified in the `computational_precision` attribute of the interpolation method as an indication of potential floating-point precision issues during the interpolation computations. The `computational_precision` attribute is mandatory and accepts the following values: (table) "32": 32-bit floating-point arithmetic, comparable to the binary32 standard in [IEEE_754] "64": 64-bit floating-point arithmetic, comparable to the binary64 standard in [IEEE_754] For the coordinates reconstitution process, using a floating-point arithmetic precision matching or exceeding the precision specified by `computational_precision` is likely to produce results with an accuracy similar to what the creator obtained during the validation of the dataset, but it cannot be guaranteed due to the software/hardware factor. As an example, `computational_precision = "64"` would specify that, using the same software and hardware as the creator of the compressed dataset, sufficient accuracy could not be reached when using a floating-point precision lower than 64-bit floating-point arithmetic in the interpolation computations required to reconstitute the coordinates. **Bibliography** **References** [IEEE_754] [IEEE Standard for Floating-Point Arithmetic](https://urldefense.us/v3/__https://ieeexplore.ieee.org/stamp/stamp.jsp?tp==8766229=8766228__;!!G2kpM7uM-TzIFchu!h6g5VosoxJPZrjMnqKldnCSEYC-DjpPbYuBWm5Jd1jE2UXrJvx-8VuSHQj4VI5g_zrcX2-BLYGI$ ), in IEEE Std 754-2019 (Revision of IEEE 754-2008) , vol., no., pp.1-84, 22 July 2019, doi: 10.1109/IEEESTD.2019.8766229. --- Rationale: The accuracy of the interpolation methods depends not only on the choices made by the data producer (tie points density, area subdivisions, interpolation method parameters, etc...) but also on the software (programming language, libraries) and on the hardware (CPU/FPU) used by the data consumers. The data producers only know about their own software and hardware, so the computational_precision attribute can only mean that the data producer used this floating point precision when they validated these data using their implementation of the interpolation method, not that using this floating point precision on any software/hardware combination will produce exactly the same results. I think the computational_precision attribute can only be considered as a hint provided by the data producer regarding numerical issues they encountered when trying to reconstruct the target variables at their full resolution with their implementation of the interpolation method: if the computational_precision exceeds the precision of the data type (e.g. a "64" computational_precision used when interpolating a float variables), then users know that the data producer did not obtain satisfying results when using a lower precision, hence they should be wary of underflow/overflow errors when they interpolate these data. So computational_precision is more of an informational hint than a compulsory instruction given to the users (unless @erget 's CF police becomes a reality), and it is not a reproductibility guarantee either. Yet it is still a useful piece of information and no one except the data producer can provide it since you need access to the original data at their native resolution to make actual checks on the accuracy of the interpolation method. As the information cannot be derived from the content of the file it makes sense to require that data producers include this attribute systematically: the computational_precision should be mandatory. Sylvain -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-872178234__;Iw!!G2kpM7uM-TzIFchu!h6g5VosoxJPZrjMnqKldnCSEYC-DjpPbYuBWm5Jd1jE2UXrJvx-8VuSHQj4VI5g_zrcX6hbMb2s$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear All, Considering that we have now renamed the term _tie point interpolation dimension_ to _subsampled dimension_, should we possibly change the title **Lossy Compression by Coordinate Sampling** to **Lossy Compression by Coordinate Subsampling** and the replace the occurrences of sample/sampled in the text with subsample/subsampled? Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-872151596__;Iw!!G2kpM7uM-TzIFchu!mbL15pJsqDLSvUSrR8qOXHUG8lT1kAbisJUrSAsSPsLm6yt1yiL1zDzJJ_2j-MJAXmUDxRzC2Cc$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS. Daniel @erget et al., > Concerning terminology, following discussion in the group, these terms seem > good candidates: > At tie-point level: "subsampled dimension", "non-interpolated dimension" > At reconstituted level: "interpolated dimension", "non-interpolated dimension" Yes, that terminology seems clear and self-explanatory to me. Thanks for your patience and carefulness. > Actionees! I take on myself an action to review the rest of the proposal (Appendix and conformance document) this week. Best wishes Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-871248157__;Iw!!G2kpM7uM-TzIFchu!gsxrOv2ysX6OgZprPT0fXVbdQLwq41YWHk6IHGuWuMSd8cWalg1hnnCaipIGE1BuDEpqgqKegaM$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
# Terminology issues Dear @JonathanGregory et al. (@AndersMS @davidhassell @oceandatalab @ajelenak) Concerning terminology, following discussion in the group, these terms seem good candidates: - At tie-point level: "subsampled dimension", "non-interpolated dimension" - At reconstituted level: "interpolated dimension", "non-interpolated dimension" "non-interpolated dimension" is repeated because it is shared across the 2 domains. Would this be an improvement in your view? # Computational precision `computational_precision` This attribute should be mandatory for data producers to specify the precision in which interpolation should take place. This means that there should be no default value; the creator specifies it. It is up to the user to interpret that and using a precision that deviates from the recommendation would not prompt an arrest through the CF police but would mean that they might have deviations in the interpolated data. This should be clearly described so that the reader of the Conventions understands the potential impacts (no need to wax eloquent here though). # Actionees! - @AndersMS will update the draft with regards to the terminology issues - @oceandatalab will propose a paragraph regarding computational precision in this issue and @AndersMS will integrate it into the document after we've agreed it here -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-870574025__;Iw!!G2kpM7uM-TzIFchu!kGgEAKsB-Hu_FZ2RBoRAraz_VHPZzWIZkfL_i3P7du4GueaRSM_VIiKAhr43UwYDkN-Jy9I_Ilo$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory That's an interesting suggestion, than you. We will discuss it in the group tomorrow. Best regards, Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-869908486__;Iw!!G2kpM7uM-TzIFchu!ggPGhpuBz480onieCZa_aXbUzqh4Y5wyPkdbpBqOzzX5ePa4gg1v2v-oEpUxms52gjS1Xc9w0yc$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS In your proposed change 10, you used the word "uncompressed", and "compression" is in the title of this proposal. I think it would be clear to speak of a "compressed dimension" of the tie point variable corresponding to an "uncompressed dimension" of the data variable, or perhaps an "expanded dimension", with the other dimensions being non-interpolation/non-interpolated. Best wishes Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-869881313__;Iw!!G2kpM7uM-TzIFchu!mh2-78HKoZSQ6C2nTntMI_4tWUo6R164e4BTZpLCc0lfTDxCudNsh7THtxdD81P9kgb47wX-6Ps$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory Dear Jonathan, Thank you for the fedback. - yes, we had a sentence saying that the size of a tie point interpolation dimension must be less than or equal to the size of the corresponding interpolated dimension. I actually deleted it, since it is a consequence of other constraints, rather than a constraint of its own. But it makes sense to state it and I will re-introduce the sentence. - I like the the term interpolated dimension that we proposed- it has made it easier for me to read and write the text. And I am hesitant to introduce an additional term like "reduced" in the text. Note that it is "tie point interpolation dimension" against "interpolated dimension", so the difference is more than just the difference between interpolation and interpolated. The way I memorize it, is that a tie point interpolation dimension is available for interpolation, whereas the corresponding dimension in the target domain is, once it exists, an interpolated dimension. Non-interpolated dimensions are the same in both the tie point domain and the target domain and are never interpolated. I will discus the last point with the rest of the group when we meet tomorrow. Best regards, Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-869854024__;Iw!!G2kpM7uM-TzIFchu!hwly8L8ZiKCCZonttxQnMfwJRAIuTdv2lKzYGkqMsy5w00yvCY_kKTbLf7_0v0Ey_PDcdsoxaUA$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @AndersMS and colleagues Thanks very much for taking my comments so seriously and for the modifications and explanations. I agree with all these improvements, with two reservations: * Do you somewhere state that the size of a tie point interpolated dimension must be less than or equal to the size of the corresponding interpolation dimension? I suggested this sentence somewhere but you haven't included it there. Maybe it is somewhere else. It seems obvious but is nonetheless worth stating. * While I appreciate you want to relate things to interpolation, I would urge you to use a different word from "interpolated", because you're depending on a very attentive reader in sentences such as "A single interpolated dimension may be associated with multiple tie point interpolation dimensions." My suggestion of "reduced" is not necessarily a good one, but it is noticeably different from "interpolation". Also, "interpolated" doesn't seem quite right to me. You mean, it's _going to be_ interpolated. It hasn't _yet_ been interpolated, though. I will study the appendix and conformance document next week sometime. Best wishes Jonathan -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-868496769__;Iw!!G2kpM7uM-TzIFchu!nvcZ73XNc658MnzpjP4cELepjBxVYQELl3stBtDUgeaXN-Gc7RoHZqyLRF3KJ0Z2adxQkfUvwBE$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
I have removed the paragraph "The same interpolation variable may be multiply mapped " as proposed [here](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-867923556__;Iw!!G2kpM7uM-TzIFchu!g4PaZBgqp7kExa0P03mvftNK-smFaeuz_eCsQLj0uchx3wlnvO1ElV9rjDburlc6U42x9S1AeNA$ ). Commit 485d3b8 -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-868373525__;Iw!!G2kpM7uM-TzIFchu!g4PaZBgqp7kExa0P03mvftNK-smFaeuz_eCsQLj0uchx3wlnvO1ElV9rjDburlc6U42x1IzTqc4$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi Anders, > I believe the following paragraph from our chapter 8 is no longer relevant I do agree. David -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-868284609__;Iw!!G2kpM7uM-TzIFchu!m8DtGOLMpBPFoDLhsQvora71ZhOlJhJzpcoamq3qW4maN7eYSf_xgf_7yefhPuqtg-JYgm1Z1bk$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear All, I believe the following paragraph from our chapter 8 is no longer relevant, after we have moved all the dimension related attributes from the data variable to the interpolation variable. The tie point variables `lat` and `lon` spanning dimension `tp_dimension1` and tie point variable `time` spanning dimension `tp_dimension2` must have each their interpolation variable. Would you agree? Anders > The same interpolation variable may be multiply mapped from the different > sets of tie point coordinate variables. For instance, if tie point variables > `lat` and `lon` span dimension `tp_dimension1` and tie point variable `time` > spans dimension `tp_dimension2`, and all three are to interpolated according > to interpolation variable `linear`, then the *`coordinate_interpolation`** > attribute could be `lat: lon: linear time: linear`. In this case it is not > possible to simultaneously map all three tie point coordinate variables to > the linear interpolation variable because > they do not all span the same axes. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-867923556__;Iw!!G2kpM7uM-TzIFchu!lqCQUxWJImo1ZZAlsowZUMpdjpmCmMKIZ_g3JJfE9hTKT5ap_CzD8agl4uEV7LqWMYhfcMmlq7o$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi again @JonathanGregory Just to add the figures have not yet been updated, I think we will do this when all text changes have ben agreed. Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-867899070__;Iw!!G2kpM7uM-TzIFchu!ieUFiqNwDJQR0e553aE6GmouQuySHf3LsJDXBkPqdfgvMYiZPJXWD0MvqqNJ_7NQ4pQhRrk4NaE$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory We have progressed with preparing the replies to your proposals. Although there are still a couple of open points, we thought it would be useful to share what we already have. We have numbered your proposal as Proposed Change 1-16 and treated each of these separately below. For each of the Proposed Changes, you will find a reply to the proposed change as well as the related commit(s). We are still working on a reply to Proposed Change 15, the other replies are completed. We are still working on completing the corresponding document updates in the form of commits for the Proposed Change 1, 2, 8, 13 and 14, the other document commits are complete. We will notify you once all replies and document commits are complete. Best regards Anders **Proposed Change 1 – Combining the tie_point_dimensions and tie_point_indices attributes** > There is one point where I have a suggestion for changing the content of the > proposal, although probably you've already discussed this possibility. If I > understand correctly, you must always have both the tie_point_dimensions and > tie_point_indices attributes of the interpolation variable, and they must > refer to the same tie point dimensions. Therefore I think a simpler design, > easier for the both data-writer and data-reader to use, would combine these > two attributes into one attribute, whose contents would be > "interpolation_dimension: tie_point_interpolation_dimension > tie_point_index_variable [interpolation_zone_dimension] > [interpolation_dimension: ...]". **Reply to Proposed Change 1** We agree with combining the tie_point_dimensions and tie_point_indices attributes in a single attribute as you suggest, but propose to put the tie_point_index_variable before the dimensions: ` interpolated_dimension: tie_point_index_variable tie_point_interpolation_dimension [interpolation_subarea_dimension] [interpolated_dimension: ...].` **Commit(s) related to Proposed Change 1** Commit(s) being prepared **Proposed Change 2 – Naming combined tie_point_dimensions and tie_point_indices to tie_points and existing tie_points to interpolation** > Also, I have some suggestions for naming: > • If you adopt my suggestion for a single attribute to replace > tie_point_dimensions and tie_point_indices, an obvious name for it would be > tie_points. You've used that name for the attribute of the data variable. > However, I would suggest that the attribute of the data variable could > equally well be called interpolation, since it names the interpolation > variable, and signals that interpolation is to be used. > **Reply to Proposed Change 2** We propose renaming the `tie_points `attribute of the data variable to `coordinate_interpolation `as this makes the name more descriptive. We propose to use the name `tie_point_mapping `for the attribute of the interpolation variable resulting from combining the `tie_point_dimensions `and `tie_point_indices `attributes. We favor this compared to `tie_points`, as the attribute does not contain or reference tie point coordinates variables. **Commit(s) related to Proposed Change 2** ea5268b f8cd983 Further commit(s) being prepared **Proposed Change 3 - Rename term "tie point interpolation dimension" to e.g. "tie point reduced dimension"** > • Your terminology has "tie point interpolation dimension" and > "interpolation dimension", but the former is not a special case of the > latter. That could be confusing, in the same way that (unfortunately) in CF > terminology an auxiliary coordinate variable is not a special kind of > coordinate variable. I suggest you rename "tie point interpolation dimension" > as e.g. "tie point reduced dimension" to avoid this misunderstanding. **Reply to Proposed Change 3** For the coordinate dimensions in the target domain, we suggest replacing the terms “interpolating dimension” and “non-interpolating dimension” with the terms “interpolated dimension” and “non-interpolated dimension”. Then we can maintain the term "tie point interpolation dimension” for the tie points. **Commit(s) related to Proposed Change 3** 7ab0c4f **Proposed Change 4 - Rename term "tie point variable" to "tie point coordinate variable"** > • A similar possible confusion is that a tie point index variable is not > a special kind of tie point variable. To avoid this confusion and add > clarity, I suggest you could rename "tie point variable" as "tie point > coordinate variable". **Reply to Proposed Change 4** We agree to rename the term "tie point variable" to "tie point coordinate variable". **Commit(s) related to Proposed Change 4** a6d37b4 **Proposed Change 5 – Renaming of term "interpolation area"** > • The terms "interpolation zone" and "interpolation area" are unhelpful > because it's not obvious from the words which one is bigger, so it's hard to > remember. If you stick with "zone" for the small one, for area it would be > better to
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi @taylor13, Your point is valid. I guess there would be two alternative solutions: 1. We remove _'or exceed'_ from the sentence _'For the coordinate reconstitution process, the floating-point arithmetic precision should match or exceed the precision specified by computational_precision attribute.'_ That would be the closets we can get to _reproducible and of predictable accuracy_ . It would leave no choice to be made by the users uncompressing the coordinates and all users would get the same results. 2. We keep _'or exceed'_ and adapt the wording along the lines of what you propose. Personally, I think that ensuring _that the results of the coordinate reconstitution process are reproducible and of predictable accuracy_ is very valuable, and my preference would be option 1. I believe that if a data creator has judged that `computational_precision = "32"` is sufficient and appropriate for the data product, it would typically also imply that there is only a limited scope for real improvements on the user side by going to 64-bit floating-point arithmetic. That would also support option 1. What do you think? Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-867659527__;Iw!!G2kpM7uM-TzIFchu!jqrMRPJgMCXBzWMK_C5wnS042F_1saeuKGKITvXgUfZ3CMnaNQLN8SlQRGCB0hCUss9SEdEuAHw$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Editorial suggestion: In the statement, ``` To ensure that the results of the coordinate reconstitution process are reproducible and of predictable accuracy, the creator of the compressed dataset may specify the floating-point arithmetic precision to be used in the interpolation method computations by ``` I think we should replace "reproducible and of predictable accuracy" with "reproducible with sufficient accuracy" (or something similar). The accuracy might for some algorithms be improved using a higher precision than specified by the `computational_precision` attribute, but such higher accuracy might be considered unwarranted for a given dataset. So the accuracy really isn't totally determined by the attribute (i.e., it isn't predictable) because the user is free to perform the calculation at a higher precision. (Hope this is correct and understandable.) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-866366247__;Iw!!G2kpM7uM-TzIFchu!giJ39YFIP5U24kb7fTEtz4MaVEwFlMKsiQC5-_rMLb7Sjsi3CDwVge-gaJ2pZbOSCdI0OgYfIS0$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
I agree. This specification of precision is good. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-865650133__;Iw!!G2kpM7uM-TzIFchu!giymN7yRurADlC5Vrrue5CAxgthC8bxxu3ExoqOWjy1DEvitSYD4K5V2HiSuYTayf0_92LMeOls$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
That looks good to me, Anders. The word _computation_ is good. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-863294118__;Iw!!G2kpM7uM-TzIFchu!hfUfW2OvzRuXshuEaWS1FWTZtwKpAQbV5PQpKuIbCJg5x4Ksi4qW6RPBISZ_rA-G2K1EXIJ9ucw$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Good idea David. Should we perhaps use _computation_ instead of _calculation_ to match the attribute name? Here I have updated the first two paragraphs and added an example: **8.3.8 Computational Precision** "The accuracy of the reconstituted coordinates will depend on the degree of subsampling, the choice of interpolation method and the choice of the floating-point arithmetic precision used in the interpolation method computations. To ensure that the results of the coordinate reconstitution process are reproducible and of predictable accuracy, the creator of the compressed dataset may specify the floating-point arithmetic precision to be used in the interpolation method computations by setting the interpolation variable’s `computational_precision` attribute to one of the following values: (table) "32": 32-bit floating-point arithmetic (default), comparable to the binary32 standard in [IEEE_754] "64": 64-bit floating-point arithmetic, comparable to the binary64 standard in [IEEE_754] For the coordinate reconstitution process, the floating-point arithmetic precision should match or exceed the precision specified by `computational_precision `attribute, or match or exceed 32-bit floating-point arithmetic if the `computational_precision` attribute has not been set. As an example, `computational_precision = "64"` would specify that the floating-point arithmetic precision should match or exceed 64-bit floating-point arithmetic. **Bibliography** **References** [IEEE_754] "IEEE Standard for Floating-Point Arithmetic," in IEEE Std 754-2019 (Revision of IEEE 754-2008) , vol., no., pp.1-84, 22 July 2019, doi: 10.1109/IEEESTD.2019.8766229. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-863211066__;Iw!!G2kpM7uM-TzIFchu!i0wYeRWjS2nDJDLdSy5CWBpzLE2atsGcqTyGc-nvFovozpXj1keEpoxqVaUjWoApPKX5PicW2FY$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Thank you, Anders. I very happy with this. A minor suggestion - perhaps change: _"...may specify the floating-point arithmetic precision by setting ..."_ to _... may specify the floating-point arithmetic precision to be used in the interpolation calculations by setting ..._ just to be extra clear which precision is being specified. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-863127657__;Iw!!G2kpM7uM-TzIFchu!jljmsixgYLhEB9G4kO0qXgYccxBApqbZjnZFevZLMetOXEX3NlJr5s25p2iFphDIhmtmE5rh3x4$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear All, Following a discussion yesterday in the team behind the proposal, we propose the 'computational_precision` attribute to be optional. Here is the proposed text, which now has a reference to [IEEE Std 754]. Feel free to comment. Anders **8.3.8 Computational Precision** The accuracy of the reconstituted coordinates will depend on the degree of subsampling, the choice of interpolation method and the choice of the floating-point arithmetic precision with which the interpolation method is applied. To ensure that the results of the coordinate reconstitution process are reproducible and of predictable accuracy, the creator of the compressed dataset may specify the floating-point arithmetic precision by setting the interpolation variable’s `computational_precision` attribute to one of the following values: (table) 32: 32-bit floating-point arithmetic (default), comparable to the binary32 standard in [IEEE_754] 64: 64-bit floating-point arithmetic, comparable to the binary64 standard in [IEEE_754] For the coordinate reconstitution process, the floating-point arithmetic precision should match or exceed the precision specified by `computational_precision` attribute, or match or exceed 32-bit floating-point arithmetic if the `computational_precision` attribute has not been set. **Bibliography** **References** [IEEE_754] "IEEE Standard for Floating-Point Arithmetic," in IEEE Std 754-2019 (Revision of IEEE 754-2008) , vol., no., pp.1-84, 22 July 2019, doi: 10.1109/IEEESTD.2019.8766229. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-863111008__;Iw!!G2kpM7uM-TzIFchu!hEeNCeW3ZDm51VrD-2Me9m0x4OMfhYYcZUZNnJciIwiZkNlfJ709ZkXEkaxKpCvVmt2WhcBFHmI$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear @JonathanGregory Thank you very much for your rich and detailed comments and suggestions, very appreciated. The team behind the proposal met today and discussed all the points you raised. We have prepared or are in the process of preparing replies to each of the points. However, before sharing these here, we would like to update the proposal text accordingly via pull requests, in order to see if the changes have other effects on the overall proposal, which we have not yet identified. Best regards, Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-861655992__;Iw!!G2kpM7uM-TzIFchu!gAUOCVb0XZQAleYhiAeU1_B19z0hEAyChd4dP59T9-_dg9vn2uLN4FOjKPlNIHYi8GOLDIqi3ak$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Dear all I've studied the text of proposed changes to Sect 8, as someone not at all involved in writing it or using these kinds of technique. (It's easier to read the files in [Daniel's repo](https://urldefense.us/v3/__https://github.com/erget/cf-conventions/blob/lossy-compression-through-coordinate-sampling/ch08.adoc__;!!G2kpM7uM-TzIFchu!iGX9_YVR65h2_BfQd8IiUVBizbXiSyxxYn9oae_IQantYdaz7fVqGPaJVaXHsSUjt-QNBr8-eT4$ ) than [the pull request](https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/pull/326/files?short_path=ebcafde*diff-ebcafde998cd56873e594e76a10c8541235a4fed3d4664a9c7733805bff39a4c__;Iw!!G2kpM7uM-TzIFchu!iGX9_YVR65h2_BfQd8IiUVBizbXiSyxxYn9oae_IQantYdaz7fVqGPaJVaXHsSUjt-QNh6J1SNs$ ) in order to see the diagrams in place.) I think it all makes sense. It's well-designed and consistent with the rest of CF. Thanks for working it out so thoughtfully and carefully. The diagrams are very good as well. I have not yet reviewed Appendix J or the conformance document. I'm going to be on leave next week, so I thought I'd contribute just this part before going. Best wishes Jonathan There is one point where I have a suggestion for changing the content of the proposal, although probably you've already discussed this possibility. If I understand correctly, you must always have both the `tie_point_dimensions` and `tie_point_indices` attributes of the interpolation variable, and they must refer to the same tie point dimensions. Therefore I think a simpler design, easier for the both data-writer and data-reader to use, would combine these two attributes into one attribute, whose contents would be "*interpolation_dimension*`:` *tie_point_interpolation_dimension* *tie_point_index_variable* [*interpolation_zone_dimension*] [*interpolation_dimension*`:` ...]". Also, I have some suggestions for naming: * If you adopt my suggestion for a single attribute to replace `tie_point_dimensions` and `tie_point_indices`, an obvious name for it would be `tie_points`. You've used that name for the attribute of the data variable. However, I would suggest that the attribute of the data variable could equally well be called `interpolation`, since it names the interpolation variable, and signals that interpolation is to be used. * Your terminology has "tie point interpolation dimension" and "interpolation dimension", but the former is not a special case of the latter. That could be confusing, in the same way that (unfortunately) in CF terminology an auxiliary coordinate variable is not a special kind of coordinate variable. I suggest you rename "tie point interpolation dimension" as e.g. "tie point reduced dimension" to avoid this misunderstanding. * A similar possible confusion is that a tie point index variable is not a special kind of tie point variable. To avoid this confusion and add clarity, I suggest you could rename "tie point variable" as "tie point coordinate variable". * The terms "interpolation zone" and "interpolation area" are unhelpful because it's not obvious from the words which one is bigger, so it's hard to remember. If you stick with "zone" for the small one, for area it would be better to use something which is more obviously much bigger, such as "province" or "realm"! Or perhaps you could use "division" or "department", since the defining characteristic is the discontinuity. In the first paragraph of Sect 8 we distinguish three methods of reduction of datset size. I would suggest minor clarifications: > There are three methods for reducing dataset size: packing, lossless > compression, and lossy compression. By packing we mean altering the data in a > way that reduces its precision **(but has no other effect on accuracy)**. By > lossless compression we mean techniques that store the data more efficiently > and result in no **loss of precision or accuracy**. By lossy compression we > mean techniques that store the data more efficiently **and retain its > precision** but result in some loss in accuracy. Then I think we could start a new paragraph with "Lossless compression only works in certain circumstances ...". By the way, isn't it the case that HDF supports per-variable gzipping? That wasn't available in the old netCDF data format for which this section was first written, so it's not mentioned, but perhaps it should be now. There are a few points where I found the text of Sect 8.3 possibly unclear or difficult to follow: * "This form of compression may also be used on a domain variable with the same effect." I think this is an unclear addition. If I understand you correctly, insead of this final sentence you could begin the paragraph with "For some applications the coordinates of a data variable or a domain variable can require considerably more storage than the data in its domain." * Tie Point Dimensions Attribute. If you adopt my suggestion above, this subsection would change its name to "Tie points attribute". It
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
I have a preference for "optional" because I suspect in most cases 32-bit will be sufficient and this would relieve data writers from including this attribute. There may be good reasons for making it mandatory; what are they? Not sure about this, but I think "should" rather than "shall" is better. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-858832756__;Iw!!G2kpM7uM-TzIFchu!jApB4nWetYeIEOsaRN6b4LlhECtpDlvJcNNJZOfQXrHvOSSzsAvJlQJsdmFHETqxe0SC-JNy-vI$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi @taylor13 and @davidhassell, Regarding the `computational_precision` attribute, it appears that we currently have two proposals: Either an optional attribute with a default value or a mandatory attribute. I have written two versions of the new section 8.3.8, one for each of the two proposals. I hope that will help deciding! Anders **Optional attribute version:** **8.3.8 Computational Precision** The accuracy of the reconstituted coordinates will depend on the degree of subsampling, the choice of interpolation method and the choice of the floating-point arithmetic precision with which the interpolation method is applied. To ensure that the results of the coordinate reconstitution process are reproducible and of predictable accuracy, the creator of the compressed dataset may specify the floating-point arithmetic precision by setting the interpolation variable’s `computational_precision` attribute to one of the following values: (table) 32: 32-bit floating-point arithmetic (default) 64: 64-bit floating-point arithmetic For the coordinate reconstitution process, the floating-point arithmetic precision should (or shall?) match or exceed the precision specified by `computational_precision` attribute, or match or exceed 32-bit floating-point arithmetic if the `computational_precision` attribute has not been set. **Mandatory attribute version:** **8.3.8 Computational Precision** The accuracy of the reconstituted coordinates will depend on the degree of subsampling, the choice of interpolation method and the choice of the floating-point arithmetic precision with which the interpolation method is applied. To ensure that the results of the coordinate reconstitution process are reproducible and of predictable accuracy, the creator of the compressed dataset must specify the floating-point arithmetic precision by setting the interpolation variable’s `computational_precision` attribute to one of the following values: (table) "32": 32-bit floating-point arithmetic "64": 64-bit floating-point arithmetic For the coordinate reconstitution process, the floating-point arithmetic precision should (or shall?) match or exceed the precision specified by `computational_precision` attribute. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-858807540__;Iw!!G2kpM7uM-TzIFchu!gtrT1ATM4i1MruwBLI8QC6A-7qNj82V3hO7OSg6QSAd5_un8FRNyoZP1kDDwPaLTAqErhmDq_7E$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Wouldn't the statement be correct as is (perhaps rewritten slightly; see below), if we indicated that if the computational_precision attribute is *not* specified, a default precision of "32" should be assumed? I would think that almost always the default precision would suffice, so for most data writers, it would be simpler if we didn't require this attribute. (But I don't feel strongly about this.) Not sure how to word this precisely. Perhaps: ``` The attributes and default values defined for the interpolation formula and its inputs ensure that the results of the coordinate reconstitution process are reproducible and of predictable accuracy. ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-857929639__;Iw!!G2kpM7uM-TzIFchu!niPp3UzLiwqz2rv9HvdKZAn4OXeMKihtu7TJYlprdCotRL8WMbfWfzusbvus5Hf5qK35YyVuwYw$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi David, Yes, I would be happy to update the PR. However, I still have one concern regarding the `computational_precision `attribute. In the introduction to _Lossy Compression by Coordinate Sampling_ in chapter 8, I am planning to change the last sentence from > The creator of the compressed dataset can control the accuracy of the > reconstituted coordinates through the degree of subsampling and the choice of > interpolation method, see [Appendix J]. to > The creator of the compressed dataset can control the accuracy of the > reconstituted coordinates through the degree of subsampling, the choice of > interpolation method (see [Appendix J]) and the choice of computational > precision (see section X). where section X will be a new short section in chapter 8 describing the `computational_precision ` attribute. Recalling that we also write in the introduction to _Lossy Compression by Coordinate Sampling_ in chapter 8 that > The metadata that define the interpolation formula and its inputs are > complete, so that the results of the coordinate reconstitution process are > well defined and of a predictable accuracy. I think it would be more consistent if we make the `computational_precision ` attribute mandatory and not optional. Otherwise the accuracy would not be predictable. Would that be agreeable? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-857728390__;Iw!!G2kpM7uM-TzIFchu!hSbdxVTIlDKpLsYnTw-Q1SwLnvmsOESRxDa6aUL2bAv8fxL9g51pmu42LJm3ERsAHFM5KuqLaaE$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi Anders - thanks, it sounds like we're currently in agreement - do you want to update the PR? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-856086921__;Iw!!G2kpM7uM-TzIFchu!jUOZpy6qAxegitsY3lqBLXpzQsfmvJ15uaf1EtLMUtaiGUjTePBRpwsB7hC4SJ6F2hF_qCqhbLM$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi David, Fine, I take your advice regarding not having a default value. That is probably also simpler - one rule less. Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-855860002__;Iw!!G2kpM7uM-TzIFchu!gdgIumfFWPPmS3_NS8LHSF6OHRqYnN3zsPCAtogRO0ZBoA09Sids1s36jaGMtp2jhtcZ7bBDCb0$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi Anders, > "The floating-point arithmetic precision should match or exceed the precision > specified by computational_precision attribute. The allowed values of > computational_precision attribute are: > > (table) "32": 32-bit floating-point arithmetic "64": 64-bit floating-point arithmetic This is good for me. > If the computational_precision attribute has not been set, then the default > value "32" applies." > > That would ensure that we can assume a minimum precision on the user side, > which would be important. Practically speaking, high level languages that > support 16-bit floating-point variables, typically use 32-bit floating-point > arithmetic for the 16-bit floating-point variables (CPU design). I'm not so sure about having a default value. In the absence of guidance from the creator, I'd probably prefer that the user is free to use whatever precision they would like. Thanks, David -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-854048598__;Iw!!G2kpM7uM-TzIFchu!mDmmC51KHcfr7XcIbx2_Ie-l2zMqZ4H0Ef03YZxmsD4h94ZeLOeIvexmL2EgCVG6OFbVA4TeD7U$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
@taylor13 by the way I'm still on the prowl for a moderator for this discussion. As I see you've taken an interest, would you be willing to take on that role? I'd be able to do it as well, but as I've been involved in this proposal for quite some time it would be nice to have a fresh set of eyes on it. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-842144571__;Iw!!G2kpM7uM-TzIFchu!mwENZvMoPtB6RLngGMcvr69cOnri-3_VVGCou7tkCFhyQ9gs8qqduu9sMBfOnfMFArnWigEf1FI$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Leaving out "base-2" is fine. Shortening the description further as you suggest would also be fine with me. I am wondering if we could change the wording to: "The floating-point arithmetic precision should match or exceed the precision specified by `computational_precision `attribute. The allowed values of `computational_precision ` attribute are: (table) `"32":` 32-bit floating-point arithmetic `"64"`: 64-bit floating-point arithmetic If the `computational_precision `attribute has not been set, then the default value "32" applies." That would ensure that we can assume a minimum precision on the user side, which would be important. Practically speaking, high level languages that support 16-bit floating-point variables, typically use 32-bit floating-point arithmetic for the 16-bit floating-point variables (CPU design). -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-842138566__;Iw!!G2kpM7uM-TzIFchu!gfbyhFC3RCWGePZYqKGDBp0A08A-7cKwm-xNyE1iOpYdCukBwdL1ehtXly3L8hNRpq9rvz-RKAU$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
looks good to me. Can we omit "base-2" from the descriptions, or is that essential? Might even reduce description to, for example: ``` "32": 32-bit floating-point arithmetic ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-841835486__;Iw!!G2kpM7uM-TzIFchu!lVIZXSYXrkaaMhc4KBdhNzimWNdsjaDSDv2h5N4BV-MP2UjwTsWe8e_ko9rQC_jCDkVn9GvbP2g$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi @taylor13 and @davidhassell, I am not fully up to date on the data types, but following the links that David sent, it appears that decimal64 is a base-10 floating-point number representation that _is intended for applications where it is necessary to emulate decimal rounding exactly, such as financial and tax computations._ I think we can disregard that for now. binary32 and binary64 are the new official IEEE 754 names for what used to be called single- and double-precision floating-point numbers respectively, and is what most of us are familiar with. I would suggest that we do not require a specific floating-point arithmetic standard to be used, but more a level of precision. If we adopt the naming convention proposed by David, it could look like: By default, the user may use any floating-point arithmetic precision they like for the interpolation calculations. If the `computational_precision `attribute has been set then the precision should match or exceed the precision specified by `computational_precision `attribute. The allowed values of `computational_precision ` attribute are: (table) `"32":` 32-bit base-2 floating-point arithmetic, such as IEEE 754 binary32 or equivalent `"64"`: 64-bit base-2 floating-point arithmetic, such as IEEE 754 binary64 or equivalent I think that would archive what we are after, while leaving the implementers the freedom to use what their programming language and computing platform offers. What do you think? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-841811319__;Iw!!G2kpM7uM-TzIFchu!kJqTO5JRkCbffwObaYvfMMczn6itdbv8uviCjgc6I6zF97V87xaKVldzmU2HHfvdZ9sO9MQH2ZE$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
I don't understand the difference between decimal64 and binary64 or what they precisely mean. If these terms specify things beyond precision, it's probably not appropriate to use them here, so I would support defining our own vocabulary, which would not confuse precision with anything else. And I too would favor (or favour) A over B. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-841281449__;Iw!!G2kpM7uM-TzIFchu!m7RcYDJ_xUXrEr1qtoI_fCzuWHJrp0aQcetxYa9pS8LB5GghpuKsZyryxb2M98W74Fh0BldDDLM$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Thanks, @taylor13 and @AndersMS, I, too, would favour A (_Using the scheme proposed above, requiring the data creator to set the computational_precision accordingly._). I'm starting to think that the we need to be clear about `"decimal64"` (or 32, 128, etc.). I'm fairly sure that we only want to specify a precision, rather than also insisting/implying that the user should use [decimal64 floating-point](https://urldefense.us/v3/__https://en.wikipedia.org/wiki/Decimal64_floating-point_format__;!!G2kpM7uM-TzIFchu!lcE7w0fiRC1zhKFwOZuysOVeBDKIFjtHeow7J8Mk4RP8ARM0PsRnnarG_VyA20h1IKdMC3GQYJE$ ) format numbers in their calculations. The same issue would arise with `"binary64"`, although I suspect that most code would use [double precision floating-point](https://urldefense.us/v3/__https://en.wikipedia.org/wiki/Double-precision_floating-point_format__;!!G2kpM7uM-TzIFchu!lcE7w0fiRC1zhKFwOZuysOVeBDKIFjtHeow7J8Mk4RP8ARM0PsRnnarG_VyA20h1IKdMB4D0jVg$ ) by default. Could the answer to be to define our own vocabulary of `"16"`, `"32",` `"64",` and `"128"`? Or am I over complicating things? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-841186599__;Iw!!G2kpM7uM-TzIFchu!lcE7w0fiRC1zhKFwOZuysOVeBDKIFjtHeow7J8Mk4RP8ARM0PsRnnarG_VyA20h1IKdMy57_c-4$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Thank you @taylor13 for the proposals and @davidhassell for the implementation details. I fully agree with your point 1, 2 and 3. There is possibly one situation that might need attention. If the coordinates subject to compression are stored in decimal64, typically we would require the computations to be decimal64 too, rater than decimal32. We could deal with that either by: A. Using the scheme proposed above, requiring the data creator to set the `computational_precision` accordingly. B. Requiring that the interpolation calculation are never carried out at a lower precision than that of the coordinates subject to compression, even if the `computational_precision` is not set. Probably A would be the cleanest, what do you think? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-840817684__;Iw!!G2kpM7uM-TzIFchu!j-pO09g70M-pQckNVKXy6kW6_dkAhOtnQbeVFsFkbJ0RkqiZV1xGBP2CQpyiOHJEaF4MtDrBfkA$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
yes, ``calculational_precision`` was a mistake; I prefer ``computational_precision``. Also I'd be happy with not referring to an external standard, and for now, just suggesting that two values, "decimal32" and "decimal64", are supported, unless someone thinks others are needed at this time. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-840687587__;Iw!!G2kpM7uM-TzIFchu!mMqPZXX-ORN68YMGfOYD2JcwfTu8Cjj3Pr99bWCImzH-psB_8t2NqS3uL0bxrb7e3VuxsHkjf9Y$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi @taylor13, 1: I agree that higher precisions should be allowed. A modified description (which could do with some rewording, but the intent is clear for now, I hope): * By default, the user may use any precision they like for the interpolation calculations. If the `computational_precision` attribute has been set then the precision should match or exceed the precision specified by `computational_precision` attribute. \<_some text about allowed values and their interpretation_\>. 2: `computational_precision` is indeed better. You mention "calculational_precision" in 3 - was that intentional? That term is also OK for me. 3: A controlled vocabulary is certainly clearer than my original proposal, both in term of defining the concept and the encoding, and the IEEE standard does indeed provide what we need. I wonder if it might be good to define the (subset) of [IEEE terms]( https://urldefense.us/v3/__https://en.wikipedia.org/wiki/IEEE_754*Basic_and_interchange_formats__;Iw!!G2kpM7uM-TzIFchu!kDvS2uzxTpXbi_SvRkMj7jpAp49Z57OgopLWHCcfl1zElW21_PdY7y2wrx2IyIgP299cXTZnO8I$ ) ourselves in a table (I'm reminded of https://urldefense.us/v3/__https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html*table-supported-units__;Iw!!G2kpM7uM-TzIFchu!kDvS2uzxTpXbi_SvRkMj7jpAp49Z57OgopLWHCcfl1zElW21_PdY7y2wrx2IyIgP299cre8ccg4$ ) rather than relying on the contents of the external standard, to avoid the potential governance issues we always have when standards outside of CF's influence are brought in. Would the "binary" terms be valid, as well as the "decimal" ones? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-840673155__;Iw!!G2kpM7uM-TzIFchu!kDvS2uzxTpXbi_SvRkMj7jpAp49Z57OgopLWHCcfl1zElW21_PdY7y2wrx2IyIgP299c5HVd6nE$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Thanks @AndersMS for the care taken to address my concern, and thanks @davidhassell for the proposed revision. A few minor comments: 1. I wonder if users could be given the freedom to do their interpolation at a _higher_ precision than specified by ``interpolation_precision``. I would hate to think that the interpolation method would be degraded by doing so. I suggest, therefore, replacing "the precision should match" with "the precision should match or exceed" or something similar. Also, a comma should follow the introductory clause, "if the interpolation_precision attribute has been set to a numerical value" and typo in "calculatins" should be corrected. 2. I think we should avoid using a specified numerical value in a format that is language dependent. Rather I would prefer following the IEEE 753 standard identifiers of precision. Also, I think "interpolation_precision" might imply something about the accuracy of the interpolation method. To avoid this confusion, perhaps the attribute could be named "computational_precision". I therefore propose the following alternative: - By default, the user may use any precision they like for the interpolation calculations, but if accuracy requires a certain precision, the ``calculational_precision`` attribute should be defined and set to one of the names defined by the IEEE 753 technical standard for floating point arithmetic (e.g., "decimal32", "decimal64", "decimal128"). If the ``calculational_precision`` attribute has been defined, all interpolation calculations should be executed at the specified precision (or higher). In the example, then, "0D" would be replaced by "decimal64". -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-840630784__;Iw!!G2kpM7uM-TzIFchu!n9ZZMaAjDE3QkZuI-IWSSnKdrAmNMu9K-4vsMU7dHGO_DbRsV-5IOghOxvuKwTPeAGuQLeUVQ_U$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
For convenience, here is the proposal for specifying the precision to be used for the interpolation calculations (slightly robustified): * By default, the user may use any precision they like for the interpolation calculatins, but if the `interpolation_precision` attribute has been set to a numerical value then the precision should match the precision of the given numerical value. ``` // Interpolation variable with NO 'interpolation_precision' attribute // => the creator is saying "you can use whatever precision you like when uncompressing" char interp ; interp:interpolation_name = "bi_linear" ; ``` ``` // Interpolation variable with 'interpolation_precision' attribute // => the creator is saying "you must use the precision I have specified when uncompressing" char interp ; interp:interpolation_name = "bi_linear" ; interp:interpolation_precision = 0D ; // use double precision when uncompressing ``` Do you think that this might work, @taylor13? Thanks, David -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-840405409__;Iw!!G2kpM7uM-TzIFchu!l4OnjJ0AHMwIChBEfZsXSJWgqU4u2BZQ7UwoFFHx-SATthDaEfz1kuUgszf5AVJ0VtyYia318y0$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hi @taylor13 Thank you very much or your [comments](https://urldefense.us/v3/__https://github.com/cf-convention/discuss/issues/37*issuecomment-832780564__;Iw!!G2kpM7uM-TzIFchu!lLfxA-1u7BwCI_w6J1bYPrIbNQuDCLnaPErHAUz5vBwAAZn2Z7S69d6LTvuHUjDOgIwCD7NGM8M$ ). We did have a flaw or a weakness in the algorithm, which we have corrected following your comments. To briefly explain: the methods of the proposal stores coordinates at a set of tie points, from which the coordinates in the target domain may then be reconstituted by interpolation. The source of the problem was the computation of the distance squared between two such tie points. The distance will never be zero and could for example be in the order of a few kilometers. As the line between the two tie points forms a right triangle with two other lines of known length, the fastest way to compute the distance squared is to use Pythagoras's theorem. However, as the two other sides both of a length significantly larger than the one we wish to calculate, the result was very sensitive to rounding in 32-bit floating-point calculations and occasionally returned zero. We have now changed the algorithm to compute the distance squared as (x*x + y*y + z*z), where (x, y, z) is the vector between the two tie points. This expression does not have the weakness explained above and has now been tested to work well. In terms of accuracy of how well the method reconstitutes the original coordinates, the change improved the performance of the internal calculations being calculated in 32-bit floating-point. However, still with errors a couple of times larger than when using 64-bit floating-point calculations. I would therefore support the [proposal ](https://urldefense.us/v3/__https://github.com/cf-convention/discuss/issues/37*issuecomment-832816324__;Iw!!G2kpM7uM-TzIFchu!lLfxA-1u7BwCI_w6J1bYPrIbNQuDCLnaPErHAUz5vBwAAZn2Z7S69d6LTvuHUjDOgIwC8Ualx30$ ) put forward by @davidhassell. The proposal avoids setting a general rule, which, as you point out, may not cover all cases. It permits setting a requirement when needed to reconstitute data with the accuracy intended by the data creator. Once again, thank you very much for your comments – further comments from your side on the proposal would be highly welcome! Cheers Anders -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-835837958__;Iw!!G2kpM7uM-TzIFchu!lLfxA-1u7BwCI_w6J1bYPrIbNQuDCLnaPErHAUz5vBwAAZn2Z7S69d6LTvuHUjDOgIwC3o24DvM$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
Re: [CF-metadata] [cf-convention/cf-conventions] Lossy Compression by Coordinate Sampling (#327)
Hello @taylor13, @AndersMS, It might be better to continue the conversion over at #37 on the precision of interpolation calculations (the comment thread starting at https://urldefense.us/v3/__https://github.com/cf-convention/discuss/issues/37*issuecomment-832142697__;Iw!!G2kpM7uM-TzIFchu!h6Yhthlz8NCEbPVKifR1ickNq6w3lTlXz0wbwrPILFrMofXCkrd5R_O1DWxIVzr03UpsDLldLpg$ ) here in this issue, as this is the main place now for discussing the PR containing the details of this proposal, of which this precision question is one. I hope that's alright, thanks, David -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://urldefense.us/v3/__https://github.com/cf-convention/cf-conventions/issues/327*issuecomment-834364246__;Iw!!G2kpM7uM-TzIFchu!h6Yhthlz8NCEbPVKifR1ickNq6w3lTlXz0wbwrPILFrMofXCkrd5R_O1DWxIVzr03UpsnaGvAIA$ This list forwards relevant notifications from Github. It is distinct from cf-metad...@cgd.ucar.edu, although if you do nothing, a subscription to the UCAR list will result in a subscription to this list. To unsubscribe from this list only, send a message to cf-metadata-unsubscribe-requ...@listserv.llnl.gov.
