Hi, Your analysis is interesting but about timestamp. My proposal is about adding nanoseconds support on time logical type. As described in AVRO-4043 [1], the maximum value of time is 8.64E13. This value doesn't exceeded the upper range value 2^53-1 recommended for common interoperability with IEEE 754 floating point representation.
[1] https://issues.apache.org/jira/browse/AVRO-4043 Regards Le lun. 6 janv. 2025 à 11:07, Clemens Vasters <cleme...@microsoft.com.invalid> a écrit : > I'd like to point out that nanoseconds timestamps are an area where the > JSON encoding is acutely broken as it stands. > > 1) Per RFC 8259 Section 6 [1], integer numbers can only be relied on in > the range -2^53+1 to 2^53-1, which is incompatible with the Avro "long" > type. Integers outside the range MUST be represented as strings in JSON, > which is often overlooked, but which Avro MUST NOT overlook as a general > purpose serialization framework. > 2) the "now" timestamp in nanoseconds takes up 61 bits and can thus not be > reliably represented in a JSON number per spec. > > Since epoch values don't include TZ info they are generally also quite > terrible for interop, which is the supposed goal of the JSON encoding, a > future release should encode timestamps and dates in RFC3339 format, at the > very least for the JSON encoding, as proposed in AVRO-3986 [2][3], which > also addresses a range of other interop problems. > > > [1] https://www.rfc-editor.org/rfc/rfc8259#section-6 > [2] https://issues.apache.org/jira/browse/AVRO-3986 > [3] avrotize/avrojson.md at master · clemensv/avrotize · GitHub< > https://github.com/clemensv/avrotize/blob/master/avrojson.md> > > ________________________________ > Von: glywk <glywk.cont...@gmail.com> > Gesendet: Samstag, 4. Januar 2025 17:42 > An: dev@avro.apache.org <dev@avro.apache.org> > Betreff: Add support of time logical type with nanoseconds precision > > [Sie erhalten nicht häufig E-Mails von glywk.cont...@gmail.com. Weitere > Informationen, warum dies wichtig ist, finden Sie unter > https://aka.ms/LearnAboutSenderIdentification ] > > Hi, > > Since release 1.12.0 Avro specification allows to represent *timestamps* > and *local timestamps* with nanoseconds precision. > > I would propose adding this precision to the *time* logical type Avro > specification to have an homogeneous precision in all times logical types. > > Whole description can be viewed in AVRO-4043 > < > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FAVRO-4043&data=05%7C02%7Cclemensv%40microsoft.com%7C31b3729197a7492e2ed108dd2ce737f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638716093824113204%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C80000%7C%7C%7C&sdata=%2BoFrs2Lv79nXomm79Y9s5253Gm2%2FoIYtnbOU9ZOZ3WU%3D&reserved=0 > <https://issues.apache.org/jira/browse/AVRO-4043>> and in PR#3125 > < > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Favro%2Fpull%2F3125&data=05%7C02%7Cclemensv%40microsoft.com%7C31b3729197a7492e2ed108dd2ce737f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638716093824129269%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C80000%7C%7C%7C&sdata=WUgtCL7HVEzr%2BgvceaLprJy5sXszgFeBcKcOyq3zxsI%3D&reserved=0 > <https://github.com/apache/avro/pull/3125>> and some implementations: > > - AVRO-4044: [C++] Add time-nanos logical type > < > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FAVRO-4044&data=05%7C02%7Cclemensv%40microsoft.com%7C31b3729197a7492e2ed108dd2ce737f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638716093824140522%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C80000%7C%7C%7C&sdata=d0kGo5mYoiWWskkCoyRWeECArsArk2X9bCAZ8A46Wlk%3D&reserved=0 > <https://issues.apache.org/jira/browse/AVRO-4044>> : PR#3147 > < > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Favro%2Fpull%2F3147&data=05%7C02%7Cclemensv%40microsoft.com%7C31b3729197a7492e2ed108dd2ce737f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638716093824152731%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C80000%7C%7C%7C&sdata=sA7fnnyE7qS2mgqrcp7LhEOGlI2iiZWEpUt0%2BvL8nLI%3D&reserved=0 > <https://github.com/apache/avro/pull/3147>> > - AVRO-4056: [Java] Add time-nanos logical type > < > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FAVRO-4056&data=05%7C02%7Cclemensv%40microsoft.com%7C31b3729197a7492e2ed108dd2ce737f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638716093824165146%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C80000%7C%7C%7C&sdata=8athsdpQivRjfXH2ka2NIdxigDQGBM8nnluRz9MvGkc%3D&reserved=0 > <https://issues.apache.org/jira/browse/AVRO-4056>> : PR#3158 > < > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Favro%2Fpull%2F3158&data=05%7C02%7Cclemensv%40microsoft.com%7C31b3729197a7492e2ed108dd2ce737f9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638716093824176194%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C80000%7C%7C%7C&sdata=4fGwWl2AjsjnhVwttKpa5LkjgXD9rmIE5t7RvHAf5G4%3D&reserved=0 > <https://github.com/apache/avro/pull/3158>> > > Regards >
