I think we should approach this idea in two parts: 1) The schema. Things like does a different unit mean a different schema fingerprint even though the bytes remain the same. What does a different unit mean for schema evolution.
2) Language specifics. Scala has different possibilities than Java. On Sat, Jun 29, 2019, 18:59 Erik Erlandson <[email protected]> wrote: > I've been puzzling over what can be done to support this in more > widely-used languages. The dilemma relative to the current language > ecosystem is that languages with "modern" type systems (Haskell, Rust, > Scala, etc) capable of supporting compile-time unit checking, in the > particular style I've been exploring, are not yet widely used. > > With respect to Java, a couple approaches are plausible. One is to enhance > the language, for example with Java-8 compiler plugins. Another might be to > implement a unit type system similar to squants > <https://github.com/typelevel/squants>. This style of unit type system is > not as flexible or intuitive as what can be done with Scala's latest type > system sorcery, but it would allow the community to build out a Java native > type system that supports compile-time unit analysis. And its coverage of > standard units could be made very good, as squants itself demonstrates. > > Python would also be a high-coverage target. I'm even less sure what to do > for python, as it has no compile-time type checking, but perhaps a > squants-like python class system would add value. Maybe python's new > type-hints feature could be leveraged? > > Regarding unit expression representation, I'm not unhappy with what I've > prototyped in `coulomb-avro`, in broad strokes. It has deficiencies that > would need addressing. It doesn't yet support standard unit abbreviations, > nor does it understand plurals (e.g. it can parse "second" but not > "seconds"). Since it's "unit" field is just a custom metadata key, there is > no enforcement. Parsers are currently instantiated via explicit lists of > types, which is a property I like, but that may not work well in a world > where multiple language bindings must be supported in a portable manner. > > > > On Sat, Jun 29, 2019 at 1:46 AM Niels Basjes <[email protected]> wrote: > > > Hi, > > > > I attended your talk in Berlin and at the end I thought "too bad this is > > only Scala". > > > > I think it's a good idea to have this in Avro. > > > > The details will be tricky: How to encode the units in the schema for > > example. > > Especially because of the automatic conversion you spoke about. > > > > Niels > > > > On Fri, Jun 28, 2019, 23:58 Erik Erlandson <[email protected]> wrote: > > > > > Hi Avro community, > > > > > > Recently I have been experimenting with avro schema that are extended > > with > > > a "unit" field. By "unit" I mean expressions like "second", or > > "megabyte" - > > > that is "units of measure". > > > > > > I delivered a short talk on my experiments at Berlin Buzzwords, which > can > > > be viewed here: > > > https://www.youtube.com/watch?v=qrQmB2KFKE8 > > > I also wrote a short blog post that may be faster to ingest: > > > > > > > > > http://erikerlandson.github.io/blog/2019/05/23/unit-types-for-avro-schema-integrating-avro-with-coulomb/ > > > > > > I received some audience interest in making this concept "first class" > > for > > > avro, and so I'm writing to see what the avro dev community thinks of > the > > > idea. One issue is that this kind of unit checking is currently only > > > available for Scala (and specifically scala 2.13 +). > > > > > > The Scala project itself is here: > > > https://github.com/erikerlandson/coulomb > > > > > > Cheers, > > > Erik > > > > > >
