Hello!
A bit tongue in cheek: the one advantage of the current Avro JSON
encoding is that it drives users rapidly to prefer the binary
encoding! In its current state, Avro isn't really a satisfactory
toolkit for JSON interoperability, while it shines for binary
interoperability. Using JSON with Avro schemas is pretty unwieldy and
a JSON data designer will almost never be entirely satisfied with the
JSON "shape" they can get... today it's useful for testing and
debugging.
That being said, it's hard to argue with improving this experience
where it can help developers that really want to use Avro JSON for
data transfer, especially for things accepting JSON where the
intention is clearly unambiguous or allowing optional attributes to be
missing. I'd be enthusiastic to see some of these improvements,
especially if we keep the possibility of generating strict Avro JSON
for forwards and backwards compatibility.
My preference would be to avoid adding JSON-specific attributes to the
spec where possible. Maybe we could consider implementing Avro JSON
"variants" by implementing encoder options, or alternative encorders
for an SDK. There's probably a nice balance between a rigorous and
interoperable (but less customizable) JSON encoding, and trying to
accommodate arbitrary JSON in the Avro project.
All my best and thanks for this analysis -- I'm excited to see where
this leads! Ryan
On Thu, Apr 18, 2024 at 8:01 PM Oscar Westra van Holthe - Kind
<os...@westravanholthe.nl> wrote:
>
> Thank you Clemens,
>
> This is a very detailed set of proposals, and it looks like it would work.
>
> I do however, feel we'd need to define a way to unions with records. Your
> proposal lists various options, of which the discriminatory option seems most
> portable to me.
>
> You mention the "displayName" proposal. I don't like that, as it mixes data
> with UI elements. The discriminator option can specify a fixed or
> configurable field to hold the type of the record.
>
> Kind regards,
> Oscar
>
>
> --
> Oscar Westra van Holthe - Kind <os...@westravanholthe.nl>
>
> Op do 18 apr. 2024 10:12 schreef Clemens Vasters via user
> <user@avro.apache.org>:
>>
>> Hi everyone,
>>
>>
>>
>> the current JSON Encoding approach severely limits interoperability with
>> other JSON serialization frameworks. In my view, the JSON Encoding is only
>> really useful if it acts as a bridge into and from JSON-centric applications
>> and it currently gets in its own way.
>>
>>
>>
>> The current encoding being what it is, there should be an alternate mode
>> that emphasizes interoperability with JSON “as-is” and allows Avro Schema to
>> describe existing JSON document instances such that I can take someone’s
>> existing JSON document in on one side of a piece of software and emit Avro
>> binary on the other side while acting on the same schema.
>>
>>
>>
>> There are four specific issues:
>>
>>
>>
>> Binary Values
>> Unions with Primitive Type Values and Enum Values
>> Unions with Record Values
>> DateTime
>>
>>
>>
>> One by one:
>>
>>
>>
>> 1. Binary values:
>>
>> ---------------------
>>
>>
>>
>> Binary values are (fixed and bytes) are encoded as escaped unicode literals.
>> While I appreciate the creative trick, it costs 6 bytes for each encoded
>> byte. I have a hard time finding any JSON libraries that provide a
>> conversion of such strings from/to byte arrays, so this approach appears to
>> be idiosyncratic for Avro’s JSON Encoding.
>>
>>
>>
>> The common way to encode binary in JSON is to use base64 encoding and that
>> is widely and well supported in libraries. Base64 is 33% larger than plain
>> bytes, the encoding chosen here is 500% (!) larger than plain bytes.
>>
>>
>>
>> The Avro decoder is schema-informed and it knows that a field is expected to
>> hold bytes, so it’s easy to mandate base64 for the field content in the
>> alternate mode.
>>
>>
>>
>> 2. Unions with Primitive Type Values and Enum Values
>>
>> ---------------------
>>
>>
>>
>> It’s common to express optionality in Avro Schema by creating a union with
>> the “null” type, e.g. [“string”, “null”]. The Avro JSON Encoding opts to
>> encode such unions, like any union, as { “{type}”: {value} } when the value
>> is non-null.
>>
>>
>>
>> This choice ignores common practice and the fact that JSON’s values are
>> dynamically typed (RFC8259 Section-3) and inherently accommodate unions. The
>> conformant way to encode a value choice of null or “string” into a JSON
>> value is plainly null and “string”.
>>
>>
>>
>> “foo” : null
>>
>> “foo”: “value”
>>
>>
>>
>> The “field default values” table in the Avro spec maps Avro types to the
>> JSON types null, boolean, integer, number, string, object, and array, all of
>> which can be encoded into and, more importantly, unambiguously decoded from
>> a JSON value. The only semi-ambiguous case is integer vs. number, which is a
>> convention in JSON rather than a distinct type, but any Avro serializer is
>> guided by type information and can easily make that distinction.
>>
>>
>>
>> 3. Unions with Record Values
>>
>> ---------------------
>>
>>
>>
>> The JSON Encoding pattern of unions also covers “record” typed values, of
>> course, and this is indeed a tricky scenario during deserialization since
>> JSON does not have any built-in notion of type hints for “object” typed
>> values.
>>
>>
>>
>> The problem of having to disambiguate instances of different types in a
>> field value is a common one also for users of JSON Schema when using the
>> “oneOf” construct, which is equivalent to Avro unions. There are two common
>> strategies:
>>
>>
>>
>> - “Duck Typing”: Every conformant JSON Schema Validator determines the
>> validity of a JSON node against a “oneOf" rule by testing the instance
>> against all available alternative schema definitions. Validation fails if
>> there is not exactly one valid match.
>>
>> - Discriminators: OpenAPI, for instance, mandates a “discriminator” field
>> (see https://spec.openapis.org/oas/latest.html#discriminator-object) for
>> disambiguating “oneOf” constructs, whereby the discriminator property is
>> part of each instance. That approach informs numerous JSON serialization
>> frameworks, which implement discriminators under that assumption.
>>
>>
>>
>> The Java Jackson library indeed supports the Avro JSON Encoding’s style of
>> putting the discriminator into a wrapper field name (JsonTypeInfo
>> annotation, JsonTypeInfo.As.WRAPPER_OBJECT). Many other frameworks only
>> support the property approach, though, including the two dominant ones for
>> .NET, Pydantic of Python, and others. There’s tooling like Redocly that
>> flags that approach as a “mistake” (see
>> https://redocly.com/docs/resources/discriminator/#property-outside-of-the-object).
>>
>>
>>
>> What that means is that most existing JSON instances with ambiguous types
>> will either use property discriminators or the implementation will rely on
>> duck typing as JSON Schema does for validation. The Avro JSON Encoding
>> approach is rare and is also counterintuitive for anyone comparing the
>> declared object structure and the JSON structure who is not familiar with
>> Avro’s encoding rules. It has confused a lot of people in our house, for
>> sure.
>>
>>
>>
>> Proposed is the following approach:
>>
>>
>>
>> a) add a new, optional “const” attribute that can be applied to any record
>> field declaration that is of a primitive type. When present, the attribute
>> causes the field to always have this value. In Avro binary encoding, the
>> field is not transmitted, at all, but the decoder yields it with the given
>> value. In Avro JSON encoding, the field is emitted and for serialization to
>> succeed for the record type, the field must be present with the given value.
>>
>> b) perform disambiguation of types by the same principle as JSON Schema for
>> oneOf, with a performance preference for matching fields flagged with
>> “const” against the incoming JSON node. When the deserializer is configured
>> by schema to know what fields and values to look for, there should not be no
>> performance hit compared to the current approach. Derialization fails if
>> there is not one unambiguous match. That is exactly in line with what JSON
>> Schema validation implementations do. JSON Schema also has a “const”
>> construct. “Const” or single-valued enums are often used as discriminator
>> helpers with JSON Schema’s oneOf.
>>
>> c) optional: add a new, optional “displayname” attribute that can hold an
>> alternate name for the field without the restrictions of the “name”
>> character set, so that discriminators like “$type” can be matched. A further
>> upside of adding this field is that it can generally be used to match
>> international characters in JSON object keys, which are obviously permitted
>> there.
>>
>>
>>
>> 4. Date Time
>>
>> ---------------------
>>
>>
>>
>> JSON data generally leans on the RFC3339 profile of ISO8601 for dates and
>> durations, not the last because JSON Schema defines these choices as
>> “format” variants for strings.
>>
>>
>>
>> If the incoming type of a field is a string instead of a number, JSON
>> deserialization in the alternate mode should interpret the logicalTypes for
>> dates as follows.
>>
>>
>>
>> “date” – RFC3339 5.6. “full-date”
>> “time-millis” – RFC3339 5.6. “date-time”
>> “time-micros” – RFC3339 5.6. “partial-time”
>> “timestamp-millis” – RFC3339 5.6 “date-time”
>> “timestamp-micros”—RFC3339 5.6 “date-time”
>> “local-timestamp-millis” – RFC3339 5.6 “date-time”, ignoring offset (but see
>> RFC 3339 4.4)
>> “local-timestamp-micros”—RFC3339 5.6 “date-time” , ignoring offset (but see
>> RFC 3339 4.4)
>> “duration” – RFC3339 Appendix A “duration”
>>
>>
>>
>> The JSON serialization in the alternate mode should have an option, and
>> default to, serializing dates as strings. Deserialization parsers MAY be
>> lenient and also accept RFC1123 5.2.13 date time strings where RFC3339 5.6
>> “date-time” is specified, but I’d make that an implementation choice.
>>
>>
>>
>>
>>
>> Best Regards
>>
>> Clemens Vasters
>>
>>
