Re: Evolving schemas and namespaces

[Anders Sundelin]

Hi Elliot,














BR

/Anders


On 2016-12-02 17:01, Elliot West wrote:

Schema compatibility (as defined by Avro) is not commutative, i.e. one schema can be read-compatible with another (writer) schema, while the reverse is not necessarily true. This is actually a good thing, because it means that schemas can evolve, taking into consideration that some clients can read certain schema versions, while others can read possibly more versions. The obvious use case is for a server, performing work on behalf of several clients. Each client uses schema negotiation, so when the server reads a message with schema S-1.7, ita can process it (possibly with its own schema S-1.8), but the response should be sent back also using the S-1.7 schema (since that was what the client had negotiated). Clients using schema S-1.8 should of course get their response in the S-1.8 schema and so on. In short, this kind of use-case requires the server (in its interface part) to be aware what schemas that the clients speak, even if it internally uses the "superset schema", where all schemas can be read (i.e. typically, the "superset schema" is read-compatible with all different versions, typically since the superset schema is the latest of these schemas).

Hi Anders,



That said, I haven't tested my expectations concretely :-)

Elliot.

If the one schema is a compatible evolution of the other, what is the need for multiple Java types? Schema compatibility implies that data written by one schema can be safety marshalled to/from a Java class generated another compatible version no?. Obviously the specific behaviours are dictated by the selected compatibility level, the version of Avro data being encoded/decoded, and the version of the schema from which that class was generated. If the compatibility rules are followed I would not expect to need multiple Java representations. On 2 December 2016 at 15:43, Anders Sundelin <anders.sunde...@ericsson.com <mailto:anders.sunde...@ericsson.com>> wrote:

    Hi Niels and Elliot,

    Thinking from the Java perspective now, the nice thing about
    namespaces (and, to a lesser degree, the name itself) is that they
    are mapped to packages (and classnames).

    If the v1 Avro spec was used as a base for generating java
    classes, then the corresponding Java class would then be
    "com.example.some.v1.MyType",

    and, correspondingly, the v2 Avro schema would generate into
    "com.example.some.v2.MyType"

    In other words, clients (or servers) could happily use both
    classes while talking to different other peers (supporting, for
    instance, protocol negotiation).

    This is the most important reason for why I think the rule you
    mention is kind of strange.

    What do you others think?

    BR

    /Anders



    On 2016-12-02 16:33, Elliot West wrote:

    There is perhaps a little ambiguity in the spec:

    From https://avro.apache.org/docs/current/spec.html#names
    <https://avro.apache.org/docs/current/spec.html#names>
      Record, enums and fixed are named types. Each has a fullname
    that is composed of two parts; a name and a namespace.*Equality
    of names is defined on the fullname*.

    From
    https://avro.apache.org/docs/current/spec.html#Schema+Resolution
    <https://avro.apache.org/docs/current/spec.html#Schema+Resolution>:
      It is an error if the two schemas do not match.
      To match, one of the following must hold:
      ...
    *both schemas are records with the same name*
      ...

    I suspect that in this case 'name' means 'fullname' and therefore
    by choosing a different namespace you've declared to Avro that
    they should be considered different types.

    If you are trying to annotate different schemas with a version
    identifier, perhaps a 'doc' property might be more appropriate?

    On 2 December 2016 at 15:11, Niels Basjes <ni...@basjes.nl
    <mailto:ni...@basjes.nl>> wrote:

        Hi,

        When I run the code below the output indicates that these two are 
incompatible in terms of schema evolution.

        The ONLY difference is the namespace (v1 and v2).

        If I remove the namespace line the are reported as 'compatible'.

        My question is why these two are considered to be incompatible?

        @Test public void evolveTest()throws IOException {
           Schema schemaV1 =new Schema.Parser().parse("{\n" +
             " \"type\": \"record\",\n" +
             " \"name\": \"Foo\",\n" +
             " \"namespace\": \"nl.example.evoleschema.v1\",\n" +
             " \"fields\": [ {\n" +
             " \"name\": \"count\",\n" +
             " \"type\": {\n" +
             " \"type\": \"enum\",\n" +
             " \"name\": \"Bar\",\n" +
             " \"symbols\": [ \"ONE\", \"TWO\", \"THREE\"]\n" +
             " }\n" +
             " } ]\n" +
             "}");

           Schema schemaV2 =new Schema.Parser().parse("{\n" +
             " \"type\": \"record\",\n" +
             " \"name\": \"Foo\",\n" +
             " \"namespace\": \"nl.example.evoleschema.v2\",\n" +
             " \"fields\": [ {\n" +
             " \"name\": \"count\",\n" +
             " \"type\": {\n" +
             " \"type\": \"enum\",\n" +
             " \"name\": \"Bar\",\n" +
             " \"symbols\": [ \"ONE\", \"TWO\", \"THREE\"]\n" +
             " }\n" +
             " } ]\n" +
             "}");

           LOG.info("{}", 
SchemaCompatibility.checkReaderWriterCompatibility(schemaV1, schemaV2).getType());
           LOG.info("{}", 
SchemaCompatibility.checkReaderWriterCompatibility(schemaV2, schemaV1).getType());
        }

-- Best regards / Met vriendelijke groeten, Niels Basjes

smime.p7s
Description: S/MIME Cryptographic Signature