Hey -- thanks for reaching out! This is the right place to reach out for JIRA account creation -- we're still adjusting to the new changes, but I can create an account for you.
I'll send an email with some details to confirm. Ryan On Wed, Dec 7, 2022 at 10:26 AM Martin Grigorov <[email protected]> wrote: > > Hi Matthew, > > Since you are not subscribed to the mailing list your messages have to be > moderated. > I didn't notice anything from you at [email protected] > > About JIRA - you can ask for an account here ([email protected]) or > [email protected]. We need an username, display name and email address to > create one for you. > > Martin > > > On Wed, Dec 7, 2022 at 10:36 AM Matthew Chng <[email protected]> > wrote: > > > Hi, > > > > Could not reach the users mailing list so testing the dev mailing list. > > Can't find contact email for Jira account creation. Can't join Slack > > because I have no ASF email account. > > > > Issue described below. > > > > Thanks, > > Matthew > > > > ---------- Forwarded message --------- > > From: Matthew Chng <[email protected]> > > Date: Wed, Dec 7, 2022 at 4:18 PM > > Subject: Possible bug with avro-js with binary serdes and schema resolution > > To: <[email protected]> > > > > > > Hi all, > > I am encountering an issue with the avro.js' NPM module where Avro > > serialized into binary buffers are not readable by a different but > > compatible reader schema (evolved). This issue is only occurring when using > > the `toBuffer()` and `fromBuffer()` methods and works as expected when > > using the `toString()` and `fromString()` JSON serdes methods. > > The following is an example of an evolving schema with the difference being > > the additional `gender` field that has a default value. > > > > const parentV1Type = avro.parse({ > > name: 'Parent', > > type: 'record', > > fields: [ > > { name: 'name', type: 'string' } > > ] > > }) > > > > const parentV2Type = avro.parse({ > > name: 'Parent', > > type: 'record', > > fields: [ > > { name: 'name', type: 'string' }, > > { name: 'gender', type: 'string', default: 'unspecified' } > > ] > > }) > > > > According to > > https://avro.apache.org/docs/1.11.1/specification/#schema-resolution > > they should be both backwards/forwards reader compatible. > > They have these properties. > > > > - both schemas are records with the same (unqualified) name > > - if the writer’s record contains a field with a name not present in the > > reader’s record, the writer’s value for that field is ignored. > > - if the reader’s record schema has a field that contains a default > > value, and writer’s schema does not have a field with the same name, > > then > > the reader should use the default value from its field. > > > > Testing with the `toString()` and `fromString()` JSON serdes methods > > indicated as such. I've created a simple test script to produce the issue. > > Also included test with nested schema. The script is included after the > > output. The errors encountered are either > > > > - truncated buffer; or > > - trailing data > > > > Script output: > > > > > > --- JSON Writer: ParentV1, Reader: ParentV2 --- > > parentV1Json: {"name":"David"} > > parentV2ReadFromV1Json: {"name":"David","gender":"unspecified"} > > > > --- JSON Writer: ParentV2, Reader: ParentV1 --- > > parentV2Json: {"name":"David","gender":"Father"} > > parentV1ReadFromV2Json: {"name":"David"} > > > > --- Buffer Writer: ParentV1, Reader: ParentV1 --- > > parentV1Buffer: <Buffer 0a 44 61 76 69 64> > > parentV1ReadFromV1Buffer: {"name":"David"} > > > > --- Buffer Writer: ParentV2, Reader: ParentV2 --- > > parentV2Buffer: <Buffer 0a 44 61 76 69 64 0c 46 61 74 68 65 72> > > parentV2ReadFromV1Buffer: {"name":"David","gender":"Father"} > > > > --- Buffer Writer: ParentV1, Reader: ParentV2 --- > > parentV1Buffer: <Buffer 0a 44 61 76 69 64> > > parentV2ReadFromV1Buffer: ERROR truncated buffer > > > > --- Buffer Writer: ParentV2, Reader: ParentV1 --- > > parentV2Buffer: <Buffer 0a 44 61 76 69 64 0c 46 61 74 68 65 72> > > parentV1ReadFromV2Buffer: ERROR trailing data > > > > --- JSON Writer: meWithParentV1, Reader: meWithParentV2 --- > > meWithParentV1Json: > > {"name":"Davidson","parent":{"name":"David"}} > > meWithParentV2ReadFromV1Json: > > {"name":"Davidson","parent":{"name":"David","gender":"unspecified"}} > > > > --- JSON Writer: meWithParentV2, Reader: meWithParentV1 --- > > meWithParentV2Json: > > {"name":"Davidson","parent":{"name":"David","gender":"Father"}} > > meWithParentV1ReadFromV2Json: > > {"name":"Davidson","parent":{"name":"David"}} > > > > --- Buffer Writer: meWithParentV1, Reader: meWithParentV1 --- > > meWithParentV1Buffer: <Buffer 10 44 61 76 69 64 73 6f 6e 0a 44 > > 61 76 69 64> > > meWithParentV1ReadFromV1Buffer: > > {"name":"Davidson","parent":{"name":"David"}} > > > > --- Buffer Writer: meWithParentV2, Reader: meWithParentV2 --- > > meWithParentV2Buffer: <Buffer 10 44 61 76 69 64 73 6f 6e 0a 44 > > 61 76 69 64 0c 46 61 74 68 65 72> > > meWithParentV2ReadFromV2Buffer: > > {"name":"Davidson","parent":{"name":"David","gender":"Father"}} > > > > --- Buffer Writer: meWithParentV1, Reader: meWithParentV2 --- > > meWithParentV1Buffer: <Buffer 10 44 61 76 69 64 73 6f 6e 0a 44 61 > > 76 69 64> > > meWithParentV2ReadFromV1Buffer: ERROR truncated buffer > > > > --- Buffer Writer: meWithParentV2, Reader: meWithParentV1 --- > > meWithParentV2Buffer: <Buffer 10 44 61 76 69 64 73 6f 6e 0a 44 61 > > 76 69 64 0c 46 61 74 68 65 72> > > meWithParentV1ReadFromV2Buffer: ERROR trailing data > > > > > > Script src: > > > > const avro = require('avro-js') > > const { writer } = require('repl') > > > > const parentV1Type = avro.parse({ > > name: 'Parent', > > type: 'record', > > fields: [ > > { name: 'name', type: 'string' } > > ] > > }) > > > > const parentV2Type = avro.parse({ > > name: 'Parent', > > type: 'record', > > fields: [ > > { name: 'name', type: 'string' }, > > { name: 'gender', type: 'string', default: 'unspecified' } > > ] > > }) > > > > const meSchema = { > > name: 'Me', > > type: 'record', > > fields: [ > > { name: 'name', type: 'string' }, > > { name: 'parent', type: 'Parent' } > > ] > > } > > > > const meWithParentV2Type = avro.parse(meSchema, { > > registry: { > > Parent: parentV2Type > > } > > }) > > > > const meWithParentV1Type = avro.parse(meSchema, { > > registry: { > > Parent: parentV1Type > > } > > }) > > > > const parentV1 = { name: 'David'} > > const parentV2 = { name: 'David', gender: 'Father' } > > > > const meWithParentV1 = { > > name: 'Davidson', > > parent: parentV1 > > } > > > > const meWithParentV2 = { > > name: 'Davidson', > > parent: parentV2 > > } > > > > console.log("") > > console.log("--- JSON Writer: ParentV1, Reader: ParentV2 ---") > > const parentV1Json = parentV1Type.toString(parentV1) > > console.log("parentV1Json: ", parentV1Json) > > const parentV2ReadFromV1Json = parentV2Type.fromString(parentV1Json) > > console.log("parentV2ReadFromV1Json: ", JSON.stringify( > > parentV2ReadFromV1Json)) > > > > console.log("") > > console.log("--- JSON Writer: ParentV2, Reader: ParentV1 ---") > > const parentV2Json = parentV2Type.toString(parentV2) > > console.log("parentV2Json: ", parentV2Json) > > const parentV1ReadFromV2Json = parentV1Type.fromString(parentV2Json) > > console.log("parentV1ReadFromV2Json: ", JSON.stringify( > > parentV1ReadFromV2Json)) > > > > console.log("") > > console.log("--- Buffer Writer: ParentV1, Reader: ParentV1 ---") > > const parentV1Buffer = parentV1Type.toBuffer(parentV1) > > console.log("parentV1Buffer: ", parentV1Buffer) > > const parentV1ReadFromV1Buffer = parentV1Type.fromBuffer(parentV1Buffer) > > console.log("parentV1ReadFromV1Buffer: ", JSON.stringify( > > parentV1ReadFromV1Buffer)) > > > > console.log("") > > console.log("--- Buffer Writer: ParentV2, Reader: ParentV2 ---") > > const parentV2Buffer = parentV2Type.toBuffer(parentV2) > > console.log("parentV2Buffer: ", parentV2Buffer) > > const parentV2ReadFromV2Buffer = parentV2Type.fromBuffer(parentV2Buffer) > > console.log("parentV2ReadFromV1Buffer: ", JSON.stringify( > > parentV2ReadFromV2Buffer)) > > > > console.log("") > > console.log("--- Buffer Writer: ParentV1, Reader: ParentV2 ---") > > console.log("parentV1Buffer: ", parentV1Buffer) > > try { > > const parentV2ReadFromV1Buffer = parentV2Type.fromBuffer(parentV1Buffer) > > console.log("parentV2ReadFromV1Buffer: ", JSON.stringify( > > parentV2ReadFromV1Buffer)) > > } catch (e) { > > console.log("parentV2ReadFromV1Buffer: ERROR ", e.message) > > } > > > > console.log("") > > console.log("--- Buffer Writer: ParentV2, Reader: ParentV1 ---") > > console.log("parentV2Buffer: ", parentV2Buffer) > > try { > > const parentV1ReadFromV2Buffer = parentV1Type.fromBuffer(parentV2Buffer) > > console.log("parentV1ReadFromV2Buffer: ", JSON.stringify( > > parentV1ReadFromV2Buffer)) > > } catch (e) { > > console.log("parentV1ReadFromV2Buffer: ERROR ", e.message) > > } > > > > console.log("") > > console.log("--- JSON Writer: meWithParentV1, Reader: meWithParentV2 ---") > > const meWithParentV1Json = meWithParentV1Type.toString(meWithParentV1) > > console.log("meWithParentV1Json: ", meWithParentV1Json) > > const meWithParentV2ReadFromV1Json = meWithParentV2Type.fromString( > > meWithParentV1Json) > > console.log("meWithParentV2ReadFromV1Json: ", JSON.stringify( > > meWithParentV2ReadFromV1Json)) > > > > console.log("") > > console.log("--- JSON Writer: meWithParentV2, Reader: meWithParentV1 ---") > > const meWithParentV2Json = meWithParentV2Type.toString(meWithParentV2) > > console.log("meWithParentV2Json: ", meWithParentV2Json) > > const meWithParentV1ReadFromV2Json = meWithParentV1Type.fromString( > > meWithParentV2Json) > > console.log("meWithParentV1ReadFromV2Json: ", JSON.stringify( > > meWithParentV1ReadFromV2Json)) > > > > console.log("") > > console.log("--- Buffer Writer: meWithParentV1, Reader: meWithParentV1 > > ---") > > const meWithParentV1Buffer = meWithParentV1Type.toBuffer(meWithParentV1) > > console.log("meWithParentV1Buffer: ", meWithParentV1Buffer) > > const meWithParentV1ReadFromV1Buffer = meWithParentV1Type.fromBuffer( > > meWithParentV1Buffer) > > console.log("meWithParentV1ReadFromV1Buffer: ", JSON.stringify( > > meWithParentV1ReadFromV1Buffer)) > > > > console.log("") > > console.log("--- Buffer Writer: meWithParentV2, Reader: meWithParentV2 > > ---") > > const meWithParentV2Buffer = meWithParentV2Type.toBuffer(meWithParentV2) > > console.log("meWithParentV2Buffer: ", meWithParentV2Buffer) > > const meWithParentV2ReadFromV2Buffer = meWithParentV2Type.fromBuffer( > > meWithParentV2Buffer) > > console.log("meWithParentV2ReadFromV2Buffer: ", JSON.stringify( > > meWithParentV2ReadFromV2Buffer)) > > > > console.log("") > > console.log("--- Buffer Writer: meWithParentV1, Reader: meWithParentV2 > > ---") > > console.log("meWithParentV1Buffer: ", meWithParentV1Buffer) > > try { > > const meWithParentV2ReadFromV1Buffer = meWithParentV2Type.fromBuffer( > > meWithParentV1Buffer) > > console.log("meWithParentV2ReadFromV1Buffer: ", JSON.stringify( > > meWithParentV2ReadFromV1Buffer)) > > } catch (e) { > > console.log("meWithParentV2ReadFromV1Buffer: ERROR ", e.message) > > } > > > > console.log("") > > console.log("--- Buffer Writer: meWithParentV2, Reader: meWithParentV1 > > ---") > > console.log("meWithParentV2Buffer: ", meWithParentV2Buffer) > > try { > > const meWithParentV1ReadFromV2Buffer = meWithParentV1Type.fromBuffer( > > meWithParentV2Buffer) > > console.log("meWithParentV1ReadFromV2Buffer: ", JSON.stringify( > > meWithParentV1ReadFromV2Buffer)) > > } catch (e) { > > console.log("meWithParentV1ReadFromV2Buffer: ERROR ", e.message) > > } > > > > > > If this is indeed a bug, how do I create a ticket in the Jira board to > > report it? Thanks. > > > > Matthew > >
