Oscar just proposed a mechanism that could give us more tools in the Java SDK for parsing/formatting Schemas (and Protocols): to provide alternative text representations of these Avro concepts in a way that is more configurable and easy to add/align features.
For info, there was another JIRA that talked about adding another "canonical" format that retained a different set of attributes than the current (very minimal) one, specifically to address schema registries -- if I could find it, we could reopen the discussion :/ I'd love to hear your take on how these concepts would work in the C# SDK! Is there an option that gives the user some flexibility to choose how they need their schema to be represented? All my best, Ryan On Tue, Mar 8, 2022 at 5:51 PM kyle minmaxcorp.com <[email protected]> wrote: > > Options to have the C# Schema.ToString() method match the Java output. > > Option #1 > Create a ToJsonString() method. This would involve creating a separate path > to generate the JSON string that ToString creates, but instead matches the > output of the Java toString() method. > > Option #2 > Overload ToString, but send an enum with a schema version. This would match > the output based on the version provided. Similar to Option #1, but a bit > more code reuse, and would potentially require maintenance for each release, > assuming there is a change in the ouput. > > Enum definition: > CSharp1.9 > CSharp1.10 > CSharp1.11 > Java1.9 > Java1.10 > Java1.11 > > Option #3 > Change ToString to match Java's toString method. This is the least amount of > work, but potentially creates a backwards compatibility issue for those that > use something like Confluent and it's Schema Registry. > > Option #4 > I have already ruled this out, but I was planning on sending a JSON schema > file to validate the output against and reformat. The problem is that the C# > version and Java version both match the Schema even though the output is > different. Although, we may want to use something similar and create a > validate method. > > > On March 7, 2022 1:49 PM Martin Grigorov <[email protected]> wrote: > > > > > > > > > > > > On Mon, Mar 7, 2022 at 8:46 PM kyle minmaxcorp.com (http://minmaxcorp.com) > > <[email protected]> wrote: > > > What exactly is the value of having the same output in mixed environments > > > ? > > > Or: what kind of problems do you face now ? > > > > > > This is actually a result of dealing with Confluent. They (Confluent) > > > happen to do an MD5 hash on the schema that you have compared to what is > > > in the registry. Since Java and C# generate that schema differently it > > > causes it to fail the MD5 hash. > > > > I understand! > > Thanks! > > I'll see whether the Rust SDK needs work too! > > > > > > > While this isn't a problem with Avro specifically, I want to apply a > > > capability to support generating the same schema. > > > > > > So the plan currently is to do the following: > > > 1. Maintain backwards compatibility. We don't want to change what it does > > > today, because we could have unintended consequences of just fixing > > > schema generation. (both with ToString and what AvroGen does). > > > 2. Extend the functionality to define a format that determines how the > > > schema can be generated. > > > > > > For number 2. I'm debating the implementation and doing a bit of > > > prototyping before writing the actual story for implementation. Once I'm > > > done, I want to run the design by you and Ryan (and anyone else that > > > wants to contribute). > > > > > > > On March 3, 2022 1:09 PM Martin Grigorov <[email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 1, 2022 at 6:42 AM kyle minmaxcorp.com > > > > (http://minmaxcorp.com) (http://minmaxcorp.com) <[email protected]> > > > > wrote: > > > > > I am noticing a difference between Java and C# versions of Avro when > > > > > I call Schema.ToString(). > > > > > > > > > > First, is that the C# version adds the namespace to each Named > > > > > schema. Second, is the order of the output. I would expect across > > > > > languages that we output the same JSON string. > > > > > > > > > > The following took a schema json string and called > > > > > Schema.Parse(string json) and Schema.parse(String jsonSchema, boolean > > > > > validate) or C# and Java. > > > > > > > > > > Original Schema string > > > > > {"type":"record","name":"TestRecord","namespace":"test.namespace","fields":[{"name":"testName","type":{"type":"record","name":"TestData","fields":[{"name":"version","type":"float","doc":"version > > > > > number of this schema"}]}}]} > > > > > > > > > > C# output of Schema.ToString() > > > > > {"type":"record","name":"TestRecord","namespace":"test.namespace","fields":[{"name":"testName","type":{"type":"record","name":"TestData","namespace":"test.namespace","fields":[{"name":"version","doc":"version > > > > > number of this schema","type":"float"}]}}]} > > > > > > > > > > Java output of Schema.toString > > > > > {"type":"record","name":"TestRecord","namespace":"test.namespace","fields":[{"name":"testName","type":{"type":"record","name":"TestData","fields":[{"name":"version","type":"float","doc":"version > > > > > number of this schema"}]}}]} > > > > > > > > > > It is not overly complicated to have the C# version match the Java > > > > > version, but in order to maintain backwards compatibility while > > > > > supporting a new output, we will need to create a Schema.ToJsonString > > > > > method, and update the WriteJson* methods as well to support the new > > > > > flow. Ideally we mark ToString() obsolete with the message to use the > > > > > ToJsonString method. Eventually, pointing ToString() to the > > > > > ToJsonString method. > > > > > > > > > > While this work is not complicated it is a lot of work and testing. > > > > > While, I personally see value in having the output being the same (I > > > > > work in a mixed technology environment), I wanted to address any > > > > > concerns with this sort of change. > > > > > > > > What exactly is the value of having the same output in mixed > > > > environments ? > > > > Or: what kind of problems do you face now ? > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > Kyle T. Schoonover
