Correct me if I'm wrong here. But as far as I understood the way of serializing the schema is using Avro, as it is part of the file. To avoid confusion there should be one way of serializing.
However, I'm not sure if this is worth the hassle of not simply implementing serializable. Also Flink there is a rather far from optimal implementation: https://github.com/apache/flink/blob/master/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/avro/ParquetAvroWriters.java#L72 This converts it to JSON and back while distributing the schema to the executors. Cheers, Fokko Op ma 15 jul. 2019 om 23:03 schreef Doug Cutting <cutt...@gmail.com>: > I can't think of a reason Schema should not implement Serializable. > > There's actually already an issue & patch for this: > > https://issues.apache.org/jira/browse/AVRO-1852 > > Doug > > On Mon, Jul 15, 2019 at 6:49 AM Ismaël Mejía <ieme...@gmail.com> wrote: > > > +dev@avro.apache.org > > > > On Mon, Jul 15, 2019 at 3:30 PM Ryan Skraba <r...@skraba.com> wrote: > > > > > > Hello! > > > > > > I'm looking for any discussion or reference why the Schema object isn't > > serializable -- I'm pretty sure this must have already been discussed > (but > > the keywords +avro +serializable +schema have MANY results in all the > > searches I did: JIRA, stack overflow, mailing list, web) > > > > > > In particular, I was at a demo today where we were asked why Schemas > > needed to be passed as strings to run in distributed tasks. I remember > > running into this problem years ago with MapReduce, and again in Spark, > and > > again in Beam... > > > > > > Is there any downside to making a Schema implement > > java.lang.Serializable? The only thing I can think of is that the schema > > _should not_ be serialized with the data, and making it non-serializable > > loosely enforces this (at the cost of continually writing different > > flavours of "Avro holders" for when you really do want to serialize it). > > > > > > Willing to create a JIRA and work on the implementation, of course! > > > > > > All my best, Ryan > > >
