(Keep the following in mind: perhaps 95%+ of Avro users do not depend on recursion but don't understand the opportunity costs of maintaining it (and thus won't speak up on this thread); the remaining 5% who depend on recursion are highly motivated to speak out against it's removal. In such a world, I'm speaking for 95% of the Avro community :-)
So far, the main justification of retaining recursion in Avro seems to be that it allows Avro to be a binary representation of JSON using the schema "share/schemas/org/apache/avro/data/Json.avsc". This justification is a bit odd. The founding philosophy of Avro is that data should have schemas (and those schemas should be able to evolve). The Avro-as-binary-rep-for-JSON argument is really the following: "recursion in Avro is good because it allows us to model JSON which allows us to model data with no schema." Grumble. But let's leave aside philosophical arguments regarding static vs dynamic typing. Let's consider communities. Json.avsc was committed in 2011 and hasn't changed since. Clients who are using Avro as a binary representation of JSON can continue to depend on the 1.8.x line of Avro. If that community of users is big-enough/mission-critical enough to maintain 1.8.x going forward, then all power to them. 1.8.x can live forever. In the meantime, for the 95% of the Avro community that is using Avro to store millions to billions of non-recursive records, and who are tired with putting up with the (opportunity) costs of supporting recursion that they never use, let's move on. On Fri, Nov 30, 2018 at 2:33 PM Ryan Blue <[email protected]> wrote: > > I've used recursion in the past to use Avro to get a binary representation > of JSON. Given the popularity of JSON and the fact that Avro includes > support for converting it, I think it makes sense to continue allowing > recursive schemas. > > On Fri, Nov 30, 2018 at 2:12 PM Michael A. Smith <[email protected]> > wrote: > > > I’m against this proposal. Sure, recursion adds complexity, but recursive > > types are also extremely powerful and one of the most interesting features > > of a tool like this. I have been experimenting in the other direction; > > considering a way to compose avro schema descriptions in avro. Recursion is > > crucial for that, so that we can lay out possible types as a union of named > > types that include themselves. > > > > Granted, it’s an experiment that would also imply a compatibility break > > with previous avros, but it opens what I think is an interesting set of > > doors instead of closing them. > > > > My 2 cents. > > > > On Fri, Nov 30, 2018 at 13:22 Raymie Stata <[email protected]> wrote: > > > > > I understand we've been willing to introduce backward-incompatible API > > > changes (not file-format changes) into minor release versions. If so, > > > here's an idea for consideration: > > > > > > Let's eliminate recursive records from Avro 1.9.x. Recursion > > > introduces a _lot_ of complexity into many parts of the Avro code > > > base. We could vastly simplify the code base, and probably speed > > > things up, by getting rid of this feature. > > > > > > The specific proposal would be that Avro 1.9.x would refuse to accept > > > recursive records, and thus would not be able to read binary files > > > written by older versions of Avro. I haven't heard of anyone actually > > > using them, so I don't think this would be a problem. > > > > > > > > -- > Ryan Blue > Software Engineer > Netflix
