Perhaps I don’t understand you. What is “a recursive record”? AFAIU real
data can not be recursive. (I would love to be shown to be wrong about
this.)
Real data is very recursive, consider the canonical definitons for
List/Tree, and anything with cons/cdr
https://en.wikipedia.org/wiki/Cons
JSON
(Thanks for the [Discuss] tip.)
Recursion is primarily a code-maintenance problem. I'm not sure how
to quantify the complexity, but certainly Schema.java itself has a lot
of logic in it to deal with recursion, as do all the "Grammar
Generator" classes plus Generic Data -- the classes that are
In the future please use "[DISCUSS]" at the start of your subject line
for these kinds of proposals. that'll get more folks to see the
discussion, e.g. when they filter this list.
as a point of clarification, 1.9.0 is a major version change for the
Avro project. the "1" is a file format version.
> using Avro
to store millions to billions of non-recursive records
Perhaps I don’t understand you. What is “a recursive record”? AFAIU real
data can not be recursive. (I would love to be shown to be wrong about
this.) Given any real data, proportional coffee, sandwiches and time, I can
write an
(Keep the following in mind: perhaps 95%+ of Avro users do not depend
on recursion but don't understand the opportunity costs of maintaining
it (and thus won't speak up on this thread); the remaining 5% who
depend on recursion are highly motivated to speak out against it's
removal. In such a
I think that recursive schemas are a powerful feature of Avro that enable
the modelling of hierarchies,
a pretty common data structure. l myself have used this feature in a large
scale system before.
The recursion is handled elegantly and naturally in the code with recursive
functions so I don't
I've used recursion in the past to use Avro to get a binary representation
of JSON. Given the popularity of JSON and the fact that Avro includes
support for converting it, I think it makes sense to continue allowing
recursive schemas.
On Fri, Nov 30, 2018 at 2:12 PM Michael A. Smith
wrote:
>
I’m against this proposal. Sure, recursion adds complexity, but recursive
types are also extremely powerful and one of the most interesting features
of a tool like this. I have been experimenting in the other direction;
considering a way to compose avro schema descriptions in avro. Recursion is
First, this creates a data incompatibility, not just an API
incompatibility, so it should not be permitted in 1.9.0. Apps that worked,
even when updated for API changes, will not be able to read data they could
before they upgraded.
Second, folks might actually use this feature in reasonable
I understand we've been willing to introduce backward-incompatible API
changes (not file-format changes) into minor release versions. If so,
here's an idea for consideration:
Let's eliminate recursive records from Avro 1.9.x. Recursion
introduces a _lot_ of complexity into many parts of the
10 matches
Mail list logo
