Nong, thanks for pointing that out. It is listed in the encoding docs
[2] so it looks like we just weren't implementing the full spec before
this. I didn't have anything else in mind, just wondering how to handle
this. Since it is already in the spec then it is fine to add this to 2.0.
It doesn't matter for this issue any more, but I do want to confirm that
we consider the 2.0 spec closed. We do, right?
rb
[2]: https://github.com/Parquet/parquet-format/blob/master/Encodings.md
On 11/19/2015 10:40 AM, Nong Li wrote:
On Thu, Nov 19, 2015 at 10:31 AM, Ryan Blue <[email protected]> wrote:
I'm working on getting the int64 delta encoding [1] contributed by lunchev
in for 1.9. It's about ready to go, but since it is a new encoding it isn't
a forward-compatible change: data stored in int64 delta can't be read by
readers that can read the entire "parquet 2.0" file format.
What do you mean? I thought this encoding has always been part of the 2.0
spec. I think for an
implementation to claim 2.0, it needs to support this.
So my question is: is the set of things to support for "parquet 2.0"
already released and closed, or do we consider that open for new additions?
Sort of my previous comment. I think the implementations has trailed the
spec for some time so no
one supports 2.0 in my opinion. Is there a change you had in mind? I have
one but I'll start up another
thread.
rb
[1]: https://github.com/apache/parquet-mr/pull/154
--
Ryan Blue
Software Engineer
Cloudera, Inc.
--
Ryan Blue
Software Engineer
Cloudera, Inc.
