notes:
Agenda and attendees:
- Anuj Phadke (impala team)
- Uwe (Blue Yonder, parquet-cpp):
-
- Discuss parquet dotnet project
- Lars (Impala):
-
- timestamp int96. Deprecate ordering
- Nandor (File format team in Cloudera)
- Zoltan (Cloudera):
-
- discuss page size recommendation
- Gabor (file formats)
- Ryan (Netflix):
-
- Been working on better read api. Rewrite record construction in
parquet-avro + 5%.
- Discuss PARQUET-787 (how we build the decoders for byte arrays)
- Marcel
- Julien (Wework):
-
- releases
Agenda:
- Deprecating ordering for int96 timestamp:
https://issues.apache.org/jira/browse/PARQUET-1065
<https://meet.google.com/linkredirect?authuser=0&dest=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FPARQUET-1065>
-
- It was decided to deprecate ordering for int96.
-
- Do not use existing min/max stat for int96
- Label int96 as not supporting ordering
- Do not write int96 anymore
- Always support reading int96 for backward compatibility with
existing files
- PR from Zoltan. Change to change int96 ordering from Unsigned to
undefined.
- Lars: Impala actually uses int96 min/max and ordering and will do
it for some time.
- Conclusion:
-
- Add language to say writing int96 is not allowed (caveat that
people can do things anyway). People should use 64bits
timestamps instead.
- Add spec of int96 to doc with warning that the only purpose is
to enable reading existing files.
- PRs:
-
- https://github.com/apache/parquet-format/pull/77
- https://github.com/apache/parquet-format/pull/49
- Action: lars to update #49
- Parquet dotnet project
-
- Discussion on wether we should import it in the apache parquet
project
- General advice is to make sure the authors are engaged enough with
the project to maintain it long term.
- We should Keep reaching out and support this effort
- Page size reco
-
- Zoltan: Create a JIRA.
- Wait for page skipping implementation to get numbers on the impact
of page size
- Look at different strategies for page size (bytes before
compression, #values, ...)
- Make some measurements
- Restart the conversation
- PARQUET-787: needs a review
https://github.com/apache/parquet-mr/pull/390
- Releases
-
- Ryan: create release jira
On Tue, Jan 9, 2018 at 8:54 AM, Julien Le Dem <[email protected]>
wrote:
> The sync is starting in a few minutes:
> https://meet.google.com/cxa-nppv-caa
> (as a reminder, everybody is welcome to join if only to be a fly on the
> wall)
>
> On Tue, Jan 9, 2018 at 2:31 AM, Lars Volker <[email protected]> wrote:
>
>> Great, I sent out an invite. If anyone wants to join but was not on the
>> invite, please let me know.
>>
>> Cheers, Lars
>>
>> On Mon, Jan 8, 2018 at 10:24 PM, Julien Le Dem <[email protected]>
>> wrote:
>>
>> > It sounds like we're doing the parquet sync tomorrow Tuesday January
>> 9th at
>> > 9am PT (5pm UTC)
>> >
>> > On Thu, Jan 4, 2018 at 9:17 AM, Marcel Kornacker <[email protected]>
>> > wrote:
>> >
>> > > My preference for next week would be Tuesday as well.
>> > >
>> > > On Thu, Jan 4, 2018 at 8:25 AM, Zoltan Ivanfi <[email protected]>
>> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > According to the latest results of the availability poll, Tuesdays
>> > seems
>> > > > to work for slightly more people than Wednesdays. I'll try to post
>> the
>> > > > chart below, let's see whether the mailing list allows it or removes
>> > it:
>> > > > [image: pasted1]
>> > > >
>> > > > I would suggest to either use Tuesdays or alternate between Tuesdays
>> > and
>> > > > Wednesdays (since the group of 9 Tuesday voters does not contain
>> all 8
>> > > > Wednesday voters). The last sync was on Tuesdays, so the next can
>> be on
>> > > > Wednesday if you would like to follow this alternating scheme.
>> > > >
>> > > > Best regards,
>> > > >
>> > > > Zoltan
>> > > >
>> > > >
>> > > > On Thu, Jan 4, 2018 at 4:27 PM Wes McKinney <[email protected]>
>> > wrote:
>> > > >
>> > > >> We have been staggering the Arrow syncs by 1 week, also on
>> Wednesdays
>> > > >> at 9am PT. If you are going to have the next Parquet sync on 1/10,
>> we
>> > > >> would have the next Arrow sync on 1/17. Let me know what you prefer
>> > > >>
>> > > >> On Thu, Jan 4, 2018 at 4:10 AM, Lars Volker <[email protected]>
>> wrote:
>> > > >> > 1/10 would work for me.
>> > > >> >
>> > > >> > On Thu, Jan 4, 2018 at 3:22 AM, Julien Le Dem <
>> > [email protected]
>> > > >
>> > > >> > wrote:
>> > > >> >
>> > > >> >> Any day of the week/time preference for the next Parquet sync?
>> > > >> >> It is usually held at 9am PT (5pm UTC) on a Wednesday.
>> > > >> >>
>> > > >>
>> > > >
>> > >
>> >
>>
>
>
