Thanks, Anna and Zoltan.

Overall, I think the test does validate most of the contract, but needs to
be improved. I found a few problems and commented on the commit. I would
also like to see this as a real integration test in Parquet that is run to
test the build. Why make this an external test?

rb

On Mon, Feb 4, 2019 at 8:42 AM Anna Szonyi <[email protected]>
wrote:

> Hi dev@,
>
> As part of acquiring the final sign offs for the parquet release, we had
> another discussion with Ryan about the testing of column indexes.
> He asked us to break down our testing into "fun size" test bites and send
> them out incrementally to the community, so everyone would have time to
> digest and focus on the testing case by case.
>
> We also agreed that we will be focusing on the write path, since that is
> the critical part that we have to support for the long-haul.
>
> I will be sending out the related tests, please feel free to ask clarifying
> questions or if you have a particular test case you want to see next,
> please let me know.
>
> As a first "taste" Zoltan created the following tool to *validate the
> contract* for the indices with random generated data:
>
> - The min value stored in the index must be less than or equal to all
>   values.
> - The max value stored in the index must be greater than or equal to all
>   values.
> - The null count stored in the index must be equal to the number of
>   nulls.
> - Only pages consisting entirely of NULL-s can be marked as a null page
>   in the index.
> - According to the ASCENDING boundary order, the min value for a page
>   must be greater than or equal to the min value of the previous page.
> - According to the ASCENDING boundary order, the max value for a page
>   must be greater than or equal to the max value of the previous page.
> - According to the DESCENDING boundary order, the min value for a page
>   must be less than or equal to the min value of the previous page.
> - According to the DESCENDING boundary order, the max value for a page
>   must be less than or equal to the max value of the previous page.
>
>
> https://github.com/zivanfi/parquet-mr/commit/0e74b4207daa0b53cf4bd0774d2632c388845cb9
> The code itself is unpolished, but fully functional and complete.
> Once the release is signed off we plan to refactor this and offer it as
> part of parquet tools or parquet cli,
> however it is perfectly fine for validation as-is.
>
> Best,
> Anna
>


-- 
Ryan Blue
Software Engineer
Netflix

Reply via email to