alamb commented on code in PR #119:
URL: https://github.com/apache/parquet-testing/pull/119#discussion_r3711695566
##########
data/README.md:
##########
@@ -589,3 +590,56 @@ java -jar
parquet-cli/target/parquet-cli-1.16.0-SNAPSHOT-runtime.jar cat /home/r
{"utf8_full_truncation": "Julia Roberts", "binary_full_truncation": "Julia
Roberts", "utf8_partial_truncation": "Julia Roberts",
"binary_partial_truncation": "Julia Roberts", "utf8_no_truncation": "Julia
Roberts", "binary_no_truncation": "Julia Roberts"}
{"utf8_full_truncation": "Kevin Bacon", "binary_full_truncation": "Kevin
Bacon", "utf8_partial_truncation": "🚀Kevin Bacon", "binary_partial_truncation":
"ÿÿ\u0001\u0002", "utf8_no_truncation": "Ke", "binary_no_truncation": "Ke"}
```
+
+## ALP encoding
+
+`alp_extended.zstd.parquet` contains FLOAT and DOUBLE columns encoded with
+[Adaptive Lossless floating-Point
(ALP)](https://github.com/apache/parquet-format/blob/master/Encodings.md#adaptive-lossless-floating-point-alp--10)
+(`ALP = 10` in the parquet.thrift `Encoding` enum).
+It was created with the code in this
[PR](https://github.com/apache/arrow/pull/49154).
+
+The same values appear in `PLAIN`-encoded reference columns so decoders can
+bit-compare the `ALP` columns against known results:
+
+| Column | Encoding | Rationale
/ coverage |
Review Comment:
I tried to explain both what is in the file, as well as to explain the
rationale for including each piece
##########
data/alp_extended.zstd.parquet:
##########
Review Comment:
You can verify this a bit with standard tools. For example, using the tools
from the Rust implementation in https://github.com/apache/arrow-rs/pull/9372
```shell
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$
~/Software/arrow-rs/target/release/parquet-schema data/alp_extended.zstd.parquet
Metadata for file: data/alp_extended.zstd.parquet
version: 2
num of rows: 9000
created by: parquet-cpp-arrow version 23.0.0-SNAPSHOT
message schema {
OPTIONAL FLOAT float_plain;
OPTIONAL DOUBLE double_plain;
OPTIONAL FLOAT float_alp_1024;
OPTIONAL DOUBLE double_alp_1024;
OPTIONAL FLOAT float_alp_4096;
OPTIONAL DOUBLE double_alp_4096;
OPTIONAL FLOAT float_alp_32;
OPTIONAL DOUBLE double_alp_32;
}
```
##########
data/alp_extended.zstd.parquet:
##########
Review Comment:
The layout:
```shell
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$
~/Software/arrow-rs/target/release/parquet-layout
data/alp_extended.zstd.parquet```
```json
{
"row_groups": [
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211391,
"length": 12
},
"column_index": {
"offset": 210367,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211403,
"length": 14
},
"column_index": {
"offset": 210395,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211417,
"length": 14
},
"column_index": {
"offset": 210431,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211431,
"length": 14
},
"column_index": {
"offset": 210459,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211445,
"length": 14
},
"column_index": {
"offset": 210495,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211459,
"length": 14
},
"column_index": {
"offset": 210523,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211473,
"length": 14
},
"column_index": {
"offset": 210559,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211487,
"length": 14
},
"column_index": {
"offset": 210587,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 6144
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211501,
"length": 13
},
"column_index": {
"offset": 210623,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211514,
"length": 13
},
"column_index": {
"offset": 210651,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211527,
"length": 13
},
"column_index": {
"offset": 210687,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211540,
"length": 13
},
"column_index": {
"offset": 210715,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211553,
"length": 13
},
"column_index": {
"offset": 210751,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211566,
"length": 13
},
"column_index": {
"offset": 210779,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211579,
"length": 13
},
"column_index": {
"offset": 210815,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211592,
"length": 13
},
"column_index": {
"offset": 210843,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 1024
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211605,
"length": 12
},
"column_index": {
"offset": 210879,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211617,
"length": 12
},
"column_index": {
"offset": 210907,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211629,
"length": 12
},
"column_index": {
"offset": 210943,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211641,
"length": 12
},
"column_index": {
"offset": 210971,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211653,
"length": 12
},
"column_index": {
"offset": 211007,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211665,
"length": 12
},
"column_index": {
"offset": 211035,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211677,
"length": 13
},
"column_index": {
"offset": 211071,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211690,
"length": 13
},
"column_index": {
"offset": 211099,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 1024
},
{
"columns": [
{
"path": "float_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211703,
"length": 13
},
"column_index": {
"offset": 211135,
"length": 28
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "double_plain",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211716,
"length": 13
},
"column_index": {
"offset": 211163,
"length": 36
},
"bloom_filter": null,
"compression": "ZSTD",
"encodings": [
"PLAIN",
"RLE"
]
},
{
"path": "float_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211729,
"length": 13
},
"column_index": {
"offset": 211199,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_1024",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211742,
"length": 13
},
"column_index": {
"offset": 211227,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211755,
"length": 13
},
"column_index": {
"offset": 211263,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_4096",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211768,
"length": 13
},
"column_index": {
"offset": 211291,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "float_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211781,
"length": 13
},
"column_index": {
"offset": 211327,
"length": 28
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
},
{
"path": "double_alp_32",
"has_offset_index": true,
"has_column_index": true,
"has_bloom_filter": false,
"offset_index": {
"offset": 211794,
"length": 13
},
"column_index": {
"offset": 211355,
"length": 36
},
"bloom_filter": null,
"compression": "UNCOMPRESSED",
"encodings": [
"RLE",
"ALP"
]
}
],
"row_count": 808
}
],
"footer": {
"metadata_size": 3866
}
}
```
##########
data/README.md:
##########
@@ -19,50 +19,51 @@
# Test data files for Parquet compatibility and regression testing
-| File | Description
|
-|----------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| delta_byte_array.parquet | string columns with
DELTA_BYTE_ARRAY encoding. See [delta_byte_array.md](delta_byte_array.md) for
details. |
-| delta_length_byte_array.parquet | string columns with
DELTA_LENGTH_BYTE_ARRAY encoding.
|
-| delta_binary_packed.parquet | INT32 and INT64 columns with
DELTA_BINARY_PACKED encoding. See
[delta_binary_packed.md](delta_binary_packed.md) for details.
|
-| delta_encoding_required_column.parquet | required INT32 and STRING
columns with delta encoding. See
[delta_encoding_required_column.md](delta_encoding_required_column.md) for
details. |
-| delta_encoding_optional_column.parquet | optional INT64 and STRING
columns with delta encoding. See
[delta_encoding_optional_column.md](delta_encoding_optional_column.md) for
details. |
-| nested_structs.rust.parquet | Used to test that the Rust
Arrow reader can lookup the correct field from a nested struct. See
[ARROW-11452](https://issues.apache.org/jira/browse/ARROW-11452) |
-| data_index_bloom_encoding_stats.parquet | optional STRING column. Contains
optional metadata: bloom filters, column index, offset index and encoding
stats. |
-| data_index_bloom_encoding_with_length.parquet | Same as
`data_index_bloom_encoding_stats.parquet` but has `bloom_filter_length`
populated in the ColumnMetaData |
-| null_list.parquet | an empty list. Generated from this
json `{"emptylist":[]}` and for the purposes of testing correct read/write
behaviour of this base case. |
-| alltypes_tiny_pages.parquet | small page sizes with dictionary
encoding with page index from
[impala](https://github.com/apache/impala/tree/master/testdata/data/alltypes_tiny_pages.parquet).
|
-| alltypes_tiny_pages_plain.parquet | small page sizes with plain
encoding with page index
[impala](https://github.com/apache/impala/tree/master/testdata/data/alltypes_tiny_pages.parquet).
|
-| rle_boolean_encoding.parquet | option boolean columns with RLE
encoding
|
-| fixed_length_byte_array.parquet | optional
FIXED_LENGTH_BYTE_ARRAY column with page index. See
[fixed_length_byte_array.md](fixed_length_byte_array.md) for details.
|
-| int32_with_null_pages.parquet | optional INT32 column with
random null pages. See [int32_with_null_pages.md](int32_with_null_pages.md) for
details. |
-| datapage_v1-uncompressed-checksum.parquet | uncompressed INT32 columns
in v1 data pages with a matching CRC |
-| datapage_v1-snappy-compressed-checksum.parquet | compressed INT32 columns in
v1 data pages with a matching CRC |
-| datapage_v1-corrupt-checksum.parquet | uncompressed INT32 columns
in v1 data pages with a mismatching CRC |
-| overflow_i16_page_cnt.parquet | row group with more than
INT16_MAX pages |
-| bloom_filter.bin | deprecated bloom filter
binary with binary header and murmur3 hashing |
-| bloom_filter.xxhash.bin | bloom filter binary with
thrift header and xxhash hashing |
-| nan_in_stats.parquet | statistics contains NaN in
max, from PyArrow 0.8.0. See note below on "NaN in stats". |
-| rle-dict-snappy-checksum.parquet | compressed and
dictionary-encoded INT32 and STRING columns in format v2 with a matching CRC |
-| plain-dict-uncompressed-checksum.parquet | uncompressed and
dictionary-encoded INT32 and STRING columns in format v1 with a matching CRC |
-| rle-dict-uncompressed-corrupt-checksum.parquet | uncompressed and
dictionary-encoded INT32 and STRING columns in format v2 with a mismatching CRC
|
-| large_string_map.brotli.parquet | MAP(STRING, INT32) with a string
column chunk of more than 2GB. See [note](#large-string-map) below |
-| float16_nonzeros_and_nans.parquet | Float16 (logical type) column with NaNs
and nonzero finite min/max values |
-| float16_zeros_and_nans.parquet | Float16 (logical type) column with NaNs
and zeros as min/max values. . See [note](#float16-files) below |
-| floating_orders_nan_count.parquet | FLOAT/DOUBLE/FLOAT16 columns in IEEE754
and TypeDefined orders across five row groups (no-NaN, mixed-NaN, all-NaN,
zero-min, zero-max) to validate nan_count, signed NaNs, and IEEE-754 zero
ordering in statistics and column index |
-| concatenated_gzip_members.parquet | 513 UINT64 numbers compressed using
2 concatenated gzip members in a single data page |
-| byte_stream_split.zstd.parquet | Standard normals with `BYTE_STREAM_SPLIT`
encoding. See [note](#byte-stream-split) below |
-| incorrect_map_schema.parquet | Contains a Map schema without explicitly
required keys, produced by Presto. See [note](#incorrect-map-schema) |
-| column_chunk_key_value_metadata.parquet | two INT32 columns, one with column
chunk key-value metadata {"foo": "bar", "thisiskeywithoutvalue": null} note
that the second key "thisiskeywithoutvalue", does not have a value, but the
value can be mapped to an empty string "" when read depending on the client |
-| sort_columns.parquet | INT64 and BYTE_ARRAY columns with first column with
nulls first and descending, second column with nulls last and ascending. This
file contains two row groups with same data and sorting columns. |
-| old_list_structure.parquet | Single LIST<LIST<INT32>> column with legacy
two-level list structure. See [old_list_structure.md](old_list_structure.md) |
-| repeated_primitive_no_list.parquet | REPEATED INT32 and BYTE_ARRAY fields
without LIST annotation. See
[note](#REPEATED-primitive-fields-with-no-LIST-annotation) |
-| map_no_value.parquet | MAP with null values, MAP with INT32 keys and no
values, and LIST<INT32> column with same values as the MAP keys. See
[map_no_value.md](map_no_value.md) |
-| page_v2_empty_compressed.parquet | An INT32 column with DataPageV2, all
values are null, the zero-sized data is compressed using ZSTD. This is a valid
non-zero bytes ZSTD stream that uncompresses into 0 bytes. |
-| datapage_v2_empty_datapage.snappy.parquet | A compressed FLOAT column with
DataPageV2, a single row, value is null, the file uses Snappy compression, but
there is no data for uncompression (see [related
issue](https://github.com/apache/arrow-rs/issues/7388)). The zero bytes must
not be attempted to be uncompressed, as this is an invalid Snappy stream. |
-| unknown-logical-type.parquet | A file containing a column annotated with a
LogicalType whose identifier has been set to an abitrary high value to check
the behaviour of an old reader reading a file written by a new writer
containing an unsupported type (see [related
issue](https://github.com/apache/arrow/issues/41764)). |
-| int96_from_spark.parquet | Single column of (deprecated) int96 values that
originated as Apache Spark microsecond-resolution timestamps. Some values are
outside the range typically representable by 64-bit nanosecond-resolution
timestamps. See [int96_from_spark.md](int96_from_spark.md) for details. |
+| File | Description
|
+|----------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| delta_byte_array.parquet | string columns with
DELTA_BYTE_ARRAY encoding. See [delta_byte_array.md](delta_byte_array.md) for
details.
|
+| delta_length_byte_array.parquet | string columns with
DELTA_LENGTH_BYTE_ARRAY encoding.
|
+| delta_binary_packed.parquet | INT32 and INT64 columns with
DELTA_BINARY_PACKED encoding. See
[delta_binary_packed.md](delta_binary_packed.md) for details.
|
+| delta_encoding_required_column.parquet | required INT32 and STRING
columns with delta encoding. See
[delta_encoding_required_column.md](delta_encoding_required_column.md) for
details.
|
+| delta_encoding_optional_column.parquet | optional INT64 and STRING
columns with delta encoding. See
[delta_encoding_optional_column.md](delta_encoding_optional_column.md) for
details.
|
+| nested_structs.rust.parquet | Used to test that the Rust
Arrow reader can lookup the correct field from a nested struct. See
[ARROW-11452](https://issues.apache.org/jira/browse/ARROW-11452)
|
+| data_index_bloom_encoding_stats.parquet | optional STRING column. Contains
optional metadata: bloom filters, column index, offset index and encoding
stats.
|
+| data_index_bloom_encoding_with_length.parquet | Same as
`data_index_bloom_encoding_stats.parquet` but has `bloom_filter_length`
populated in the ColumnMetaData
|
+| null_list.parquet | an empty list. Generated from this
json `{"emptylist":[]}` and for the purposes of testing correct read/write
behaviour of this base case.
|
+| alltypes_tiny_pages.parquet | small page sizes with dictionary
encoding with page index from
[impala](https://github.com/apache/impala/tree/master/testdata/data/alltypes_tiny_pages.parquet).
|
+| alltypes_tiny_pages_plain.parquet | small page sizes with plain
encoding with page index
[impala](https://github.com/apache/impala/tree/master/testdata/data/alltypes_tiny_pages.parquet).
|
+| rle_boolean_encoding.parquet | option boolean columns with RLE
encoding
|
+| fixed_length_byte_array.parquet | optional
FIXED_LENGTH_BYTE_ARRAY column with page index. See
[fixed_length_byte_array.md](fixed_length_byte_array.md) for details.
|
+| int32_with_null_pages.parquet | optional INT32 column with
random null pages. See [int32_with_null_pages.md](int32_with_null_pages.md) for
details.
|
+| datapage_v1-uncompressed-checksum.parquet | uncompressed INT32 columns
in v1 data pages with a matching CRC
|
+| datapage_v1-snappy-compressed-checksum.parquet | compressed INT32 columns in
v1 data pages with a matching CRC
|
+| datapage_v1-corrupt-checksum.parquet | uncompressed INT32 columns
in v1 data pages with a mismatching CRC
|
+| overflow_i16_page_cnt.parquet | row group with more than
INT16_MAX pages
|
+| bloom_filter.bin | deprecated bloom filter
binary with binary header and murmur3 hashing
|
+| bloom_filter.xxhash.bin | bloom filter binary with
thrift header and xxhash hashing
|
+| nan_in_stats.parquet | statistics contains NaN in
max, from PyArrow 0.8.0. See note below on "NaN in stats".
|
+| rle-dict-snappy-checksum.parquet | compressed and
dictionary-encoded INT32 and STRING columns in format v2 with a matching CRC
|
+| plain-dict-uncompressed-checksum.parquet | uncompressed and
dictionary-encoded INT32 and STRING columns in format v1 with a matching CRC
|
+| rle-dict-uncompressed-corrupt-checksum.parquet | uncompressed and
dictionary-encoded INT32 and STRING columns in format v2 with a mismatching CRC
|
+| large_string_map.brotli.parquet | MAP(STRING, INT32) with a string
column chunk of more than 2GB. See [note](#large-string-map) below
|
+| float16_nonzeros_and_nans.parquet | Float16 (logical type) column with NaNs
and nonzero finite min/max values
|
+| float16_zeros_and_nans.parquet | Float16 (logical type) column with NaNs
and zeros as min/max values. . See [note](#float16-files) below
|
+| floating_orders_nan_count.parquet | FLOAT/DOUBLE/FLOAT16 columns in IEEE754
and TypeDefined orders across five row groups (no-NaN, mixed-NaN, all-NaN,
zero-min, zero-max) to validate nan_count, signed NaNs, and IEEE-754 zero
ordering in statistics and column index
|
+| concatenated_gzip_members.parquet | 513 UINT64 numbers compressed using
2 concatenated gzip members in a single data page
|
+| byte_stream_split.zstd.parquet | Standard normals with `BYTE_STREAM_SPLIT`
encoding. See [note](#byte-stream-split) below
|
+| incorrect_map_schema.parquet | Contains a Map schema without explicitly
required keys, produced by Presto. See [note](#incorrect-map-schema)
|
+| column_chunk_key_value_metadata.parquet | two INT32 columns, one with column
chunk key-value metadata {"foo": "bar", "thisiskeywithoutvalue": null} note
that the second key "thisiskeywithoutvalue", does not have a value, but the
value can be mapped to an empty string "" when read depending on the client
|
+| sort_columns.parquet | INT64 and BYTE_ARRAY columns with first column with
nulls first and descending, second column with nulls last and ascending. This
file contains two row groups with same data and sorting columns.
|
+| old_list_structure.parquet | Single LIST<LIST<INT32>> column with legacy
two-level list structure. See [old_list_structure.md](old_list_structure.md)
|
+| repeated_primitive_no_list.parquet | REPEATED INT32 and BYTE_ARRAY fields
without LIST annotation. See
[note](#REPEATED-primitive-fields-with-no-LIST-annotation)
|
+| map_no_value.parquet | MAP with null values, MAP with INT32 keys and no
values, and LIST<INT32> column with same values as the MAP keys. See
[map_no_value.md](map_no_value.md)
|
+| page_v2_empty_compressed.parquet | An INT32 column with DataPageV2, all
values are null, the zero-sized data is compressed using ZSTD. This is a valid
non-zero bytes ZSTD stream that uncompresses into 0 bytes.
|
+| datapage_v2_empty_datapage.snappy.parquet | A compressed FLOAT column with
DataPageV2, a single row, value is null, the file uses Snappy compression, but
there is no data for uncompression (see [related
issue](https://github.com/apache/arrow-rs/issues/7388)). The zero bytes must
not be attempted to be uncompressed, as this is an invalid Snappy stream.
|
+| unknown-logical-type.parquet | A file containing a column annotated with a
LogicalType whose identifier has been set to an abitrary high value to check
the behaviour of an old reader reading a file written by a new writer
containing an unsupported type (see [related
issue](https://github.com/apache/arrow/issues/41764)).
|
+| int96_from_spark.parquet | Single column of (deprecated) int96 values that
originated as Apache Spark microsecond-resolution timestamps. Some values are
outside the range typically representable by 64-bit nanosecond-resolution
timestamps. See [int96_from_spark.md](int96_from_spark.md) for details.
|
| int96_timestamp_order.parquet | Single `required int96` column written with
the `INT96_TIMESTAMP_ORDER` column order ([parquet-format
#584](https://github.com/apache/parquet-format/pull/584)). Values are chosen so
a byte-wise comparison disagrees with the chronological order, so the min/max
statistics (and column index) are only correct for a reader that honors the new
order. See [int96_timestamp_order.md](int96_timestamp_order.md) for details. |
-| binary_truncated_min_max.parquet | A file containing six columns with exact,
fully-truncated and partially-truncated max and min statistics and with the
expected is_{min/max}_value_exact. (see
[note](Binary-truncated-min-and-max-statistics)).|
+| binary_truncated_min_max.parquet | A file containing six columns with exact,
fully-truncated and partially-truncated max and min statistics and with the
expected is_{min/max}_value_exact. (see
[note](Binary-truncated-min-and-max-statistics)).
|
+| alp_extended.zstd.parquet | FLOAT and DOUBLE columns encoded using Adaptive
Lossless floating-Point (ALP). See [note](#alp-encoding) below
|
Review Comment:
this is the new line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
