This is an automated email from the ASF dual-hosted git repository.
maplefu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/parquet-testing.git
The following commit(s) were added to refs/heads/master by this push:
new e7b28dd Add primitive_null metadata for example Variant values (#84)
e7b28dd is described below
commit e7b28dd520fb3c9f0908daa84c0ef20d83c73794
Author: Andrew Lamb <[email protected]>
AuthorDate: Fri May 23 08:02:17 2025 -0400
Add primitive_null metadata for example Variant values (#84)
* Add primitive_null metadata
* Update variant/README.md
Co-authored-by: Fokko Driesprong <[email protected]>
* Update variant/README.md
---------
Co-authored-by: Fokko Driesprong <[email protected]>
---
variant/README.md | 18 ++++++++++++++++--
variant/primitive_null.metadata | Bin 0 -> 3 bytes
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/variant/README.md b/variant/README.md
index 71e4da8..e335caf 100644
--- a/variant/README.md
+++ b/variant/README.md
@@ -45,8 +45,22 @@ Each example consists of 2 files:
## Regenerating these files
-The files were generated by running the [`regen.py`](regen.py) script that
uses Apache Spark to
-generate the files.
+The files in this directory were initially generated by running the
[`regen.py`](regen.py)
+script which used Apache Spark to generate the files. The files have been
subsequently modified
+when necessary to ensure that they conform to the Parquet spec.
+
+### Modification 1: Created metadata for `primitive_null` as a single byte
(`0x01`)
+
+Per <https://github.com/apache/parquet-testing/issues/81>, Spark did not
generate
+any metadata for `null` and left `primitive_null.metadata` empty.
+The metadata for `primitive_null` should be the same 3 bytes as other
primitive types
+* header = `0x01`
+* dictionary_size = `0x00`
+* `dictionary_size + 1 = 1` byte values: `0x00`
+
+```shell
+cp primitive_int8.metadata primitive_null.metadata
+```
[Variant]:
https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
[primitive types listed in the spec]:
https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#value-data-for-primitive-type-basic_type0
diff --git a/variant/primitive_null.metadata b/variant/primitive_null.metadata
index e69de29..12db478 100644
Binary files a/variant/primitive_null.metadata and
b/variant/primitive_null.metadata differ
