etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1842551182
##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
}
```
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that
`LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+ repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
1. If the repeated field is not a group, then its type is the element type and
elements are required.
2. If the repeated field is a group with multiple fields, then its type is the
element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+ field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
or uses the `LIST`-annotated group's name with `_tuple` appended then the
repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
field's repetition.
Examples that can be interpreted using these rules:
```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
optional group my_list (LIST) {
repeated int32 element;
}
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
optional group my_list (LIST) {
repeated group element {
required binary str (STRING);
required int32 num;
};
}
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+ repeated group array (LIST) {
+ repeated int32 array;
+ };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
optional group my_list (LIST) {
repeated group array {
required binary str (STRING);
};
}
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
optional group my_list (LIST) {
repeated group my_list_tuple {
required binary str (STRING);
};
}
+
+// Rule 5: List<OneTuple<List<Integer>>> (nullable outer list, non-null
elements)
+optional group my_list (LIST) {
Review Comment:
Looking back at the original wording of the rules, I think what is now rule
3 would have been handled by the old rule 4 (now rule 5), except the added
wrinkle of the repeated group being named `array` made it ambiguous. The new
rule 3 handles this now, so I'm not sure if rule 5 is reachable any longer.
Rule 1 is for the repeated field not being a group. All other rules apply
when the repeated field is a group. If it's a multi-element group, then it's a
list of struct (rule 2). If it's a nested 2-level list, then list of list (rule
3). If it's 3-level structure with special naming, it's list of one-tuple (rule
4). What other 1-element group is unhandled now? If the repeated group contains
a single `required` or `optional` field, it's a 3-level structure with misnamed
parts. The repeated group cannot contain a single `repeated` field since that
would be mixing 1-level with 3-level.
Am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
