lidavidm commented on a change in pull request #11646:
URL: https://github.com/apache/arrow/pull/11646#discussion_r745843060
##########
File path: format/Message.fbs
##########
@@ -117,6 +117,40 @@ table DictionaryBatch {
isDelta: bool = false;
}
+/// A range of field nodes, identified by their offset in the schema.
+/// The offsets are zero-indexed.
+struct FieldNodeRange {
+ /// The starting offset (inclusive)
+ start: long;
+
+ /// The ending offset (exclusive)
+ end: long;
+}
+
+/// A data header describing the shared memory layout of a "bag" of "columns".
+/// It is similar to a RecordBatch but not every top level FieldNode is
required
+/// to be included in the wire payload.
+table ColumnBag {
+ /// If not provided, all field nodes are included and this payload is
+ /// identical to a RecordBatch. Otherwise the reader needs to skip
+ /// top level FieldNodes that were not included.
+ includedNodes: [FieldNodeRange];
Review comment:
Yes, thanks.
I think consumers can validate, so long as they have the schema: they would
traverse the schema as usual, and look at both nodes and includedNodes. If
while traversing a type they ran into a buffer not included in includedNodes,
then the message is invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
