nbauernfeind commented on a change in pull request #11646:
URL: https://github.com/apache/arrow/pull/11646#discussion_r745794854
##########
File path: format/Message.fbs
##########
@@ -117,6 +117,40 @@ table DictionaryBatch {
isDelta: bool = false;
}
+/// A range of field nodes, identified by their offset in the schema.
+/// The offsets are zero-indexed.
+struct FieldNodeRange {
+ /// The starting offset (inclusive)
+ start: long;
+
+ /// The ending offset (exclusive)
+ end: long;
+}
+
+/// A data header describing the shared memory layout of a "bag" of "columns".
+/// It is similar to a RecordBatch but not every top level FieldNode is
required
+/// to be included in the wire payload.
+table ColumnBag {
+ /// If not provided, all field nodes are included and this payload is
+ /// identical to a RecordBatch. Otherwise the reader needs to skip
+ /// top level FieldNodes that were not included.
+ includedNodes: [FieldNodeRange];
Review comment:
Ooh. I wish there was a way to tell. I don't think you can tell that a
message is a degenerate. The FieldNode themselves do not identify which node or
location in the schema it belongs to. They are expected to be visited in-order
(parent, children, grandchildren; then the parent's siblings -- the other root
field nodes, etc). The point of FieldNodeRange is that this is the list of top
level field nodes only -- not including nested nodes. So, as typical, it is
impossible to tell if the message was written incorrectly. You typically only
find out from out-of-memory access where a node thinks it's an array but it's
really just a column of uint64.
Point is, I think I'm preventing this by not being able to describe it on
the wire.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
