[
https://issues.apache.org/jira/browse/ARROW-5916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884058#comment-16884058
]
Wes McKinney edited comment on ARROW-5916 at 7/12/19 6:08 PM:
--------------------------------------------------------------
I think that the fields' length needs to be respected in {{GetFieldMetadata}}
(since this can vary for nested fields). The appropriate fix is probably here:
https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/reader.cc#L368
If the arrays come back from {{LoadArray}} longer than {{num_rows}}, then the
record batch should be constructed then sliced.
was (Author: wesmckinn):
I think that the fields' length needs to be respected in {{GetFieldMetadata}}
(since this can vary for nested fields). The appropriate fix is probably here:
> [C++] Allow RecordBatch.length to be less than array lengths
> ------------------------------------------------------------
>
> Key: ARROW-5916
> URL: https://issues.apache.org/jira/browse/ARROW-5916
> Project: Apache Arrow
> Issue Type: New Feature
> Reporter: John Muehlhausen
> Priority: Minor
> Attachments: test.arrow_ipc
>
>
> 0.13 ignored RecordBatch.length. 0.14 requires that RecordBatch.length and
> array length be equal. As per
> [https://lists.apache.org/thread.html/2692dd8fe09c92aa313bded2f4c2d4240b9ef75a8604ec214eb02571@%3Cdev.arrow.apache.org%3E]
> , we discussed changing this so that RecordBatch.length can be [0,array
> length].
> If RecordBatch.length is less than array length, the reader should ignore
> the portion of the array(s) beyond RecordBatch.length. This will allow
> partially populated batches to be read in scenarios identified in the above
> discussion.
> {code:c++}
> Status GetFieldMetadata(int field_index, ArrayData* out) {
> auto nodes = metadata_->nodes();
> // pop off a field
> if (field_index >= static_cast<int>(nodes->size())) {
> return Status::Invalid("Ran out of field metadata, likely malformed");
> }
> const flatbuf::FieldNode* node = nodes->Get(field_index);
> * //out->length = node->length();*
> * out->length = metadata_->length();*
> out->null_count = node->null_count();
> out->offset = 0;
> return Status::OK();
> }
> {code}
> Attached is a test IPC File containing a batch with length 1, array length 3.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
