[
https://issues.apache.org/jira/browse/ARROW-264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429441#comment-15429441
]
Julien Le Dem commented on ARROW-264:
-------------------------------------
[~wesmckinn] this sounds good. I'm considering reworking the RecordBatch
message. I find the flattened FieldNode and Buffer lists very error prone since
there are various factors that influence the number of buffers returned for a
FieldNode. I'd rather have a FieldNode tree mimicking the Schema and have a
list of Buffers in each. That would be more explicit and easier to manipulate.
It would also make bugs related to reading the buffers a lot more obvious.
Right now it is easy to misattribute the buffers and not know about it.
> Create an Arrow File format
> ---------------------------
>
> Key: ARROW-264
> URL: https://issues.apache.org/jira/browse/ARROW-264
> Project: Apache Arrow
> Issue Type: Improvement
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
>
> File layout:
> (DictionaryBatch, RecordBatch, Schema as defined in Message.fbs)
> {noformat}
> MAGIC: ARROW1
> (
> DictionaryBatch: DictionaryBatch Header (FlatBuffer)
> DictionaryBatch: DictionaryBatch Body (buffers concatenated)
> )*
> (
> RecordBacth: RecordBatch Header (FlatBuffer)
> RecordBacth: RecordBatch Body (buffers concatenated)
> )+
> Footer: Flatbuffer
> Footer length: int (4 bytes unsigned LE)
> MAGIC: ARROW1
> {noformat}
> Footer definition:
> {noformat}
> table Footer {
> schema: org.apache.arrow.flatbuf.Schema;
> dictionaries: [ Block ];
> recordBatches: [ Block ];
> }
> struct Block {
> offset: long;
> metaDataLength: int;
> bodyLength: long;
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
