tustvold commented on code in PR #5179:
URL: https://github.com/apache/arrow-rs/pull/5179#discussion_r1418100875
##########
arrow-ipc/src/reader.rs:
##########
@@ -498,10 +498,34 @@ pub fn read_dictionary(
Ok(())
}
+/// Read the data for a given block
+fn read_block<R: Read + Seek>(mut reader: R, block: &Block) -> Result<Buffer,
ArrowError> {
+ reader.seek(SeekFrom::Start(block.offset() as u64))?;
+ let body_len = block.bodyLength().to_usize().unwrap();
+ let metadata_len = block.metaDataLength().to_usize().unwrap();
+ let total_len = body_len.checked_add(metadata_len).unwrap();
+
+ let mut buf = MutableBuffer::from_len_zeroed(total_len);
+ reader.read_exact(&mut buf)?;
+ Ok(buf.into())
+}
+
+/// Parse an encapsulated message
+///
+///
<https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format>
+fn parse_message(buf: &[u8]) -> Result<Message, ArrowError> {
+ let buf = match &buf[..4] == &CONTINUATION_MARKER {
+ true => &buf[8..],
+ false => &buf[4..],
+ };
Review Comment:
This does highlight a somewhat peculiar quirk of the IPC file format, it
doesn't actually care about the size prefixes. I guess this is just a
historical artifact of the fact the IPC streams came first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
