martin-g commented on code in PR #530:
URL: https://github.com/apache/avro-rs/pull/530#discussion_r3620709110
##########
avro/src/reader/block.rs:
##########
@@ -295,6 +431,26 @@ impl<'r, R: Read> Block<'r, R> {
}
}
+impl<R: Read + Seek> Block<'_, R> {
+ /// Seek the underlying stream to `offset` and read the block there.
+ /// Validates the sync marker to confirm it's a real block boundary.
+ /// Returns an error if no valid block can be read at the offset
+ /// (e.g., the offset is at or past EOF).
Review Comment:
https://github.com/apache/avro-rs/pull/530/changes#diff-6d2c5562c5bd4d218417ad6c2eedd5e2a4f1cad8980fba3198a896aed8828adeR303-R306
returns `Ok` on EOF, not an `Err`.
##########
avro/src/reader/block.rs:
##########
@@ -162,7 +287,18 @@ impl<'r, R: Read> Block<'r, R> {
// and replace `buf` with the new one, instead of reusing the
same buffer.
// We can address this by using some "limited read" type to
decode directly
// into the buffer. But this is fine, for now.
- self.codec.decompress(&mut self.buf)
+ let next = self.codec.decompress(&mut self.buf);
+
+ // Make sure the position points only to a valid block
+ self.current_block_info = match next {
Review Comment:
Should this be done earlier ?!
If there is a problem with the marker then any call to
`Reader::current_block()` will return wrong data.
##########
avro/src/reader/block.rs:
##########
@@ -51,14 +159,27 @@ pub(super) struct Block<'r, R> {
pub(super) user_metadata: HashMap<String, Vec<u8>>,
names_refs: Names,
human_readable: bool,
+ /// Byte offset where data blocks begin (right after header and sync
marker).
+ pub(super) data_start: Option<u64>,
+ /// Position and record count of the currently loaded block.
+ pub(super) current_block_info: Option<BlockPosition>,
}
impl<'r, R: Read> Block<'r, R> {
pub(super) fn new(
reader: R,
schemata: Vec<&'r Schema>,
human_readable: bool,
+ track_positions: bool,
) -> AvroResult<Block<'r, R>> {
+ let reader = if track_positions {
+ PositionTracker::Tracking {
+ inner: reader,
+ pos: 0,
Review Comment:
If the underlying reader R is already partially read then `pos: 0` won't be
correct.
I am not sure how feasible this is though.
We can add one more optional field to ReaderBuilder - `initial_position`,
with a default value of `0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
