Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-07 Thread via GitHub


alamb merged PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-07 Thread via GitHub


alamb commented on PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3165620921

   🚀 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2255542788


##
arrow-avro/src/reader/mod.rs:
##
@@ -158,39 +162,131 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.expect_prefix
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(&mut self, buf: &[u8]) -> Result, 
ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if !self.expect_prefix {
+return Ok(None);
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match self.fingerprint_algorithm {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes| 
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+// NOTE: Incomplete fingerprint consumes no bytes.
+let consumed = fingerprint_size.map_or(0, |n| n + 
SINGLE_OBJECT_MAGIC.len());
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Insufficient bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch 
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+let new_decoder = self.cache.shift_remove(&new_fingerprint);
+let new_decoder = match new_decoder {
+Some(decoder) => decoder,
+None => {
+return Err(ArrowError::ParseError(format!(
+"Unknown fingerprint: {new_fingerprint:?}"
+)))
+}
+};
+

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


jecsand838 commented on PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3156930240

   > Good enough for now, I think.
   > 
   > There are lots of follow-ups to simplify the logic, especially places 
where we funnel two distinct cases through one code path that has to tease them 
back apart. But I think there's ongoing work to address those?
   
   100% I have that code mostly ready. Should only take me a few hours to get 
that follow-up PR up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2255515847


##
arrow-avro/src/reader/mod.rs:
##
@@ -220,34 +320,119 @@ impl ReaderBuilder {
 /// - `batch_size` = 1024
 /// - `strict_mode` = false
 /// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
 pub fn new() -> Self {
 Self::default()
 }
 
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) -> 
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(), 
self.utf8_view)
+fn make_record_decoder(
+&self,
+writer_schema: &Schema,
+reader_schema: Option<&AvroSchema>,
+) -> Result {
+let root = match reader_schema {
+Some(reader_schema) => {
+
AvroFieldBuilder::new(writer_schema).with_reader_schema(reader_schema.clone())
+}
+_ => AvroFieldBuilder::new(writer_schema),
+}

Review Comment:
   nit:
   ```suggestion
   let mut builder = AvroFieldBuilder::new(writer_schema);
   if let Some(reader_schema) = reader_schema {
   builder = builder.with_reader_schema(reader_schema.clone());
   }
   let root = builder
   ```



##
arrow-avro/src/reader/mod.rs:
##
@@ -158,39 +162,131 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.expect_prefix
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(&mut self, buf: &[u8]) -> Result, 
ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if !self.expect_prefix {
+return Ok(None);
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match self.fingerprint_algorithm {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes| 
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+  

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


jecsand838 commented on PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3156567917

   @scovich @alamb Now this one is much smaller. Was there any remaining 
changes we still needed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254942398


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > (I don't know how often we'd even be looking up schemas, in the common 
case?)
   
   For the streaming raw `Decoder` it will be very common imo.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254784213


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   Also I pushed these changes to the other PR as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254783570


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   Of course.
   
   First rule of optimization: Don't!
   
   (I don't know how often we'd even be looking up schemas, in the common case?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254741278


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   @scovich would you be alright if we revisit that in a follow-up?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254342687


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   Nice! The only downside is, it has to deserialize (albeit in place) every 
time we access the schema; it would be nice if there were a way to capture and 
cache the result (alongside the backing string), but that's where lifetimes get 
really messy unless you're willing to resort to tricks like interior mutability.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254342687


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   Nice! The only downside is, it has to deserialize (albeit in place) every 
time we access the schema; it would be nice if there were a way to capture the 
result, but that's where lifetimes get really messy unless you're willing to 
resort to tricks like interior mutability.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-05 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2253530422


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   @scovich I just pushed up an attempt at handling the schemas internally. I 
managed to get rid of all of the `static` usage. Let me know what you think.
   
   It's built around this new `AvroSchema` type in `schema.rs`:
   ```rust
   /// A wrapper for an Avro schema in its JSON string representation.
   #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
   pub struct AvroSchema {
   /// The Avro schema as a JSON string.
   pub json_string: String,
   }
   
   impl AvroSchema {
   /// Creates a new `AvroSchema` from a JSON string.
   pub fn new(json_string: String) -> Self {
   Self { json_string }
   }
   
   /// Deserializes and returns the `AvroSchema`.
   ///
   /// The returned schema borrows from `self`.
   pub fn schema(&self) -> Result, ArrowError> {
   serde_json::from_str(self.json_string.as_str())
   .map_err(|e| ArrowError::ParseError(format!("Invalid Avro schema 
JSON: {e}")))
   }
   
   /// Returns the Rabin fingerprint of the schema.
   pub fn fingerprint(&self) -> Result {
   generate_fingerprint_rabin(&self.schema()?)
   }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252866464


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > `HeaderDecoder` seems able to get around this by parsing a json string 
from the first row of an Avro file during the `ReaderBuilder::build` process
   
   The `serde_json` code is doing [zero-copy 
deserialization](https://serde.rs/lifetimes.html) from the header's byte 
buffer. There's an implicit lifetime as if declared by:
   ```rust
   pub fn schema<'a>(&'a self) -> Result>, ArrowError> {
   ```
   
   So the resulting schema inherits the lifetime of the header itself. AFAIK, 
the only way we could get a similar effect here is to add a lifetime param to 
the `Decoder`. I guess we could also try to create schemas internally, but 
self-referential lifetimes tend to get really messy really fast in rust.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252856054


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > trying to think around the problem to keep this public api similar to the 
other arrow crates.
   
   AFAIK, all the other crates define self-standing schemas that don't need 
lifetimes in the first place...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252820147


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   @scovich You're right. Here's where my research led me: Requiring `'static` 
lifetimes for Avro schemas means either restricting schemas to static data or 
leaking heap memory for dynamic schema components.
   
   One thing I noticed is that the `HeaderDecoder` seems able to get around 
this by parsing a json string from the first row of an Avro file during the 
`ReaderBuilder::build` process, i.e.
   
   ```rust
   pub fn schema(&self) -> Result>, ArrowError> {
   self.get(SCHEMA_METADATA_KEY)
   .map(|x| {
   serde_json::from_slice(x).map_err(|e| {
   ArrowError::ParseError(format!("Failed to parse Avro 
schema JSON: {e}"))
   })
   })
   .transpose()
   }
   ```
   
   Do you think an approach that uses JSON Strings instead of `Schema`, for 
instance:
   
   ```rust
   /// Sets the Avro reader schema.
   ///
   /// If a schema is not provided, the schema will be read from the Avro 
file header.
   pub fn with_reader_schema(mut self, reader_schema: String) -> Self {
   ...
   }
   
   /// Sets the `SchemaStore` used for resolving writer schemas.
   ///
   /// This is necessary when decoding single-object encoded data that 
identifies
   /// schemas by a fingerprint. The store allows the decoder to look up the
   /// full writer schema from a fingerprint embedded in the data.
   ///
   /// Defaults to `None`.
   pub fn with_writer_schema_store(mut self, store: HashMap) -> Self {
   ...
   }
   ```
   
   Then parses the each `Schema` inside the `build` and `build_decoder` methods 
would be a way around this?
   
   UPDATE:
   
   Maybe we could also pre-build the `cache: IndexMap,` from JSON strings in the `ReaderBuilder::build_decoder` method 
and then pass that into the new `Decoder` so we can remove:
   
   ```rust
   reader_schema: Option>,
   writer_schema_store: Option>,
   ```
   
   We'll pay for this with start-up cost and memory footprint. I suspect for 
smaller and even medium sized caches it will be worth it, not sure how large 
the cache needs to be for it not to be worth it though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252820147


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   @scovich You're right. Here's where my research led me: Requiring `'static` 
lifetimes for Avro schemas means either restricting schemas to static data or 
leaking heap memory for dynamic schema components.
   
   One thing I noticed is that the `HeaderDecoder` seems able to get around 
this by parsing a json string from the first row of an Avro file during the 
`ReaderBuilder::build` process, i.e.
   
   ```rust
   pub fn schema(&self) -> Result>, ArrowError> {
   self.get(SCHEMA_METADATA_KEY)
   .map(|x| {
   serde_json::from_slice(x).map_err(|e| {
   ArrowError::ParseError(format!("Failed to parse Avro 
schema JSON: {e}"))
   })
   })
   .transpose()
   }
   ```
   
   Do you think an approach that uses JSON Strings instead of `Schema`, for 
instance:
   
   ```rust
   /// Sets the Avro reader schema.
   ///
   /// If a schema is not provided, the schema will be read from the Avro 
file header.
   pub fn with_reader_schema(mut self, reader_schema: String) -> Self {
   ...
   }
   
   /// Sets the `SchemaStore` used for resolving writer schemas.
   ///
   /// This is necessary when decoding single-object encoded data that 
identifies
   /// schemas by a fingerprint. The store allows the decoder to look up the
   /// full writer schema from a fingerprint embedded in the data.
   ///
   /// Defaults to `None`.
   pub fn with_writer_schema_store(mut self, store: HashMap) -> Self {
   ...
   }
   ```
   
   Then parses the each `Schema` inside the `build` and `build_decoder` methods 
would be a way around this?
   
   UPDATE:
   
   Maybe we could also pre-build the `cache: IndexMap,` from JSON strings in the `ReaderBuilder::build_decoder` method 
and then pass that into the new `Decoder` so we can remove:
   
   ```rust
   reader_schema: Option>,
   writer_schema_store: Option>,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252836569


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > (maybe the answer is just that Decoder and its builder need a lifetime 
param so the compiler can figure out lifetimes based on the calling context?)
   
   Also I don't have any issues with this answer. Just trying to think around 
the problem to keep this public api similar to the other arrow crates.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252820147


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   @scovich You're right. Here's where my research led me: Requiring `'static` 
lifetimes for Avro schemas means either restricting schemas to static data or 
leaking heap memory for dynamic schema components.
   
   One thing I noticed is that the `HeaderDecoder` seems able to get around 
this by parsing a json string from the first row of an Avro file during the 
`ReaderBuilder::build` process, i.e.
   
   ```rust
   pub fn schema(&self) -> Result>, ArrowError> {
   self.get(SCHEMA_METADATA_KEY)
   .map(|x| {
   serde_json::from_slice(x).map_err(|e| {
   ArrowError::ParseError(format!("Failed to parse Avro 
schema JSON: {e}"))
   })
   })
   .transpose()
   }
   ```
   
   Do you think an approach that uses JSON Strings instead of `Schema`, for 
instance:
   
   ```rust
   /// Sets the Avro reader schema.
   ///
   /// If a schema is not provided, the schema will be read from the Avro 
file header.
   pub fn with_reader_schema(mut self, reader_schema: String) -> Self {
   ...
   }
   
   /// Sets the `SchemaStore` used for resolving writer schemas.
   ///
   /// This is necessary when decoding single-object encoded data that 
identifies
   /// schemas by a fingerprint. The store allows the decoder to look up the
   /// full writer schema from a fingerprint embedded in the data.
   ///
   /// Defaults to `None`.
   pub fn with_writer_schema_store(mut self, store: HashMap) -> Self {
   ...
   }
   ```
   
   Then parses the each `Schema` inside the `build` and `build_decoder` methods 
would be a way around this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252820147


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   @scovich You're right. Here's where my research led me: Requiring `'static` 
lifetimes for Avro schemas means either restricting schemas to static data or 
leaking heap memory for dynamic schema components.
   
   One thing I noticed is that the `HeaderDecoder` seems able to get around 
this by parsing a json string from the first row of an Avro file during the 
`ReaderBuilder::build` process, i.e.
   
   ```rust
   pub fn schema(&self) -> Result>, ArrowError> {
   self.get(SCHEMA_METADATA_KEY)
   .map(|x| {
   serde_json::from_slice(x).map_err(|e| {
   ArrowError::ParseError(format!("Failed to parse Avro 
schema JSON: {e}"))
   })
   })
   .transpose()
   }
   ```
   
   Do you think an approach that uses JSON Strings instead of `Schema`, for 
instance:
   
   ```rust
   /// Sets the Avro reader schema.
   ///
   /// If a schema is not provided, the schema will be read from the Avro 
file header.
   pub fn with_reader_schema(mut self, reader_schema: String) -> Self {
   ...
   }
   
   /// Sets the `SchemaStore` used for resolving writer schemas.
   ///
   /// This is necessary when decoding single-object encoded data that 
identifies
   /// schemas by a fingerprint. The store allows the decoder to look up the
   /// full writer schema from a fingerprint embedded in the data.
   ///
   /// Defaults to `None`.
   pub fn with_writer_schema_store(mut self, store: HashMap) -> Self {
   ...
   }
   ```
   
   Then parse the `Schema` inside the `build` and `build_decoder` methods would 
be a way around this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252770776


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   (maybe the answer is just that `Decoder` and its builder need a lifetime 
param so the compiler can figure out lifetimes based on the calling context?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252769379


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   The unit tests don't create any schemas with `TypeName::Ref` -- only 
`TypeName::Primitive` -- and the `Attributes` all use string constants. But 
what if the read schema were user input (i.e. parsed from JSON) and then fed to 
a decoder?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252763116


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   I will admit I don't fully understand `'static` in rust. It doesn't 
_necessarily_ imply a memory leak, but it does seem difficult (albeit 
apparently not impossible) to apply `'static` lifetime to anything that will be 
freed before the program terminates.
   
   One way would be to never create a schema that actually uses `TypeName::Ref` 
nor any `Attributes`, for example. But that's kind of cheating -- may as well 
not support them at that point. 
   
   So I guess the question would be: Can one create a `Decoder` with Some 
`reader_schema`, where the memory allocated to create that reader schema 
actually gets freed when the decoder gets dropped? 
   
   A refinement of that question would be: Can one create a `Decoder` with Some 
`reader_schema` that does use `Attributes` or `TypeName::Ref`, with dynamically 
allocated strings that can be freed after the decoder is dropped? 
   
   Or, as I suspect, would one be forced to leak them (or use compile-time 
constant strings instead), in order to get the code to compile?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3152727580

   > > To move forward, would it be best to simply focus on getting the 
`schema.rs` changes in first (~500-600 LOC) and then follow-up with a 
`arrow-avro/src/reader/mod.rs` PR?
   > 
   > Interesting. By "schema.rs changes" you mean adding the new `SchemaStore` 
and fingerprinting infrastructure? With unit tests but not yet integrated into 
the actual decoder? That does seem like a good idea. I don't think there's any 
outstanding controversy in that part of the code (just a couple of follow-on 
items)?
   
   Thats correct and that's my understanding as well.
   
   > > I do apologize for both the size of this PR and the quality issues.
   > 
   > This is _NOT_ the kind of run of the mill logic that I would normally 
associate with "quality issues." For me, at least, this has been an 
invigorating PR to review -- not a frustrating one. Some hard core software 
engineering here with depth and subtlety to work through.
   
   I really appreciate that! I just thought I should have caught some of that 
subtlety upfront.
   
   > > Let me know what I should do and I'll get on it immediately.
   > 
   > Now that we know more, the split you propose seems quite reasonable; I'm 
not sure it was obvious two weeks ago tho?
   
   It crossed my mind honestly. I just thought that:
   1. It would be helpful to see the context of how the `SchemaStore` would be 
used.
   2. I saw the functionality as coupled (even if loosely) and wanted to ensure 
there was test coverage in `mod.rs`
   
   I'll get that PR up right now and link it here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252735697


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > The main concern for this PR is:
   > 
   
   > > Does that [static lifetimes here] mean memory they [the passed-in 
schemas] reference must leak for all practical purposes?
   
   I haven't had to resort to any memory leaks in `arrow-avro`. The `AvroField` 
logic is also bound by the same lifetime and the `Schema` is only used to 
create a root `AvroField` with a `Codec` which in turn is then used to create a 
`RecordDecoder`. I haven't had to resort to any `box::leak` OR reference cycles 
and I was careful to add bounding to the cache. 
   
   Also inside of the `Decoder` when making a new `RecordDecoder` (i.e. 
`create_decoder_for`) I don't resort to using `clone` on either of the schemas 
and no new `Schema` are created either. Each `RecordDecoder` can only use the 
same `reader_schema` and set of `writer_schema`.
   
   If I'm missing something however I apologize in advance. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252735697


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > The main concern for this PR is:
   > 
   
   > > Does that [static lifetimes here] mean memory they [the passed-in 
schemas] reference must leak for all practical purposes?
   
   I haven't had to resort to any memory leaks in `arrow-avro`. The `AvroField` 
logic is also bound by the same lifetime and the `Schema` is only used to 
create a root `AvroField` with a `Codec` which in turn is then used to create a 
`RecordDecoder`. I haven't had to resort to any `box::leak` OR reference cycles 
and I was careful to add bounding to the cache. 
   
   Also inside of the `Decoder` when making a new `RecordDecoder` I don't 
resort to using `clone` on either of the schemas and no new `Schema` are 
created either. Each `RecordDecoder` can only use the same `reader_schema` and 
set of `writer_schema`.
   
   If I'm missing something however I apologize in advance. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252735697


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > The main concern for this PR is:
   > 
   
   > > Does that [static lifetimes here] mean memory they [the passed-in 
schemas] reference must leak for all practical purposes?
   
   I haven't had to resort to any memory leaks here. The `AvroField` logic is 
also bound by the same lifetime and the `Schema` is only used to create a root 
`AvroField` with a `Codec` which in turn is then used to create a 
`RecordDecoder`. I haven't had to resort to any `box::leak` OR reference cycles 
and I was careful to add bounding to the cache. 
   
   Also inside of the `Decoder` when making a new `RecordDecoder` I don't 
resort to using `clone` on either of the schemas and no new `Schema` are 
created either. Each `RecordDecoder` can only use the same `reader_schema` and 
set of `writer_schema`.
   
   If I'm missing something however I apologize in advance. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


scovich commented on PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3152638873

   > this PR has a lot of comments and quite a few refinements were needed. I 
also know it was way too big.
   
   Honestly, I think the "state machine" for avro decoding is just really 
complex. It felt like a lot of the questions and churn ultimately come from 
that underlying complexity. And my lack of familiarity with the avro spec. Not 
sure how easily those issues could be avoided merely by raising a smaller PR?
   
   > To move forward, would it be best to simply focus on getting the 
`schema.rs` changes in first (~500-600 LOC) and then follow-up with a 
`arrow-avro/src/reader/mod.rs` PR?
   
   Interesting. By "schema.rs changes" you mean adding the new `SchemaStore` 
and fingerprinting infrastructure? With unit tests but not yet integrated into 
the actual decoder? That does seem like a good idea. I don't think there's any 
outstanding controversy in that part of the code (just a couple of follow-on 
items)?
   
   > I do apologize for both the size of this PR and the quality issues. 
   
   This is _NOT_ the kind of run of the mill logic that I would normally 
associate with "quality issues." 
   For me, at least, this has been an invigorating PR to review -- not a 
frustrating one. 
   Some hard core software engineering here with depth and subtlety to work 
through.
   
   > Let me know what I should do and I'll get on it immediately.
   
   Now that we know more, the split you propose seems quite reasonable; I'm not 
sure it was obvious two weeks ago tho?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252706588


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   The main concern for this PR is:
   
   > Does that [static lifetimes here] mean memory they [the passed-in schemas] 
reference must leak for all practical purposes?
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3152179396

   @scovich @alamb 
   
   I know this PR has a lot of comments and quite a few refinements were 
needed. I also know it was way too big. 
   
   To move forward, would it be best to simply focus on getting the `schema.rs` 
changes in first (~500-600 LOC) and then follow-up with a 
`arrow-avro/src/reader/mod.rs` PR?
   
   I do apologize for both the size of this PR and the quality issues. I'll do 
better going forward. Thank you again @scovich for the amount of time and 
effort you've put into reviewing this.
   
   Let me know what I should do and I'll get on it immediately. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252426527


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   @scovich 
   
   >  But is it really so expensive to just copy the string that we're willing 
to deal with a proliferation of lifetimes?
   
   In real world practice it's hard to say. That's one of the reasons I tried 
to improve benchmarking in my other PR. So we can start getting numbers to 
inform our changes. 
   
   > Or am I missing some obvious/easy place where the lifetimes make a big 
difference that justifies the complexity?
   
   I think the real source of the complexity here was my attempt to use the 
same `Decoder` and `decode` method for decoding both single object encodings 
and files. I have a plan to improve this in my very next PR before I do 
anything else.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251801967


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > (I don't actually understand why avro schemas have lifetimes in the first 
place? Especially if they anyway require static lifetime in practice?)
   
   I'm not 100% certain why that decision was made. I was simply trying to 
extend from the original design pattern. 
   
   My opinion is we're taking advantage of an Avro nuance (knowing the schemas 
in advance of record decoding) to minimize overhead via a borrowed schema 
approach that leverages zero-copy parsing. This makes sense since every 
scenario requires us to know the writer and reader schema (if using schema 
evolution) in advance of instantiating a `RecordDecoder`. 
   



##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   Because the use of lifetimes predates both this PR and my involvement, would 
you be ok with this being addressed in a follow-up PR prior to going public? 
@scovich 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251943653


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   Because the use of lifetimes predates both this PR and my involvement, would 
you be ok with this being addressed in a follow-up PR prior to going public? 
@scovich 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251938878


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..], 
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes| 
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch 
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+   

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251862329


##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +369,91 @@ impl ReaderBuilder {
 /// - `batch_size` = 1024
 /// - `strict_mode` = false
 /// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
 pub fn new() -> Self {
 Self::default()
 }
 
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) -> 
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(), 
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let root = match reader_schema {
+Some(reader_schema) if !compare_schemas(writer_schema, 
reader_schema)? => {
+
AvroFieldBuilder::new(writer_schema).with_reader_schema(reader_schema)
+}
+_ => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode)
+.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
+}
+
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+#[cfg(feature = "lru")]
+let capacity = 
NonZeroUsize::new(self.decoder_cache_size).unwrap_or(NonZeroUsize::MIN); // 
NonZeroUsize::MIN is 1
+Decoder {
+batch_size: self.batch_size,
+remaining_capacity: self.batch_size,
+active_fingerprint,
+active_decoder,
+#[cfg(feature = "lru")]
+cache: LruCache::new(capacity),
+#[cfg(not(feature = "lru"))]
+cache: IndexMap::new(),
+max_cache_size: self.decoder_cache_size,
+reader_schema,
+utf8_view: self.utf8_view,
+writer_schema_store,
+strict_mode: self.strict_mode,
+pending_schema: None,
+}
 }
 
-fn build_impl(self, reader: &mut R) -> Result<(Header, 
Decoder), ArrowError> {
-let header = read_header(reader)?;
-let record_decoder = if let Some(schema) = &self.schema {
-self.make_record_decoder(schema)?
-} else {
-let avro_schema: Option> = header
+fn make_decoder(&self, header: Option<&Header>) -> Result {
+if let Some(hdr) = header {
+let writer_schema = hdr
 .schema()
-.map_err(|e| ArrowError::ExternalError(Box::new(e)))?;
-let avro_schema = avro_schema.ok_or_else(|| {
-ArrowError::ParseError("No Avro schema present in file 
header".to_string())
+.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
+.ok_or_else(|| {
+ArrowError::ParseError("No Avro schema present in file 
header".into())
+})?;
+let record_decoder =
+self.make_record_decoder(&writer_schema, 
self.reader_schema.as_ref())?;
+return Ok(self.make_decoder_with_parts(record_decoder, None, None, 
None));
+}
+let writer_schema_store = 
self.writer_schema_store.as_ref().ok_or_else(|| {
+ArrowError::ParseError("Writer schema store required for raw 
Avro".into())
+})?;
+let fingerprint = self
+.active_fingerprint
+.or_else(|| writer_schema_store.fingerprints().into_iter().next())
+.ok_or_else(|| {
+ArrowError::ParseError(
+"Writer schema store must contain at least one 
schema".into(),
+)
 })?;
-self.make_record_decoder(&avro_schema)?
-};
-let decoder = Decoder::new(record_decoder, self.batch_size);
-Ok((header, decoder))
+let writer_schema = 
writer_schema_store.lookup(&fingerprint).ok_or_else(|| {
+ArrowError::ParseError("Active fingerprint not found in schema 
store".into())
+})?;
+let record_decoder =
+self.make_record_decoder(writer_schema, 
self.reader_schema.as_ref())?;
+Ok(self.make_decoder_with_parts(
+record_decoder,
+Some(fingerprint),
+self.reader_schema.clone(),
+self.writer_schema_store.clone(),

Review Comment:
   > But what about the reader schema? What's the story behind the smell there?
   
   What's happening here is the file read path won't ever switch 
`RecordDecoder` during decoding 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251839467


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..], 
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.

Review Comment:
   @scovich 
   
   I'm actually refactoring this code in my next PR to handle chunking partial 
records. Is it okay if we pick this up there? It will be ~300 LOC and 
exclusively focused on decoding. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251801967


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > (I don't actually understand why avro schemas have lifetimes in the first 
place? Especially if they anyway require static lifetime in practice?)
   
   I'm not 100% certain why that decision was made. I was simply trying to 
extend from the original design pattern. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251797593


##
arrow-avro/Cargo.toml:
##
@@ -57,7 +57,8 @@ xz = { version = "0.1", default-features = false, optional = 
true }
 crc = { version = "3.0", optional = true }
 strum_macros = "0.27"
 uuid = "1.17"
-indexmap = "2.10"
+lru  = { version = "0.12", optional = true }
+indexmap = "2.10.0"

Review Comment:
   I'll change this to be `2.10`, mistake on my part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251801967


##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) -> 
Result {
 /// A low-level interface for decoding Avro-encoded bytes into Arrow 
`RecordBatch`.
 #[derive(Debug)]
 pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
 batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,

Review Comment:
   > (I don't actually understand why avro schemas have lifetimes in the first 
place? Especially if they anyway require static lifetime in practice?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251790346


##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +369,91 @@ impl ReaderBuilder {
 /// - `batch_size` = 1024
 /// - `strict_mode` = false
 /// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
 pub fn new() -> Self {
 Self::default()
 }
 
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) -> 
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(), 
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let root = match reader_schema {
+Some(reader_schema) if !compare_schemas(writer_schema, 
reader_schema)? => {
+
AvroFieldBuilder::new(writer_schema).with_reader_schema(reader_schema)
+}
+_ => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode)
+.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
+}
+
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+#[cfg(feature = "lru")]
+let capacity = 
NonZeroUsize::new(self.decoder_cache_size).unwrap_or(NonZeroUsize::MIN); // 
NonZeroUsize::MIN is 1

Review Comment:
   The CI fmt job is aware of this crate and I always run fmt on the code 
before pushing up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251781361


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..], 
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes| 
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch 
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+   

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251721555


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..], 
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes| 
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch 
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+  

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251720636


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..], 
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes| 
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch 
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+  

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-04 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251711943


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..], 
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes| 
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch 
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+   

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-02 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249257375


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..], 
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes| 
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch 
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+  

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-02 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249240259


##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +369,91 @@ impl ReaderBuilder {
 /// - `batch_size` = 1024
 /// - `strict_mode` = false
 /// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
 pub fn new() -> Self {
 Self::default()
 }
 
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) -> 
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(), 
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let root = match reader_schema {
+Some(reader_schema) if !compare_schemas(writer_schema, 
reader_schema)? => {
+
AvroFieldBuilder::new(writer_schema).with_reader_schema(reader_schema)
+}
+_ => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode)
+.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
+}
+
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+#[cfg(feature = "lru")]
+let capacity = 
NonZeroUsize::new(self.decoder_cache_size).unwrap_or(NonZeroUsize::MIN); // 
NonZeroUsize::MIN is 1
+Decoder {
+batch_size: self.batch_size,
+remaining_capacity: self.batch_size,
+active_fingerprint,
+active_decoder,
+#[cfg(feature = "lru")]
+cache: LruCache::new(capacity),
+#[cfg(not(feature = "lru"))]
+cache: IndexMap::new(),
+max_cache_size: self.decoder_cache_size,
+reader_schema,
+utf8_view: self.utf8_view,
+writer_schema_store,
+strict_mode: self.strict_mode,
+pending_schema: None,
+}
 }
 
-fn build_impl(self, reader: &mut R) -> Result<(Header, 
Decoder), ArrowError> {
-let header = read_header(reader)?;
-let record_decoder = if let Some(schema) = &self.schema {
-self.make_record_decoder(schema)?
-} else {
-let avro_schema: Option> = header
+fn make_decoder(&self, header: Option<&Header>) -> Result {
+if let Some(hdr) = header {
+let writer_schema = hdr
 .schema()
-.map_err(|e| ArrowError::ExternalError(Box::new(e)))?;
-let avro_schema = avro_schema.ok_or_else(|| {
-ArrowError::ParseError("No Avro schema present in file 
header".to_string())
+.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
+.ok_or_else(|| {
+ArrowError::ParseError("No Avro schema present in file 
header".into())
+})?;
+let record_decoder =
+self.make_record_decoder(&writer_schema, 
self.reader_schema.as_ref())?;
+return Ok(self.make_decoder_with_parts(record_decoder, None, None, 
None));
+}
+let writer_schema_store = 
self.writer_schema_store.as_ref().ok_or_else(|| {
+ArrowError::ParseError("Writer schema store required for raw 
Avro".into())
+})?;
+let fingerprint = self
+.active_fingerprint
+.or_else(|| writer_schema_store.fingerprints().into_iter().next())
+.ok_or_else(|| {
+ArrowError::ParseError(
+"Writer schema store must contain at least one 
schema".into(),
+)
 })?;
-self.make_record_decoder(&avro_schema)?
-};
-let decoder = Decoder::new(record_decoder, self.batch_size);
-Ok((header, decoder))
+let writer_schema = 
writer_schema_store.lookup(&fingerprint).ok_or_else(|| {
+ArrowError::ParseError("Active fingerprint not found in schema 
store".into())
+})?;
+let record_decoder =
+self.make_record_decoder(writer_schema, 
self.reader_schema.as_ref())?;
+Ok(self.make_decoder_with_parts(
+record_decoder,
+Some(fingerprint),
+self.reader_schema.clone(),
+self.writer_schema_store.clone(),

Review Comment:
   This is a weird code smell. Some of the state comes from `self` internally, 
while other state has to be copied manually by caller even tho it's _also_ 
coming from `self`?
   
   We 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-02 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249229633


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..], 
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema 
switch, or we need
+// more bytes to make a decision. Either way, this decoding 
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes| 
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch 
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+  

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3146237014

   @scovich I got those latest changes pushed up. Really improved the quality 
of the code. I definitely appreciate these deep reviews.
   
   I'm finishing up the development of single object encoding partial record 
chunking logic + some additional decoder cleanups. I plan to have that in a 
(much smaller) follow-up PR as soon as we get this one merged in. 
   
   Let me know what you think and thank you again!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249126660


##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
 self
 }
 
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
 ///
 /// If a schema is not provided, the schema will be read from the Avro 
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) -> 
Self {
+self.reader_schema = Some(reader_schema);
 self
 }
 
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that 
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) -> 
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded 
data.
+///
+/// This is useful when the data stream does not begin with a schema 
definition
+/// or fingerprint, allowing the decoder to start with a known schema from 
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may 
need to switch
+/// between different decoders. This cache avoids rebuilding them from 
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|
+// | Some| Some--| None---| Ok-|
+// | Some| Some--| Some---| Ok-|
+fn validate(&self) -> Result<(), ArrowError> {
+match (
+self.writer_schema_store.is_some(),
+self.reader_schema.is_some(),
+self.active_fingerprint.is_some(),
+) {
+// Row 3: No store, reader schema present, no fingerprint
+(false, true, false)
+// Row 7: Store is present, reader schema is resent, no fingerprint
+| (true, true, false)
+// Row 8: Store present, reader schema present, fingerprint present
+| (true, true, true) => Ok(()),
+// Fingerprint without a store (rows 2 & 4)
+(false, _, true) => Err(ArrowError::InvalidArgumentError(
+"Active fingerprint requires a writer schema store".into(),
+)),
+// Store present but no reader schema (rows 5 & 6)
+(true, false, _) => Err(ArrowError::InvalidArgumentError(
+"Reader schema must be set when writer schema store is 
provided".into(),
+)),
+// No schema store or reader schema provided (row 1)
+(false, false, _) => Err(ArrowError::InvalidArgumentError(

Review Comment:
   @scovich So you were correct here. I was able to complete remove the 
`validate` method and ended up simplifying a lot of the logic. For instance I 
managed to slim down `make_decoder` to this:
   
   ```rust
   fn make_decoder(&self, header: Option<&Header>) -> Result {
   if let Some(hdr) = header {
   let writer_schema = hdr
   .schema()
   .map_err(|e| ArrowError::ExternalError(Box::new(e)))?
   .ok_or_else(|| {
   ArrowError::ParseError("No Avro schema present in file 
header".into())
   })?;
   let record_decoder =
   self.make_record_decoder(&writer_schema, 
self.reader_schema.as_ref())?;
   return Ok(self.make_decoder_with_parts(record_decoder, None, 
None, None));
   }
   let writer_schema_store 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249126660


##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
 self
 }
 
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
 ///
 /// If a schema is not provided, the schema will be read from the Avro 
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) -> 
Self {
+self.reader_schema = Some(reader_schema);
 self
 }
 
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that 
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) -> 
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded 
data.
+///
+/// This is useful when the data stream does not begin with a schema 
definition
+/// or fingerprint, allowing the decoder to start with a known schema from 
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may 
need to switch
+/// between different decoders. This cache avoids rebuilding them from 
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|
+// | Some| Some--| None---| Ok-|
+// | Some| Some--| Some---| Ok-|
+fn validate(&self) -> Result<(), ArrowError> {
+match (
+self.writer_schema_store.is_some(),
+self.reader_schema.is_some(),
+self.active_fingerprint.is_some(),
+) {
+// Row 3: No store, reader schema present, no fingerprint
+(false, true, false)
+// Row 7: Store is present, reader schema is resent, no fingerprint
+| (true, true, false)
+// Row 8: Store present, reader schema present, fingerprint present
+| (true, true, true) => Ok(()),
+// Fingerprint without a store (rows 2 & 4)
+(false, _, true) => Err(ArrowError::InvalidArgumentError(
+"Active fingerprint requires a writer schema store".into(),
+)),
+// Store present but no reader schema (rows 5 & 6)
+(true, false, _) => Err(ArrowError::InvalidArgumentError(
+"Reader schema must be set when writer schema store is 
provided".into(),
+)),
+// No schema store or reader schema provided (row 1)
+(false, false, _) => Err(ArrowError::InvalidArgumentError(

Review Comment:
   @scovich So you were correct here. I was able to complete remove the 
`validate` method and ended up simplifying a lot of the logic. For instance I 
managed to slim down `make_decoder` to this:
   
   ```rust
   fn make_decoder(&self, header: Option<&Header>) -> Result {
   if let Some(hdr) = header {
   let writer_schema = hdr
   .schema()
   .map_err(|e| ArrowError::ExternalError(Box::new(e)))?
   .ok_or_else(|| {
   ArrowError::ParseError("No Avro schema present in file 
header".into())
   })?;
   let record_decoder =
   self.make_record_decoder(&writer_schema, 
self.reader_schema.as_ref())?;
   return Ok(self.make_decoder_with_parts(record_decoder, None, 
None, None));
   }
   let writer_schema_store 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249126125


##
arrow-avro/src/schema.rs:
##
@@ -260,13 +276,389 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+/// 
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified 
`FingerprintAlgorithm`.
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form 
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+/// 

+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas 
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is 
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single 
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its 
operations.
+/// This ensures consistency when generating fingerprints and looking up 
schemas.
+/// All schemas registered will have their fingerprint computed with this 
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string 
slices
+/// contained within the stored [`Schema`] objects. This means the 
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each schema in the slice is registered with the 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249126029


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and 
stop.
+total_consumed += n;
+break;
+}
+}
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint::<8>(&buf[SINGLE_OBJECT_MAGIC.len()..], 
|bytes| {
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249125735


##
arrow-avro/src/schema.rs:
##
@@ -260,13 +276,389 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+/// 
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified 
`FingerprintAlgorithm`.
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form 
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+/// 

+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas 
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is 
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single 
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its 
operations.
+/// This ensures consistency when generating fingerprints and looking up 
schemas.
+/// All schemas registered will have their fingerprint computed with this 
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string 
slices
+/// contained within the stored [`Schema`] objects. This means the 
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each schema in the slice is registered with the 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249116924


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and 
stop.
+total_consumed += n;
+break;
+}
+}
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.

Review Comment:
   Unit tests covering these corner cases were included in my latest push.
   
   I also improved the `test_decode_stream_with_schema` test to check that the 
prefix parsing works correctly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249115313


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}

Review Comment:
   @scovich I got this figured out. Currently we can only decode full single 
object encoded records via `Decoder::decode`. In my very next PR I'll add a 
partial record chunking feature to the `Decoder` which will resolve this. 
   
   I have it about 90% complete on my local machine, I just didn't want to grow 
this PR any further.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249114536


##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +330,98 @@ impl ReaderBuilder {
 /// - `batch_size` = 1024
 /// - `strict_mode` = false
 /// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fp` = None
+/// - `static_store_mode` = false
 pub fn new() -> Self {
 Self::default()
 }
 
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) -> 
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(), 
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let field_builder = match reader_schema {
+Some(rs) if !compare_schemas(writer_schema, rs)? => {
+AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)
+}
+Some(rs) => AvroFieldBuilder::new(rs),
+None => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode);
+let root = field_builder.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
 }
 
-fn build_impl(self, reader: &mut R) -> Result<(Header, 
Decoder), ArrowError> {
-let header = read_header(reader)?;
-let record_decoder = if let Some(schema) = &self.schema {
-self.make_record_decoder(schema)?
-} else {
-let avro_schema: Option> = header
-.schema()
-.map_err(|e| ArrowError::ExternalError(Box::new(e)))?;
-let avro_schema = avro_schema.ok_or_else(|| {
-ArrowError::ParseError("No Avro schema present in file 
header".to_string())
-})?;
-self.make_record_decoder(&avro_schema)?
-};
-let decoder = Decoder::new(record_decoder, self.batch_size);
-Ok((header, decoder))
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+Decoder {
+batch_size: self.batch_size,
+remaining_capacity: self.batch_size,
+active_fingerprint,
+active_decoder,
+cache: IndexMap::new(),
+max_cache_size: self.decoder_cache_size,
+reader_schema,
+utf8_view: self.utf8_view,
+writer_schema_store,
+strict_mode: self.strict_mode,
+pending_schema: None,
+}
+}
+
+fn make_decoder(&self, header: Option<&Header>) -> Result {
+match header {
+Some(hdr) => {
+let writer_schema = hdr
+.schema()
+.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
+.ok_or_else(|| {
+ArrowError::ParseError("No Avro schema present in file 
header".into())
+})?;
+let record_decoder =
+self.make_record_decoder(&writer_schema, 
self.reader_schema.as_ref())?;
+Ok(self.make_decoder_with_parts(record_decoder, None, None, 
None))
+}
+None => {
+let reader_schema = self.reader_schema.clone().ok_or_else(|| {
+ArrowError::ParseError("Reader schema required for raw 
Avro".into())
+})?;
+let (init_fingerprint, initial_decoder) =
+if let (Some(schema_store), Some(fingerprint)) =
+(&self.writer_schema_store, self.active_fingerprint)
+{
+// An initial fingerprint is provided, use it to look 
up the first schema.
+let writer_schema = 
schema_store.lookup(&fingerprint).ok_or_else(|| {
+ArrowError::ParseError(
+"Active fingerprint not found in schema 
store".into(),
+)
+})?;
+let decoder =
+self.make_record_decoder(writer_schema, 
Some(&reader_schema))?;
+(Some(fingerprint), decoder)
+} else {
+// No initial fingerprint; the first record must 
contain one.
+// A decoder is created from the reader schema only.
+let decoder = self.make_record_decoder(&reader_schema, 
None)?;
+   

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248969563


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}

Review Comment:
   Yeah there's nothing about the wire format that prevents partial reads, the 
only rule I came across is the minimum amount that can be read was one "datum" 
at a time.
   
   > Sure, but the Some(n) case does exactly the same thing (just for a 
different reason). That's why I had suggested to combine them and just use a 
comment to explain what's going on.
   
   I'm going with this approach for this PR as it's logical. I may follow-up 
later with another PR to revisit some of the complexity of the `Decoder`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248969563


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}

Review Comment:
   Yeah there's nothing about the wire format that prevents partial reads, the 
only rule I came across is the minimum amount that can be read was one "datum" 
at a time, i.e. a row's field.
   
   > Sure, but the Some(n) case does exactly the same thing (just for a 
different reason). That's why I had suggested to combine them and just use a 
comment to explain what's going on.
   
   I'm going with this approach for this PR as it's logical. I may follow-up 
later with another PR to revisit some of the complexity of the `Decoder`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248969563


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}

Review Comment:
   Yeah there's nothing about the wire format that prevents partial reads, the 
only rule I came across is the minimum amount that can be read was one "datum" 
at a time, i.e. a row's field.
   
   > Sure, but the Some(n) case does exactly the same thing (just for a 
different reason). That's why I had suggested to combine them and just use a 
comment to explain what's going on.
   
   I'm going with this approach for this PR as it's logical. I may follow-up 
later with another PR to revisit some of the complexity on the `Reader` and 
`Decoder`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248843889


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}

Review Comment:
   > Not sure why it should throw an error... is there some special property of 
the wire format that prevents partial reads?
   
   I'm researching this pretty deeply right now. I'll report back here soon.
   
   I also think I found a huge simplification opportunity related to this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248843889


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}

Review Comment:
   > Not sure why it should throw an error... is there some special property of 
the wire format that prevents partial reads?
   
   I'm researching this pretty deeply right now. I'll report back here soon.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248778865


##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +365,97 @@ impl ReaderBuilder {
 /// - `batch_size` = 1024
 /// - `strict_mode` = false
 /// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
 pub fn new() -> Self {
 Self::default()
 }
 
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) -> 
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(), 
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let field_builder = match reader_schema {
+Some(rs) if !compare_schemas(writer_schema, rs)? => {
+AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)
+}
+Some(rs) => AvroFieldBuilder::new(rs),
+None => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode);
+let root = field_builder.build()?;

Review Comment:
   The above sounds good, but I was only commenting on where to put the 
`with_xxx` calls above? That seems like a very localized change, orthogonal to 
the conditional logic you mention?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248776365


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and 
stop.
+total_consumed += n;
+break;
+}
+}
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint::<8>(&buf[SINGLE_OBJECT_MAGIC.len()..], 
|bytes| {
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248768996


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}

Review Comment:
   > use `Some(0)` to cover a scenario where magic bytes were read, but there 
wasn't enough remaining bytes in the buffer for the fingerprint. My thought was 
to then return so the caller could add more bytes.
   
   Sure, but the `Some(n)` case does exactly the same thing (just for a 
different reason). That's why I had suggested to combine them and just use a 
comment to explain what's going on.
   
   > digging into the Java implementation's code however I noticed they throw 
an error for this scenario
   
   Not sure why it should throw an error... is there some special property of 
the wire format that prevents partial reads?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248718082


##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
 self
 }
 
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
 ///
 /// If a schema is not provided, the schema will be read from the Avro 
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) -> 
Self {
+self.reader_schema = Some(reader_schema);
 self
 }
 
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that 
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) -> 
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded 
data.
+///
+/// This is useful when the data stream does not begin with a schema 
definition
+/// or fingerprint, allowing the decoder to start with a known schema from 
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may 
need to switch
+/// between different decoders. This cache avoids rebuilding them from 
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|

Review Comment:
   That's another 3am mistake on my part. I'll fix that as well.
   
   The `reader_schema` isn't always required. It's technically the 
`writer_schema` that is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248718082


##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
 self
 }
 
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
 ///
 /// If a schema is not provided, the schema will be read from the Avro 
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) -> 
Self {
+self.reader_schema = Some(reader_schema);
 self
 }
 
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that 
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) -> 
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded 
data.
+///
+/// This is useful when the data stream does not begin with a schema 
definition
+/// or fingerprint, allowing the decoder to start with a known schema from 
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may 
need to switch
+/// between different decoders. This cache avoids rebuilding them from 
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|

Review Comment:
   That's another 3am mistake on my part. I'll fix that as well. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248715880


##
arrow-avro/src/schema.rs:
##
@@ -15,12 +15,27 @@
 // specific language governing permissions and limitations
 // under the License.
 
+use arrow_schema::ArrowError;
 use serde::{Deserialize, Serialize};
+use serde_json::{json, Value};
+use std::collections::hash_map::Entry;
 use std::collections::HashMap;
+use strum_macros::AsRefStr;
 
 /// The metadata key used for storing the JSON encoded [`Schema`]
 pub const SCHEMA_METADATA_KEY: &str = "avro.schema";
 
+/// The Avro single‑object encoding “magic” bytes (`0xC3 0x01`)
+pub const SINGLE_OBJECT_MAGIC: [u8; 2] = [0xC3, 0x01];

Review Comment:
   @scovich From the research I was doing it seems what prevents false 
positives is having the magic bytes **located at the start** of the message.  
   
   Here's a twist to consider though, the `Decoder::decoder` method was 
developed to be generic enough to handle both:
   1. OCF block encodings (blocks of Avro records without fingerprints, writer 
schema is determined from OCF header which has it's own 4 magic byte sequence)
   2. Message single object encodings (an encoding of a single Avro record 
starting with the 2 byte magic + 8 byte fingerprint OR 1 byte magic + 4 byte id 
fingerprint if Confluent variant).
   
   I'm beginning to wonder if we'd benefit from having two decoder methods. 
   
   For instance both OCF and the message format support schema evolution via 
writer and reader schemas. In message format the writer schema is determined 
via a fingerprint located on **each Avro message** that may or may not change 
between messages. Meanwhile in OCF format the writer schema is provided in the 
header and will remain static for the full file which in turn is a sequence of 
block encodings comprised of individual Avro records.
   
   My original plan was to use the `Reader` for OCF and the stand-alone 
`Decoder` for Messages (while being re-used in the `Reader`). I'm beginning to 
question that approach.
   
   A second alternative is to define a `DecodeMode` enum for the `Decoder` to 
used for identifying which decode flow is required.
   
   Either way I need to make some modifications based on this new research. 
I'll have something pushed up by this evening. Ty for bringing this up!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248632689


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and 
stop.
+total_consumed += n;
+break;
+}
+}
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint::<8>(&buf[SINGLE_OBJECT_MAGIC.len()..], 
|bytes| {
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint = 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248589128


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and 
stop.
+total_consumed += n;
+break;
+}
+}
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.

Review Comment:
   I can add that in. I was planning to bring over much more extensive test 
coverage in the follow-up PRs as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248586749


##
arrow-avro/src/schema.rs:
##
@@ -467,37 +465,50 @@ impl<'a> SchemaStore<'a> {
 }
 }
 
-#[inline]
 fn quote(s: &str) -> Result {
 serde_json::to_string(s)
 .map_err(|e| ArrowError::ComputeError(format!("Failed to quote string: 
{e}")))
 }
 
-fn make_fullname(name: &str, namespace_attr: Option<&str>, enclosing_ns: 
Option<&str>) -> String {
-match namespace_attr.or(enclosing_ns) {
-Some(ns) if !name.contains('.') => format!("{ns}.{name}"),
-_ => name.to_string(),
+// Avro names are defined by a `name` and an optional `namespace`.
+// The full name is composed of the namespace and the name, separated by a dot.
+//
+// Avro specification defines two ways to specify a full name:
+// 1. The `name` attribute contains the full name (e.g., "a.b.c.d").
+//In this case, the `namespace` attribute is ignored.
+// 2. The `name` attribute contains the simple name (e.g., "d") and the
+//`namespace` attribute contains the namespace (e.g., "a.b.c").
+//
+// Each part of the name must match the regex `^[A-Za-z_][A-Za-z0-9_]*$`.
+// Complex paths with quotes or backticks like `a."hi".b` are not supported.
+//
+// This function constructs the full name and extracts the namespace,
+// handling both ways of specifying the name. It prioritizes a namespace
+// defined within the `name` attribute itself, then the explicit 
`namespace_attr`,
+// and finally the `enclosing_ns`.
+fn make_full_name(
+name: &str,
+namespace_attr: Option<&str>,
+enclosing_ns: Option<&str>,
+) -> Result<(String, Option), ArrowError> {

Review Comment:
   I iterated on this function a few times last night and neglected to update 
the signature. I'll get that handled.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248584861


##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +330,98 @@ impl ReaderBuilder {
 /// - `batch_size` = 1024
 /// - `strict_mode` = false
 /// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fp` = None
+/// - `static_store_mode` = false
 pub fn new() -> Self {
 Self::default()
 }
 
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) -> 
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(), 
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let field_builder = match reader_schema {
+Some(rs) if !compare_schemas(writer_schema, rs)? => {
+AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)
+}
+Some(rs) => AvroFieldBuilder::new(rs),
+None => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode);
+let root = field_builder.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
 }
 
-fn build_impl(self, reader: &mut R) -> Result<(Header, 
Decoder), ArrowError> {
-let header = read_header(reader)?;
-let record_decoder = if let Some(schema) = &self.schema {
-self.make_record_decoder(schema)?
-} else {
-let avro_schema: Option> = header
-.schema()
-.map_err(|e| ArrowError::ExternalError(Box::new(e)))?;
-let avro_schema = avro_schema.ok_or_else(|| {
-ArrowError::ParseError("No Avro schema present in file 
header".to_string())
-})?;
-self.make_record_decoder(&avro_schema)?
-};
-let decoder = Decoder::new(record_decoder, self.batch_size);
-Ok((header, decoder))
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+Decoder {
+batch_size: self.batch_size,
+remaining_capacity: self.batch_size,
+active_fingerprint,
+active_decoder,
+cache: IndexMap::new(),
+max_cache_size: self.decoder_cache_size,
+reader_schema,
+utf8_view: self.utf8_view,
+writer_schema_store,
+strict_mode: self.strict_mode,
+pending_schema: None,
+}
+}
+
+fn make_decoder(&self, header: Option<&Header>) -> Result {
+match header {
+Some(hdr) => {
+let writer_schema = hdr
+.schema()
+.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
+.ok_or_else(|| {
+ArrowError::ParseError("No Avro schema present in file 
header".into())
+})?;
+let record_decoder =
+self.make_record_decoder(&writer_schema, 
self.reader_schema.as_ref())?;
+Ok(self.make_decoder_with_parts(record_decoder, None, None, 
None))
+}
+None => {
+let reader_schema = self.reader_schema.clone().ok_or_else(|| {
+ArrowError::ParseError("Reader schema required for raw 
Avro".into())
+})?;
+let (init_fingerprint, initial_decoder) =
+if let (Some(schema_store), Some(fingerprint)) =
+(&self.writer_schema_store, self.active_fingerprint)
+{
+// An initial fingerprint is provided, use it to look 
up the first schema.
+let writer_schema = 
schema_store.lookup(&fingerprint).ok_or_else(|| {
+ArrowError::ParseError(
+"Active fingerprint not found in schema 
store".into(),
+)
+})?;
+let decoder =
+self.make_record_decoder(writer_schema, 
Some(&reader_schema))?;
+(Some(fingerprint), decoder)
+} else {
+// No initial fingerprint; the first record must 
contain one.
+// A decoder is created from the reader schema only.
+let decoder = self.make_record_decoder(&reader_schema, 
None)?;
+   

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248580794


##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +365,97 @@ impl ReaderBuilder {
 /// - `batch_size` = 1024
 /// - `strict_mode` = false
 /// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
 pub fn new() -> Self {
 Self::default()
 }
 
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) -> 
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(), 
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let field_builder = match reader_schema {
+Some(rs) if !compare_schemas(writer_schema, rs)? => {
+AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)
+}
+Some(rs) => AvroFieldBuilder::new(rs),
+None => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode);
+let root = field_builder.build()?;

Review Comment:
   @scovich There's some conditional logic that I needed to capture here.
   
   1. If the `reader_schema` is provided and it is different from the 
`writer_schema`, then we will have schema resolution. In that scenario we need 
to pass in both the `writer_schema` and `reader_schema` into the 
`AvroFieldBuilder`, i.e. 
`AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)` (I'll rename `rs` 
that is pretty bad).
   2. If there's a `reader_schema` and it's identical to the `writer_schema` 
then we will have no schema resolution, it just becomes added overhead.
   3. If there's no `reader_schema` then there's no schema resolution and we 
use the `writer_schema`.
   
   That being said I will clean this code up because it's neither clear nor 
optimal. For example I could have just merged branches 2 and 3. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248559975


##
arrow-avro/src/schema.rs:
##
@@ -467,37 +465,50 @@ impl<'a> SchemaStore<'a> {
 }
 }
 
-#[inline]
 fn quote(s: &str) -> Result {
 serde_json::to_string(s)
 .map_err(|e| ArrowError::ComputeError(format!("Failed to quote string: 
{e}")))
 }
 
-fn make_fullname(name: &str, namespace_attr: Option<&str>, enclosing_ns: 
Option<&str>) -> String {
-match namespace_attr.or(enclosing_ns) {
-Some(ns) if !name.contains('.') => format!("{ns}.{name}"),
-_ => name.to_string(),
+// Avro names are defined by a `name` and an optional `namespace`.
+// The full name is composed of the namespace and the name, separated by a dot.
+//
+// Avro specification defines two ways to specify a full name:
+// 1. The `name` attribute contains the full name (e.g., "a.b.c.d").
+//In this case, the `namespace` attribute is ignored.
+// 2. The `name` attribute contains the simple name (e.g., "d") and the
+//`namespace` attribute contains the namespace (e.g., "a.b.c").
+//
+// Each part of the name must match the regex `^[A-Za-z_][A-Za-z0-9_]*$`.
+// Complex paths with quotes or backticks like `a."hi".b` are not supported.
+//
+// This function constructs the full name and extracts the namespace,
+// handling both ways of specifying the name. It prioritizes a namespace
+// defined within the `name` attribute itself, then the explicit 
`namespace_attr`,
+// and finally the `enclosing_ns`.

Review Comment:
   ty!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248553873


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and 
stop.
+total_consumed += n;
+break;
+}
 }
-let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
-total_consumed += n;
-self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
+// Attempt to handle a single‑object‑encoding prefix at the current 
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller 
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and 
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint::<8>(&buf[SINGLE_OBJECT_MAGIC.len()..], 
|bytes| {

Review Comment:
   Good call, ty for that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248544320


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}

Review Comment:
   My intention was to use `Some(0)` to cover a scenario where magic bytes were 
read, but there wasn't enough remaining bytes in the buffer for the 
fingerprint. My thought was to then return so the caller could add more bytes. 
I did just realize I'd need to change a mistake on L217 though for that.
   
   After digging into the Java implementation's code however I noticed they 
throw an error for this scenario: 
https://github.com/apache/avro/blob/0606c33514e084cc0e131ab1f12a1a91e68e07c1/lang/java/avro/src/main/java/org/apache/avro/message/BinaryMessageDecoder.java#L150



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248544320


##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
 FingerprintAlgorithm::Rabin,
 SchemaStore::fingerprint_algorithm,
 );
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
 while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
-// a schema change is detected or there are insufficient bytes 
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n = 
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}

Review Comment:
   My intention was to use `Some(0)` to cover a scenario where magic bytes were 
read, but there wasn't enough remaining bytes in the buffer for the 
fingerprint. My thought was to then return so the caller could add more bytes.
   
   After digging into the Java implementation's code however I noticed they 
throw an error for this scenario: 
https://github.com/apache/avro/blob/0606c33514e084cc0e131ab1f12a1a91e68e07c1/lang/java/avro/src/main/java/org/apache/avro/message/BinaryMessageDecoder.java#L150



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248529510


##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
 self
 }
 
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
 ///
 /// If a schema is not provided, the schema will be read from the Avro 
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) -> 
Self {
+self.reader_schema = Some(reader_schema);
 self
 }
 
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that 
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) -> 
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded 
data.
+///
+/// This is useful when the data stream does not begin with a schema 
definition
+/// or fingerprint, allowing the decoder to start with a known schema from 
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may 
need to switch
+/// between different decoders. This cache avoids rebuilding them from 
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|

Review Comment:
   Or, if it's correct, and `reader_schema` _is_ always required, then we can 
eliminate a dimension from the truth table and greatly simplify the validation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248525484


##
arrow-avro/src/codec.rs:
##
@@ -180,15 +182,15 @@ impl<'a> TryFrom<&Schema<'a>> for AvroField {
 /// Builder for an [`AvroField`]
 #[derive(Debug)]
 pub struct AvroFieldBuilder<'a> {
-writer_schema: &'a Schema<'a>,
-reader_schema: Option<&'a Schema<'a>>,
+writer_schema: &'a AvroSchema<'a>,
+reader_schema: Option<&'a AvroSchema<'a>>,

Review Comment:
   That sounds great. Reduces noise on this PR, and we can evaluate 
`AvroSchema` vs. `Schema` on its own merits in a separate PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248512537


##
arrow-avro/src/codec.rs:
##
@@ -180,15 +182,15 @@ impl<'a> TryFrom<&Schema<'a>> for AvroField {
 /// Builder for an [`AvroField`]
 #[derive(Debug)]
 pub struct AvroFieldBuilder<'a> {
-writer_schema: &'a Schema<'a>,
-reader_schema: Option<&'a Schema<'a>>,
+writer_schema: &'a AvroSchema<'a>,
+reader_schema: Option<&'a AvroSchema<'a>>,

Review Comment:
   I originally thought the same and my personal preference was to refer to it 
as `Schema`. 
   
   Then I thought it maybe easier for people who aren't familiar with this 
crate to read the code if it was aliased as `AvroSchema`. Due to the size of 
this PR though, I can just revert it back for now. It would probably be best to 
tackle it explicitly in a dedicated PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248037550


##
arrow-avro/src/codec.rs:
##
@@ -180,15 +182,15 @@ impl<'a> TryFrom<&Schema<'a>> for AvroField {
 /// Builder for an [`AvroField`]
 #[derive(Debug)]
 pub struct AvroFieldBuilder<'a> {
-writer_schema: &'a Schema<'a>,
-reader_schema: Option<&'a Schema<'a>>,
+writer_schema: &'a AvroSchema<'a>,
+reader_schema: Option<&'a AvroSchema<'a>>,

Review Comment:
   This seems like a pretty sweeping change (affects lots of other code outside 
this PR). Given that we're in the avro crate, maybe it's ok (or even better?) 
to let `Schema` be the avro schema, and just use `ArrowSchema[Ref]` for the 
exceptional cases where we're not dealing with avro schemas?
   
   Then again, we're surrounded by other `AvroXxx` typenames here, so maybe the 
change is just making things more uniform?
   
   What are your thoughts?



##
arrow-avro/src/codec.rs:
##
@@ -474,7 +477,7 @@ impl<'a> Resolver<'a> {
 }
 }
 
-/// Parses a [`AvroDataType`] from the provided [`Schema`] and the given 
`name` and `namespace`
+/// Parses a [`AvroDataType`] from the provided `Schema` and the given `name` 
and `namespace`

Review Comment:
   ```suggestion
   /// Parses a [`AvroDataType`] from the provided `schema` and the given 
`name` and `namespace`
   ```



##
arrow-avro/src/schema.rs:
##
@@ -467,37 +465,50 @@ impl<'a> SchemaStore<'a> {
 }
 }
 
-#[inline]
 fn quote(s: &str) -> Result {
 serde_json::to_string(s)
 .map_err(|e| ArrowError::ComputeError(format!("Failed to quote string: 
{e}")))
 }
 
-fn make_fullname(name: &str, namespace_attr: Option<&str>, enclosing_ns: 
Option<&str>) -> String {
-match namespace_attr.or(enclosing_ns) {
-Some(ns) if !name.contains('.') => format!("{ns}.{name}"),
-_ => name.to_string(),
+// Avro names are defined by a `name` and an optional `namespace`.
+// The full name is composed of the namespace and the name, separated by a dot.
+//
+// Avro specification defines two ways to specify a full name:
+// 1. The `name` attribute contains the full name (e.g., "a.b.c.d").
+//In this case, the `namespace` attribute is ignored.
+// 2. The `name` attribute contains the simple name (e.g., "d") and the
+//`namespace` attribute contains the namespace (e.g., "a.b.c").
+//
+// Each part of the name must match the regex `^[A-Za-z_][A-Za-z0-9_]*$`.
+// Complex paths with quotes or backticks like `a."hi".b` are not supported.
+//
+// This function constructs the full name and extracts the namespace,
+// handling both ways of specifying the name. It prioritizes a namespace
+// defined within the `name` attribute itself, then the explicit 
`namespace_attr`,
+// and finally the `enclosing_ns`.

Review Comment:
   Really nice comment!



##
arrow-avro/src/schema.rs:
##
@@ -467,37 +465,50 @@ impl<'a> SchemaStore<'a> {
 }
 }
 
-#[inline]
 fn quote(s: &str) -> Result {
 serde_json::to_string(s)
 .map_err(|e| ArrowError::ComputeError(format!("Failed to quote string: 
{e}")))
 }
 
-fn make_fullname(name: &str, namespace_attr: Option<&str>, enclosing_ns: 
Option<&str>) -> String {
-match namespace_attr.or(enclosing_ns) {
-Some(ns) if !name.contains('.') => format!("{ns}.{name}"),
-_ => name.to_string(),
+// Avro names are defined by a `name` and an optional `namespace`.
+// The full name is composed of the namespace and the name, separated by a dot.
+//
+// Avro specification defines two ways to specify a full name:
+// 1. The `name` attribute contains the full name (e.g., "a.b.c.d").
+//In this case, the `namespace` attribute is ignored.
+// 2. The `name` attribute contains the simple name (e.g., "d") and the
+//`namespace` attribute contains the namespace (e.g., "a.b.c").
+//
+// Each part of the name must match the regex `^[A-Za-z_][A-Za-z0-9_]*$`.
+// Complex paths with quotes or backticks like `a."hi".b` are not supported.
+//
+// This function constructs the full name and extracts the namespace,
+// handling both ways of specifying the name. It prioritizes a namespace
+// defined within the `name` attribute itself, then the explicit 
`namespace_attr`,
+// and finally the `enclosing_ns`.
+fn make_full_name(
+name: &str,
+namespace_attr: Option<&str>,
+enclosing_ns: Option<&str>,
+) -> Result<(String, Option), ArrowError> {

Review Comment:
   I don't see any code in this function that returns Err?



##
arrow-avro/src/schema.rs:
##
@@ -260,13 +276,389 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248380990


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,134 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is 
None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
+// a schema change is detected or there are insufficient bytes 
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
 /// Produce a `RecordBatch` if at least one row is fully decoded, returning
 /// `Ok(None)` if no new rows are available.
 pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) = 
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) = 
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder, 
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() || 
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let full_len = prefix_len(hash_type);
+if buf.len() < full_len {
+return Ok(Some(0)); // Not enough data to read the full fingerprint
+}
+let fp_bytes = &buf[2..full_len];
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {
+return Err(ArrowError::ParseError(format!(
+"Invalid Rabin fingerprint length, expected 8, got {}",
+fp_bytes.len()
+)));
+};
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+}
+};

Review Comment:
   @scovich Actually the approach you suggested below was an extremely clean 
and elegant way to implement this. I hadn't fully gone through it when I wrote 
the prior comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248375384


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is 
None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
+// a schema change is detected or there are insufficient bytes 
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
 /// Produce a `RecordBatch` if at least one row is fully decoded, returning
 /// `Ok(None)` if no new rows are available.
 pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) = 
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) = 
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder, 
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() || 
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);

Review Comment:
   > I think my only opinion is: Not in this PR.
   
   100% agree with you on that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248022723


##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +450,70 @@ impl ReaderBuilder {
 self
 }
 
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
 ///
 /// If a schema is not provided, the schema will be read from the Avro 
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) -> 
Self {
+self.reader_schema = Some(reader_schema);
 self
 }
 
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that 
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) -> 
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded 
data.
+///
+/// This is useful when the data stream does not begin with a schema 
definition
+/// or fingerprint, allowing the decoder to start with a known schema from 
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may 
need to switch
+/// between different decoders. This cache avoids rebuilding them from 
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+fn validate(&self) -> Result<(), ArrowError> {
+match (
+self.writer_schema_store.as_ref(),
+self.reader_schema.as_ref(),
+self.active_fingerprint.as_ref(),
+) {
+(Some(_), None, _) => Err(ArrowError::ParseError(
+"Reader schema must be set when writer schema store is 
provided".into(),
+)),
+(None, _, Some(_)) => Err(ArrowError::ParseError(
+"Active fingerprint requires a writer schema store".into(),
+)),
+_ => Ok(()),
+}
+}
+
 /// Create a [`Reader`] from this builder and a `BufRead`
 pub fn build(self, mut reader: R) -> Result, 
ArrowError> {
-let (header, decoder) = self.build_impl(&mut reader)?;
+self.validate()?;
+let header = read_header(&mut reader)?;
+let decoder = self.make_decoder(Some(&header))?;
 Ok(Reader {
 reader,
 header,

Review Comment:
   Oh! I missed that `header` is actually used twice and so needs to be a named 
value. Sorry about that. So really the suggestion should have been to inline 
the `decoder` value that is only used once. But we're definitely in nitpick 
territory at this point.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248016140


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is 
None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
+// a schema change is detected or there are insufficient bytes 
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
 /// Produce a `RecordBatch` if at least one row is fully decoded, returning
 /// `Ok(None)` if no new rows are available.
 pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) = 
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) = 
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder, 
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() || 
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);

Review Comment:
   > > From what I can tell, this whole operation should actually be infallible
   > 
   > There's actually one scenario this operation is fallible imo, and that's 
if the `Fingerprint` isn't found in the `SchemaStore`, i.e.
   > 
   > ```rust
   > let writer_schema = store.lookup(&new_fingerprint).ok_or_else(|| {
   > ArrowError::ParseError(format!("Unknown fingerprint: 
{new_fingerprint:?}"))
   > })?;
   > ```
   
   Ah, I somehow totally missed that. You are of course correct. 
   
   > I've been toying with the idea of defining a `SchemaStore` trait. This 
would enable users to create their own `SchemaStore` implementations... I'm 
curious what you think of the idea?
   
   I think my only opinion is: Not in this PR.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3143694026

   @scovich I got those latest changes pushed up. Let me know what you think.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-08-01 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2247182724


##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +450,70 @@ impl ReaderBuilder {
 self
 }
 
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
 ///
 /// If a schema is not provided, the schema will be read from the Avro 
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) -> 
Self {
+self.reader_schema = Some(reader_schema);
 self
 }
 
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that 
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) -> 
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded 
data.
+///
+/// This is useful when the data stream does not begin with a schema 
definition
+/// or fingerprint, allowing the decoder to start with a known schema from 
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may 
need to switch
+/// between different decoders. This cache avoids rebuilding them from 
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+fn validate(&self) -> Result<(), ArrowError> {
+match (
+self.writer_schema_store.as_ref(),
+self.reader_schema.as_ref(),
+self.active_fingerprint.as_ref(),
+) {
+(Some(_), None, _) => Err(ArrowError::ParseError(
+"Reader schema must be set when writer schema store is 
provided".into(),
+)),
+(None, _, Some(_)) => Err(ArrowError::ParseError(
+"Active fingerprint requires a writer schema store".into(),
+)),
+_ => Ok(()),
+}
+}
+
 /// Create a [`Reader`] from this builder and a `BufRead`
 pub fn build(self, mut reader: R) -> Result, 
ArrowError> {
-let (header, decoder) = self.build_impl(&mut reader)?;
+self.validate()?;
+let header = read_header(&mut reader)?;
+let decoder = self.make_decoder(Some(&header))?;
 Ok(Reader {
 reader,
 header,

Review Comment:
   @scovich If you're alright with it, I'd like to follow up on this one in a 
future PR. There's some trickiness to this one (Header has a `Schema` with a 
lifetime in it, I can't push it down too far or it turns into a rabbit hole). 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-31 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2246957540


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is 
None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
+// a schema change is detected or there are insufficient bytes 
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
 /// Produce a `RecordBatch` if at least one row is fully decoded, returning
 /// `Ok(None)` if no new rows are available.
 pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) = 
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) = 
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder, 
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() || 
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);

Review Comment:
   @scovich I'm finishing these changes now and getting close to pushing up. 
   
   > From what I can tell, this whole operation should actually be infallible
   
   There's actually one scenario this operation is fallible imo, and that's if 
the `Fingerprint` isn't found in the `SchemaStore`, i.e.
   
   ```rust
   let writer_schema = store.lookup(&new_fingerprint).ok_or_else(|| {
   ArrowError::ParseError(format!("Unknown fingerprint: 
{new_fingerprint:?}"))
   })?;
   ```
   
   The soonest we could get the new `Fingerprint` is in either 
`handle_fingerprint` or `handle_prefix` (not as clean imo). A `SchemaStore` 
missing an otherwise valid `Fingerprint` should propagate an error back to the 
caller. This will allow the caller to then recreate the `Decoder` with an 
updated `SchemaStore`, etc.
   
   As a side note, I've been toying with the idea of defining a `SchemaStore` 
trait. This would enable users to create their own `SchemaStore` 
implementations, including ones which could 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-31 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2245962100


##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+/// 
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified 
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form 
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+/// 

+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas 
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is 
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single 
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its 
operations.
+/// This ensures consistency when generating fingerprints and looking up 
schemas.
+/// All schemas registered will have their fingerprint computed with this 
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string 
slices
+/// contained within the stored [`Schema`] objects. This means the 
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-31 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2245932943


##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+/// 
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified 
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form 
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+/// 

+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas 
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is 
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single 
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its 
operations.
+/// This ensures consistency when generating fingerprints and looking up 
schemas.
+/// All schemas registered will have their fingerprint computed with this 
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string 
slices
+/// contained within the stored [`Schema`] objects. This means the 
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-31 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2245920424


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,134 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is 
None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
+// a schema change is detected or there are insufficient bytes 
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
 /// Produce a `RecordBatch` if at least one row is fully decoded, returning
 /// `Ok(None)` if no new rows are available.
 pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) = 
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) = 
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder, 
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() || 
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let full_len = prefix_len(hash_type);
+if buf.len() < full_len {
+return Ok(Some(0)); // Not enough data to read the full fingerprint
+}
+let fp_bytes = &buf[2..full_len];
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {
+return Err(ArrowError::ParseError(format!(
+"Invalid Rabin fingerprint length, expected 8, got {}",
+fp_bytes.len()
+)));
+};
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+}
+};

Review Comment:
   My main concern is the combination of split logic and magic constants 
produces a large bug surface. Hard to get the code correct, hard for others to 
convince themselves it's correct, and easy for future changes to break things 
if one part of the code ever gets out of sync.
   
   It's less clear to me whether that needs 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-31 Thread via GitHub


scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2245909595


##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+/// 
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified 
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form 
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+/// 

+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas 
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is 
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single 
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its 
operations.
+/// This ensures consistency when generating fingerprints and looking up 
schemas.
+/// All schemas registered will have their fingerprint computed with this 
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string 
slices
+/// contained within the stored [`Schema`] objects. This means the 
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-31 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244570973


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is 
None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
+// a schema change is detected or there are insufficient bytes 
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
 /// Produce a `RecordBatch` if at least one row is fully decoded, returning
 /// `Ok(None)` if no new rows are available.
 pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) = 
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) = 
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder, 
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() || 
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);

Review Comment:
   > suppose there really is a schema change, but the buffer has only 1 byte 
left. Then by returning None we would tell our caller that no schema change is 
coming and the caller would attempt to consume the byte as part of a (possibly 
single-byte) record instead.
   
   Ah, I think I see what you mean. Also I really like your suggested approach. 
I'll pick this back up in the morning and finish incorporating these 
recommendations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-31 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244531096


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is 
None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
+// a schema change is detected or there are insufficient bytes 
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
 /// Produce a `RecordBatch` if at least one row is fully decoded, returning
 /// `Ok(None)` if no new rows are available.
 pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) = 
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) = 
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder, 
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() || 
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let fp_bytes = &buf[2..]; // safe thanks to the `starts_with` check 
above
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {

Review Comment:
   This is extremely useful information. I'm going through this now. 
   
   > but I don't know if that "extract an array from a slice at a given offset" 
pattern arises often enough in this crate to be worth creating a similar helper 
here?
   
   I haven't come across that yet in this crate, but if I do again I may as 
well make a helper.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244485742


##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+/// 
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified 
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form 
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+/// 

+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas 
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is 
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single 
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its 
operations.
+/// This ensures consistency when generating fingerprints and looking up 
schemas.
+/// All schemas registered will have their fingerprint computed with this 
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string 
slices
+/// contained within the stored [`Schema`] objects. This means the 
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244483679


##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+/// 
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified 
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form 
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+/// 

+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas 
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is 
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single 
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its 
operations.
+/// This ensures consistency when generating fingerprints and looking up 
schemas.
+/// All schemas registered will have their fingerprint computed with this 
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string 
slices
+/// contained within the stored [`Schema`] objects. This means the 
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244482440


##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+/// 
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified 
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form 
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+/// 

+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas 
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is 
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single 
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its 
operations.
+/// This ensures consistency when generating fingerprints and looking up 
schemas.
+/// All schemas registered will have their fingerprint computed with this 
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string 
slices
+/// contained within the stored [`Schema`] objects. This means the 
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244481107


##
arrow-avro/src/reader/mod.rs:
##
@@ -520,20 +498,13 @@ impl ReaderBuilder {
 self.writer_schema_store.as_ref(),
 self.reader_schema.as_ref(),
 self.active_fingerprint.as_ref(),
-self.static_store_mode,
 ) {
-(Some(_), None, _, _) => Err(ArrowError::ParseError(
+(Some(_), None, _) => Err(ArrowError::ParseError(
 "Reader schema must be set when writer schema store is 
provided".into(),
 )),
-(None, _, Some(_), _) => Err(ArrowError::ParseError(
+(None, _, Some(_)) => Err(ArrowError::ParseError(
 "Active fingerprint requires a writer schema store".into(),
 )),
-(None, _, _, true) => Err(ArrowError::ParseError(
-"static_store_mode=true requires a writer schema store".into(),
-)),
-(Some(_), _, None, true) => Err(ArrowError::ParseError(
-"static_store_mode=true requires an active fingerprint".into(),
-)),
 _ => Ok(()),

Review Comment:
   I pushed up those improvements and included a truth table in the comments. 
Really good idea.
   
   Also I removed the `self.validate()?;` call from the `ReaderBuilder::build` 
method since it's only really needed in `ReaderBuilder::build_decoder`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244476447


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,134 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is 
None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
+// a schema change is detected or there are insufficient bytes 
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
 /// Produce a `RecordBatch` if at least one row is fully decoded, returning
 /// `Ok(None)` if no new rows are available.
 pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) = 
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) = 
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder, 
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() || 
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let full_len = prefix_len(hash_type);
+if buf.len() < full_len {
+return Ok(Some(0)); // Not enough data to read the full fingerprint
+}
+let fp_bytes = &buf[2..full_len];
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {
+return Err(ArrowError::ParseError(format!(
+"Invalid Rabin fingerprint length, expected 8, got {}",
+fp_bytes.len()
+)));
+};
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+}
+};
+// If the fingerprint indicates a schema change, prepare to switch 
decoders.
+if self.active_fingerprint != Some(new_fp) {
+self.prepare_schema_switch(new_fp)?;
+// If there are already decoded rows, we must flush them first.
+// Forcing the batch to be full ensures `flush` is called 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244475539


##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,134 @@ impl Decoder {
 ///
 /// Returns the number of bytes consumed.
 pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first 
message \
+ (writer_schema_store is set but active_fingerprint is 
None)"
+.into(),
+));
+}
 let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows < 
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..], 
1)?;
-// A successful call to record_decoder.decode means one row was 
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid 
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an 
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) = 
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It 
may be completed early if
+// a schema change is detected or there are insufficient bytes 
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
 }
 Ok(total_consumed)
 }
 
 /// Produce a `RecordBatch` if at least one row is fully decoded, returning
 /// `Ok(None)` if no new rows are available.
 pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) = 
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) = 
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder, 
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() || 
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let full_len = prefix_len(hash_type);
+if buf.len() < full_len {
+return Ok(Some(0)); // Not enough data to read the full fingerprint
+}
+let fp_bytes = &buf[2..full_len];
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {
+return Err(ArrowError::ParseError(format!(
+"Invalid Rabin fingerprint length, expected 8, got {}",
+fp_bytes.len()
+)));
+};
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+}
+};

Review Comment:
   @scovich 
   
   I just saw your other newest comments and I'm about to address the `<[u8; 
8]>::try_from(fingerprint_bytes)` issue you raised.
   
   I left `prefix_len` in and moved `let fingerprint_bytes = &buf[2..];` to 
inside of the `FingerprintAlgorithm::Rabin` branch because I planned to fast 
follow on getting 

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub


jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r226409


##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
 pub attributes: Attributes<'a>,
 }
 
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come 
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+/// 
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified 
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form 
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+/// 

+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas 
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is 
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single 
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its 
operations.
+/// This ensures consistency when generating fingerprints and looking up 
schemas.
+/// All schemas registered will have their fingerprint computed with this 
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string 
slices
+/// contained within the stored [`Schema`] objects. This means the 
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// 

  1   2   >