Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
alamb merged PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
alamb commented on PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3165620921 🚀 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2255542788
##
arrow-avro/src/reader/mod.rs:
##
@@ -158,39 +162,131 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.expect_prefix
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(&mut self, buf: &[u8]) -> Result,
ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if !self.expect_prefix {
+return Ok(None);
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match self.fingerprint_algorithm {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes|
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+// NOTE: Incomplete fingerprint consumes no bytes.
+let consumed = fingerprint_size.map_or(0, |n| n +
SINGLE_OBJECT_MAGIC.len());
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Insufficient bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+let new_decoder = self.cache.shift_remove(&new_fingerprint);
+let new_decoder = match new_decoder {
+Some(decoder) => decoder,
+None => {
+return Err(ArrowError::ParseError(format!(
+"Unknown fingerprint: {new_fingerprint:?}"
+)))
+}
+};
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3156930240 > Good enough for now, I think. > > There are lots of follow-ups to simplify the logic, especially places where we funnel two distinct cases through one code path that has to tease them back apart. But I think there's ongoing work to address those? 100% I have that code mostly ready. Should only take me a few hours to get that follow-up PR up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2255515847
##
arrow-avro/src/reader/mod.rs:
##
@@ -220,34 +320,119 @@ impl ReaderBuilder {
/// - `batch_size` = 1024
/// - `strict_mode` = false
/// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
pub fn new() -> Self {
Self::default()
}
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) ->
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(),
self.utf8_view)
+fn make_record_decoder(
+&self,
+writer_schema: &Schema,
+reader_schema: Option<&AvroSchema>,
+) -> Result {
+let root = match reader_schema {
+Some(reader_schema) => {
+
AvroFieldBuilder::new(writer_schema).with_reader_schema(reader_schema.clone())
+}
+_ => AvroFieldBuilder::new(writer_schema),
+}
Review Comment:
nit:
```suggestion
let mut builder = AvroFieldBuilder::new(writer_schema);
if let Some(reader_schema) = reader_schema {
builder = builder.with_reader_schema(reader_schema.clone());
}
let root = builder
```
##
arrow-avro/src/reader/mod.rs:
##
@@ -158,39 +162,131 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.expect_prefix
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..])? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(&mut self, buf: &[u8]) -> Result,
ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if !self.expect_prefix {
+return Ok(None);
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match self.fingerprint_algorithm {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes|
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3156567917 @scovich @alamb Now this one is much smaller. Was there any remaining changes we still needed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254942398
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> (I don't know how often we'd even be looking up schemas, in the common
case?)
For the streaming raw `Decoder` it will be very common imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254784213
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
Also I pushed these changes to the other PR as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254783570
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
Of course.
First rule of optimization: Don't!
(I don't know how often we'd even be looking up schemas, in the common case?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254741278
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
@scovich would you be alright if we revisit that in a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254342687
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
Nice! The only downside is, it has to deserialize (albeit in place) every
time we access the schema; it would be nice if there were a way to capture and
cache the result (alongside the backing string), but that's where lifetimes get
really messy unless you're willing to resort to tricks like interior mutability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2254342687
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
Nice! The only downside is, it has to deserialize (albeit in place) every
time we access the schema; it would be nice if there were a way to capture the
result, but that's where lifetimes get really messy unless you're willing to
resort to tricks like interior mutability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2253530422
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
@scovich I just pushed up an attempt at handling the schemas internally. I
managed to get rid of all of the `static` usage. Let me know what you think.
It's built around this new `AvroSchema` type in `schema.rs`:
```rust
/// A wrapper for an Avro schema in its JSON string representation.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct AvroSchema {
/// The Avro schema as a JSON string.
pub json_string: String,
}
impl AvroSchema {
/// Creates a new `AvroSchema` from a JSON string.
pub fn new(json_string: String) -> Self {
Self { json_string }
}
/// Deserializes and returns the `AvroSchema`.
///
/// The returned schema borrows from `self`.
pub fn schema(&self) -> Result, ArrowError> {
serde_json::from_str(self.json_string.as_str())
.map_err(|e| ArrowError::ParseError(format!("Invalid Avro schema
JSON: {e}")))
}
/// Returns the Rabin fingerprint of the schema.
pub fn fingerprint(&self) -> Result {
generate_fingerprint_rabin(&self.schema()?)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252866464
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> `HeaderDecoder` seems able to get around this by parsing a json string
from the first row of an Avro file during the `ReaderBuilder::build` process
The `serde_json` code is doing [zero-copy
deserialization](https://serde.rs/lifetimes.html) from the header's byte
buffer. There's an implicit lifetime as if declared by:
```rust
pub fn schema<'a>(&'a self) -> Result>, ArrowError> {
```
So the resulting schema inherits the lifetime of the header itself. AFAIK,
the only way we could get a similar effect here is to add a lifetime param to
the `Decoder`. I guess we could also try to create schemas internally, but
self-referential lifetimes tend to get really messy really fast in rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252856054
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> trying to think around the problem to keep this public api similar to the
other arrow crates.
AFAIK, all the other crates define self-standing schemas that don't need
lifetimes in the first place...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252820147
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
@scovich You're right. Here's where my research led me: Requiring `'static`
lifetimes for Avro schemas means either restricting schemas to static data or
leaking heap memory for dynamic schema components.
One thing I noticed is that the `HeaderDecoder` seems able to get around
this by parsing a json string from the first row of an Avro file during the
`ReaderBuilder::build` process, i.e.
```rust
pub fn schema(&self) -> Result>, ArrowError> {
self.get(SCHEMA_METADATA_KEY)
.map(|x| {
serde_json::from_slice(x).map_err(|e| {
ArrowError::ParseError(format!("Failed to parse Avro
schema JSON: {e}"))
})
})
.transpose()
}
```
Do you think an approach that uses JSON Strings instead of `Schema`, for
instance:
```rust
/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
pub fn with_reader_schema(mut self, reader_schema: String) -> Self {
...
}
/// Sets the `SchemaStore` used for resolving writer schemas.
///
/// This is necessary when decoding single-object encoded data that
identifies
/// schemas by a fingerprint. The store allows the decoder to look up the
/// full writer schema from a fingerprint embedded in the data.
///
/// Defaults to `None`.
pub fn with_writer_schema_store(mut self, store: HashMap) -> Self {
...
}
```
Then parses the each `Schema` inside the `build` and `build_decoder` methods
would be a way around this?
UPDATE:
Maybe we could also pre-build the `cache: IndexMap,` from JSON strings in the `ReaderBuilder::build_decoder` method
and then pass that into the new `Decoder` so we can remove:
```rust
reader_schema: Option>,
writer_schema_store: Option>,
```
We'll pay for this with start-up cost and memory footprint. I suspect for
smaller and even medium sized caches it will be worth it, not sure how large
the cache needs to be for it not to be worth it though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252820147
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
@scovich You're right. Here's where my research led me: Requiring `'static`
lifetimes for Avro schemas means either restricting schemas to static data or
leaking heap memory for dynamic schema components.
One thing I noticed is that the `HeaderDecoder` seems able to get around
this by parsing a json string from the first row of an Avro file during the
`ReaderBuilder::build` process, i.e.
```rust
pub fn schema(&self) -> Result>, ArrowError> {
self.get(SCHEMA_METADATA_KEY)
.map(|x| {
serde_json::from_slice(x).map_err(|e| {
ArrowError::ParseError(format!("Failed to parse Avro
schema JSON: {e}"))
})
})
.transpose()
}
```
Do you think an approach that uses JSON Strings instead of `Schema`, for
instance:
```rust
/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
pub fn with_reader_schema(mut self, reader_schema: String) -> Self {
...
}
/// Sets the `SchemaStore` used for resolving writer schemas.
///
/// This is necessary when decoding single-object encoded data that
identifies
/// schemas by a fingerprint. The store allows the decoder to look up the
/// full writer schema from a fingerprint embedded in the data.
///
/// Defaults to `None`.
pub fn with_writer_schema_store(mut self, store: HashMap) -> Self {
...
}
```
Then parses the each `Schema` inside the `build` and `build_decoder` methods
would be a way around this?
UPDATE:
Maybe we could also pre-build the `cache: IndexMap,` from JSON strings in the `ReaderBuilder::build_decoder` method
and then pass that into the new `Decoder` so we can remove:
```rust
reader_schema: Option>,
writer_schema_store: Option>,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252836569
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> (maybe the answer is just that Decoder and its builder need a lifetime
param so the compiler can figure out lifetimes based on the calling context?)
Also I don't have any issues with this answer. Just trying to think around
the problem to keep this public api similar to the other arrow crates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252820147
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
@scovich You're right. Here's where my research led me: Requiring `'static`
lifetimes for Avro schemas means either restricting schemas to static data or
leaking heap memory for dynamic schema components.
One thing I noticed is that the `HeaderDecoder` seems able to get around
this by parsing a json string from the first row of an Avro file during the
`ReaderBuilder::build` process, i.e.
```rust
pub fn schema(&self) -> Result>, ArrowError> {
self.get(SCHEMA_METADATA_KEY)
.map(|x| {
serde_json::from_slice(x).map_err(|e| {
ArrowError::ParseError(format!("Failed to parse Avro
schema JSON: {e}"))
})
})
.transpose()
}
```
Do you think an approach that uses JSON Strings instead of `Schema`, for
instance:
```rust
/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
pub fn with_reader_schema(mut self, reader_schema: String) -> Self {
...
}
/// Sets the `SchemaStore` used for resolving writer schemas.
///
/// This is necessary when decoding single-object encoded data that
identifies
/// schemas by a fingerprint. The store allows the decoder to look up the
/// full writer schema from a fingerprint embedded in the data.
///
/// Defaults to `None`.
pub fn with_writer_schema_store(mut self, store: HashMap) -> Self {
...
}
```
Then parses the each `Schema` inside the `build` and `build_decoder` methods
would be a way around this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252820147
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
@scovich You're right. Here's where my research led me: Requiring `'static`
lifetimes for Avro schemas means either restricting schemas to static data or
leaking heap memory for dynamic schema components.
One thing I noticed is that the `HeaderDecoder` seems able to get around
this by parsing a json string from the first row of an Avro file during the
`ReaderBuilder::build` process, i.e.
```rust
pub fn schema(&self) -> Result>, ArrowError> {
self.get(SCHEMA_METADATA_KEY)
.map(|x| {
serde_json::from_slice(x).map_err(|e| {
ArrowError::ParseError(format!("Failed to parse Avro
schema JSON: {e}"))
})
})
.transpose()
}
```
Do you think an approach that uses JSON Strings instead of `Schema`, for
instance:
```rust
/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
pub fn with_reader_schema(mut self, reader_schema: String) -> Self {
...
}
/// Sets the `SchemaStore` used for resolving writer schemas.
///
/// This is necessary when decoding single-object encoded data that
identifies
/// schemas by a fingerprint. The store allows the decoder to look up the
/// full writer schema from a fingerprint embedded in the data.
///
/// Defaults to `None`.
pub fn with_writer_schema_store(mut self, store: HashMap) -> Self {
...
}
```
Then parse the `Schema` inside the `build` and `build_decoder` methods would
be a way around this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252770776
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
(maybe the answer is just that `Decoder` and its builder need a lifetime
param so the compiler can figure out lifetimes based on the calling context?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252769379
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
The unit tests don't create any schemas with `TypeName::Ref` -- only
`TypeName::Primitive` -- and the `Attributes` all use string constants. But
what if the read schema were user input (i.e. parsed from JSON) and then fed to
a decoder?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252763116
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
I will admit I don't fully understand `'static` in rust. It doesn't
_necessarily_ imply a memory leak, but it does seem difficult (albeit
apparently not impossible) to apply `'static` lifetime to anything that will be
freed before the program terminates.
One way would be to never create a schema that actually uses `TypeName::Ref`
nor any `Attributes`, for example. But that's kind of cheating -- may as well
not support them at that point.
So I guess the question would be: Can one create a `Decoder` with Some
`reader_schema`, where the memory allocated to create that reader schema
actually gets freed when the decoder gets dropped?
A refinement of that question would be: Can one create a `Decoder` with Some
`reader_schema` that does use `Attributes` or `TypeName::Ref`, with dynamically
allocated strings that can be freed after the decoder is dropped?
Or, as I suspect, would one be forced to leak them (or use compile-time
constant strings instead), in order to get the code to compile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3152727580 > > To move forward, would it be best to simply focus on getting the `schema.rs` changes in first (~500-600 LOC) and then follow-up with a `arrow-avro/src/reader/mod.rs` PR? > > Interesting. By "schema.rs changes" you mean adding the new `SchemaStore` and fingerprinting infrastructure? With unit tests but not yet integrated into the actual decoder? That does seem like a good idea. I don't think there's any outstanding controversy in that part of the code (just a couple of follow-on items)? Thats correct and that's my understanding as well. > > I do apologize for both the size of this PR and the quality issues. > > This is _NOT_ the kind of run of the mill logic that I would normally associate with "quality issues." For me, at least, this has been an invigorating PR to review -- not a frustrating one. Some hard core software engineering here with depth and subtlety to work through. I really appreciate that! I just thought I should have caught some of that subtlety upfront. > > Let me know what I should do and I'll get on it immediately. > > Now that we know more, the split you propose seems quite reasonable; I'm not sure it was obvious two weeks ago tho? It crossed my mind honestly. I just thought that: 1. It would be helpful to see the context of how the `SchemaStore` would be used. 2. I saw the functionality as coupled (even if loosely) and wanted to ensure there was test coverage in `mod.rs` I'll get that PR up right now and link it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252735697
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> The main concern for this PR is:
>
> > Does that [static lifetimes here] mean memory they [the passed-in
schemas] reference must leak for all practical purposes?
I haven't had to resort to any memory leaks in `arrow-avro`. The `AvroField`
logic is also bound by the same lifetime and the `Schema` is only used to
create a root `AvroField` with a `Codec` which in turn is then used to create a
`RecordDecoder`. I haven't had to resort to any `box::leak` OR reference cycles
and I was careful to add bounding to the cache.
Also inside of the `Decoder` when making a new `RecordDecoder` (i.e.
`create_decoder_for`) I don't resort to using `clone` on either of the schemas
and no new `Schema` are created either. Each `RecordDecoder` can only use the
same `reader_schema` and set of `writer_schema`.
If I'm missing something however I apologize in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252735697
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> The main concern for this PR is:
>
> > Does that [static lifetimes here] mean memory they [the passed-in
schemas] reference must leak for all practical purposes?
I haven't had to resort to any memory leaks in `arrow-avro`. The `AvroField`
logic is also bound by the same lifetime and the `Schema` is only used to
create a root `AvroField` with a `Codec` which in turn is then used to create a
`RecordDecoder`. I haven't had to resort to any `box::leak` OR reference cycles
and I was careful to add bounding to the cache.
Also inside of the `Decoder` when making a new `RecordDecoder` I don't
resort to using `clone` on either of the schemas and no new `Schema` are
created either. Each `RecordDecoder` can only use the same `reader_schema` and
set of `writer_schema`.
If I'm missing something however I apologize in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252735697
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> The main concern for this PR is:
>
> > Does that [static lifetimes here] mean memory they [the passed-in
schemas] reference must leak for all practical purposes?
I haven't had to resort to any memory leaks here. The `AvroField` logic is
also bound by the same lifetime and the `Schema` is only used to create a root
`AvroField` with a `Codec` which in turn is then used to create a
`RecordDecoder`. I haven't had to resort to any `box::leak` OR reference cycles
and I was careful to add bounding to the cache.
Also inside of the `Decoder` when making a new `RecordDecoder` I don't
resort to using `clone` on either of the schemas and no new `Schema` are
created either. Each `RecordDecoder` can only use the same `reader_schema` and
set of `writer_schema`.
If I'm missing something however I apologize in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3152638873 > this PR has a lot of comments and quite a few refinements were needed. I also know it was way too big. Honestly, I think the "state machine" for avro decoding is just really complex. It felt like a lot of the questions and churn ultimately come from that underlying complexity. And my lack of familiarity with the avro spec. Not sure how easily those issues could be avoided merely by raising a smaller PR? > To move forward, would it be best to simply focus on getting the `schema.rs` changes in first (~500-600 LOC) and then follow-up with a `arrow-avro/src/reader/mod.rs` PR? Interesting. By "schema.rs changes" you mean adding the new `SchemaStore` and fingerprinting infrastructure? With unit tests but not yet integrated into the actual decoder? That does seem like a good idea. I don't think there's any outstanding controversy in that part of the code (just a couple of follow-on items)? > I do apologize for both the size of this PR and the quality issues. This is _NOT_ the kind of run of the mill logic that I would normally associate with "quality issues." For me, at least, this has been an invigorating PR to review -- not a frustrating one. Some hard core software engineering here with depth and subtlety to work through. > Let me know what I should do and I'll get on it immediately. Now that we know more, the split you propose seems quite reasonable; I'm not sure it was obvious two weeks ago tho? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252706588
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
The main concern for this PR is:
> Does that [static lifetimes here] mean memory they [the passed-in schemas]
reference must leak for all practical purposes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3152179396 @scovich @alamb I know this PR has a lot of comments and quite a few refinements were needed. I also know it was way too big. To move forward, would it be best to simply focus on getting the `schema.rs` changes in first (~500-600 LOC) and then follow-up with a `arrow-avro/src/reader/mod.rs` PR? I do apologize for both the size of this PR and the quality issues. I'll do better going forward. Thank you again @scovich for the amount of time and effort you've put into reviewing this. Let me know what I should do and I'll get on it immediately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2252426527
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
@scovich
> But is it really so expensive to just copy the string that we're willing
to deal with a proliferation of lifetimes?
In real world practice it's hard to say. That's one of the reasons I tried
to improve benchmarking in my other PR. So we can start getting numbers to
inform our changes.
> Or am I missing some obvious/easy place where the lifetimes make a big
difference that justifies the complexity?
I think the real source of the complexity here was my attempt to use the
same `Decoder` and `decode` method for decoding both single object encodings
and files. I have a plan to improve this in my very next PR before I do
anything else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251801967
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> (I don't actually understand why avro schemas have lifetimes in the first
place? Especially if they anyway require static lifetime in practice?)
I'm not 100% certain why that decision was made. I was simply trying to
extend from the original design pattern.
My opinion is we're taking advantage of an Avro nuance (knowing the schemas
in advance of record decoding) to minimize overhead via a borrowed schema
approach that leverages zero-copy parsing. This makes sense since every
scenario requires us to know the writer and reader schema (if using schema
evolution) in advance of instantiating a `RecordDecoder`.
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
Because the use of lifetimes predates both this PR and my involvement, would
you be ok with this being addressed in a follow-up PR prior to going public?
@scovich
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251943653
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
Because the use of lifetimes predates both this PR and my involvement, would
you be ok with this being addressed in a follow-up PR prior to going public?
@scovich
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251938878
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..],
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes|
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251862329
##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +369,91 @@ impl ReaderBuilder {
/// - `batch_size` = 1024
/// - `strict_mode` = false
/// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
pub fn new() -> Self {
Self::default()
}
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) ->
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(),
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let root = match reader_schema {
+Some(reader_schema) if !compare_schemas(writer_schema,
reader_schema)? => {
+
AvroFieldBuilder::new(writer_schema).with_reader_schema(reader_schema)
+}
+_ => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode)
+.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
+}
+
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+#[cfg(feature = "lru")]
+let capacity =
NonZeroUsize::new(self.decoder_cache_size).unwrap_or(NonZeroUsize::MIN); //
NonZeroUsize::MIN is 1
+Decoder {
+batch_size: self.batch_size,
+remaining_capacity: self.batch_size,
+active_fingerprint,
+active_decoder,
+#[cfg(feature = "lru")]
+cache: LruCache::new(capacity),
+#[cfg(not(feature = "lru"))]
+cache: IndexMap::new(),
+max_cache_size: self.decoder_cache_size,
+reader_schema,
+utf8_view: self.utf8_view,
+writer_schema_store,
+strict_mode: self.strict_mode,
+pending_schema: None,
+}
}
-fn build_impl(self, reader: &mut R) -> Result<(Header,
Decoder), ArrowError> {
-let header = read_header(reader)?;
-let record_decoder = if let Some(schema) = &self.schema {
-self.make_record_decoder(schema)?
-} else {
-let avro_schema: Option> = header
+fn make_decoder(&self, header: Option<&Header>) -> Result {
+if let Some(hdr) = header {
+let writer_schema = hdr
.schema()
-.map_err(|e| ArrowError::ExternalError(Box::new(e)))?;
-let avro_schema = avro_schema.ok_or_else(|| {
-ArrowError::ParseError("No Avro schema present in file
header".to_string())
+.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
+.ok_or_else(|| {
+ArrowError::ParseError("No Avro schema present in file
header".into())
+})?;
+let record_decoder =
+self.make_record_decoder(&writer_schema,
self.reader_schema.as_ref())?;
+return Ok(self.make_decoder_with_parts(record_decoder, None, None,
None));
+}
+let writer_schema_store =
self.writer_schema_store.as_ref().ok_or_else(|| {
+ArrowError::ParseError("Writer schema store required for raw
Avro".into())
+})?;
+let fingerprint = self
+.active_fingerprint
+.or_else(|| writer_schema_store.fingerprints().into_iter().next())
+.ok_or_else(|| {
+ArrowError::ParseError(
+"Writer schema store must contain at least one
schema".into(),
+)
})?;
-self.make_record_decoder(&avro_schema)?
-};
-let decoder = Decoder::new(record_decoder, self.batch_size);
-Ok((header, decoder))
+let writer_schema =
writer_schema_store.lookup(&fingerprint).ok_or_else(|| {
+ArrowError::ParseError("Active fingerprint not found in schema
store".into())
+})?;
+let record_decoder =
+self.make_record_decoder(writer_schema,
self.reader_schema.as_ref())?;
+Ok(self.make_decoder_with_parts(
+record_decoder,
+Some(fingerprint),
+self.reader_schema.clone(),
+self.writer_schema_store.clone(),
Review Comment:
> But what about the reader schema? What's the story behind the smell there?
What's happening here is the file read path won't ever switch
`RecordDecoder` during decoding
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251839467
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..],
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
Review Comment:
@scovich
I'm actually refactoring this code in my next PR to handle chunking partial
records. Is it okay if we pick this up there? It will be ~300 LOC and
exclusively focused on decoding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251801967
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> (I don't actually understand why avro schemas have lifetimes in the first
place? Especially if they anyway require static lifetime in practice?)
I'm not 100% certain why that decision was made. I was simply trying to
extend from the original design pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251797593
##
arrow-avro/Cargo.toml:
##
@@ -57,7 +57,8 @@ xz = { version = "0.1", default-features = false, optional =
true }
crc = { version = "3.0", optional = true }
strum_macros = "0.27"
uuid = "1.17"
-indexmap = "2.10"
+lru = { version = "0.12", optional = true }
+indexmap = "2.10.0"
Review Comment:
I'll change this to be `2.10`, mistake on my part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251801967
##
arrow-avro/src/reader/mod.rs:
##
@@ -124,23 +132,26 @@ fn read_header(mut reader: R) ->
Result {
/// A low-level interface for decoding Avro-encoded bytes into Arrow
`RecordBatch`.
#[derive(Debug)]
pub struct Decoder {
-record_decoder: RecordDecoder,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
batch_size: usize,
-decoded_rows: usize,
+remaining_capacity: usize,
+#[cfg(feature = "lru")]
+cache: LruCache,
+#[cfg(not(feature = "lru"))]
+cache: IndexMap,
+max_cache_size: usize,
+reader_schema: Option>,
+writer_schema_store: Option>,
Review Comment:
> (I don't actually understand why avro schemas have lifetimes in the first
place? Especially if they anyway require static lifetime in practice?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251790346
##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +369,91 @@ impl ReaderBuilder {
/// - `batch_size` = 1024
/// - `strict_mode` = false
/// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
pub fn new() -> Self {
Self::default()
}
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) ->
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(),
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let root = match reader_schema {
+Some(reader_schema) if !compare_schemas(writer_schema,
reader_schema)? => {
+
AvroFieldBuilder::new(writer_schema).with_reader_schema(reader_schema)
+}
+_ => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode)
+.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
+}
+
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+#[cfg(feature = "lru")]
+let capacity =
NonZeroUsize::new(self.decoder_cache_size).unwrap_or(NonZeroUsize::MIN); //
NonZeroUsize::MIN is 1
Review Comment:
The CI fmt job is aware of this crate and I always run fmt on the code
before pushing up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251781361
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..],
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes|
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251721555
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..],
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes|
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251720636
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..],
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes|
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2251711943
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..],
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes|
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249257375
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..],
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes|
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249240259
##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +369,91 @@ impl ReaderBuilder {
/// - `batch_size` = 1024
/// - `strict_mode` = false
/// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
pub fn new() -> Self {
Self::default()
}
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) ->
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(),
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let root = match reader_schema {
+Some(reader_schema) if !compare_schemas(writer_schema,
reader_schema)? => {
+
AvroFieldBuilder::new(writer_schema).with_reader_schema(reader_schema)
+}
+_ => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode)
+.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
+}
+
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+#[cfg(feature = "lru")]
+let capacity =
NonZeroUsize::new(self.decoder_cache_size).unwrap_or(NonZeroUsize::MIN); //
NonZeroUsize::MIN is 1
+Decoder {
+batch_size: self.batch_size,
+remaining_capacity: self.batch_size,
+active_fingerprint,
+active_decoder,
+#[cfg(feature = "lru")]
+cache: LruCache::new(capacity),
+#[cfg(not(feature = "lru"))]
+cache: IndexMap::new(),
+max_cache_size: self.decoder_cache_size,
+reader_schema,
+utf8_view: self.utf8_view,
+writer_schema_store,
+strict_mode: self.strict_mode,
+pending_schema: None,
+}
}
-fn build_impl(self, reader: &mut R) -> Result<(Header,
Decoder), ArrowError> {
-let header = read_header(reader)?;
-let record_decoder = if let Some(schema) = &self.schema {
-self.make_record_decoder(schema)?
-} else {
-let avro_schema: Option> = header
+fn make_decoder(&self, header: Option<&Header>) -> Result {
+if let Some(hdr) = header {
+let writer_schema = hdr
.schema()
-.map_err(|e| ArrowError::ExternalError(Box::new(e)))?;
-let avro_schema = avro_schema.ok_or_else(|| {
-ArrowError::ParseError("No Avro schema present in file
header".to_string())
+.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
+.ok_or_else(|| {
+ArrowError::ParseError("No Avro schema present in file
header".into())
+})?;
+let record_decoder =
+self.make_record_decoder(&writer_schema,
self.reader_schema.as_ref())?;
+return Ok(self.make_decoder_with_parts(record_decoder, None, None,
None));
+}
+let writer_schema_store =
self.writer_schema_store.as_ref().ok_or_else(|| {
+ArrowError::ParseError("Writer schema store required for raw
Avro".into())
+})?;
+let fingerprint = self
+.active_fingerprint
+.or_else(|| writer_schema_store.fingerprints().into_iter().next())
+.ok_or_else(|| {
+ArrowError::ParseError(
+"Writer schema store must contain at least one
schema".into(),
+)
})?;
-self.make_record_decoder(&avro_schema)?
-};
-let decoder = Decoder::new(record_decoder, self.batch_size);
-Ok((header, decoder))
+let writer_schema =
writer_schema_store.lookup(&fingerprint).ok_or_else(|| {
+ArrowError::ParseError("Active fingerprint not found in schema
store".into())
+})?;
+let record_decoder =
+self.make_record_decoder(writer_schema,
self.reader_schema.as_ref())?;
+Ok(self.make_decoder_with_parts(
+record_decoder,
+Some(fingerprint),
+self.reader_schema.clone(),
+self.writer_schema_store.clone(),
Review Comment:
This is a weird code smell. Some of the state comes from `self` internally,
while other state has to be copied manually by caller even tho it's _also_
coming from `self`?
We
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249229633
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +165,175 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(n) = self.handle_prefix(&data[total_consumed..],
hash_type)? {
+// We either consumed a prefix (n > 0) and need a schema
switch, or we need
+// more bytes to make a decision. Either way, this decoding
attempt is finished.
+total_consumed += n;
+}
+// No prefix: decode one row and keep going.
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+self.remaining_capacity -= 1;
+total_consumed += n;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint(&buf[SINGLE_OBJECT_MAGIC.len()..], |bytes|
{
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
fingerprint_from(fingerprint_bytes.try_into().unwrap());
+// If the fingerprint indicates a schema change, prepare to switch
decoders.
+if self.active_fingerprint != Some(new_fingerprint) {
+#[cfg(feature = "lru")]
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3146237014 @scovich I got those latest changes pushed up. Really improved the quality of the code. I definitely appreciate these deep reviews. I'm finishing up the development of single object encoding partial record chunking logic + some additional decoder cleanups. I plan to have that in a (much smaller) follow-up PR as soon as we get this one merged in. Let me know what you think and thank you again! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249126660
##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
self
}
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) ->
Self {
+self.reader_schema = Some(reader_schema);
self
}
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) ->
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded
data.
+///
+/// This is useful when the data stream does not begin with a schema
definition
+/// or fingerprint, allowing the decoder to start with a known schema from
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may
need to switch
+/// between different decoders. This cache avoids rebuilding them from
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|
+// | Some| Some--| None---| Ok-|
+// | Some| Some--| Some---| Ok-|
+fn validate(&self) -> Result<(), ArrowError> {
+match (
+self.writer_schema_store.is_some(),
+self.reader_schema.is_some(),
+self.active_fingerprint.is_some(),
+) {
+// Row 3: No store, reader schema present, no fingerprint
+(false, true, false)
+// Row 7: Store is present, reader schema is resent, no fingerprint
+| (true, true, false)
+// Row 8: Store present, reader schema present, fingerprint present
+| (true, true, true) => Ok(()),
+// Fingerprint without a store (rows 2 & 4)
+(false, _, true) => Err(ArrowError::InvalidArgumentError(
+"Active fingerprint requires a writer schema store".into(),
+)),
+// Store present but no reader schema (rows 5 & 6)
+(true, false, _) => Err(ArrowError::InvalidArgumentError(
+"Reader schema must be set when writer schema store is
provided".into(),
+)),
+// No schema store or reader schema provided (row 1)
+(false, false, _) => Err(ArrowError::InvalidArgumentError(
Review Comment:
@scovich So you were correct here. I was able to complete remove the
`validate` method and ended up simplifying a lot of the logic. For instance I
managed to slim down `make_decoder` to this:
```rust
fn make_decoder(&self, header: Option<&Header>) -> Result {
if let Some(hdr) = header {
let writer_schema = hdr
.schema()
.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
.ok_or_else(|| {
ArrowError::ParseError("No Avro schema present in file
header".into())
})?;
let record_decoder =
self.make_record_decoder(&writer_schema,
self.reader_schema.as_ref())?;
return Ok(self.make_decoder_with_parts(record_decoder, None,
None, None));
}
let writer_schema_store
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249126660
##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
self
}
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) ->
Self {
+self.reader_schema = Some(reader_schema);
self
}
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) ->
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded
data.
+///
+/// This is useful when the data stream does not begin with a schema
definition
+/// or fingerprint, allowing the decoder to start with a known schema from
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may
need to switch
+/// between different decoders. This cache avoids rebuilding them from
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|
+// | Some| Some--| None---| Ok-|
+// | Some| Some--| Some---| Ok-|
+fn validate(&self) -> Result<(), ArrowError> {
+match (
+self.writer_schema_store.is_some(),
+self.reader_schema.is_some(),
+self.active_fingerprint.is_some(),
+) {
+// Row 3: No store, reader schema present, no fingerprint
+(false, true, false)
+// Row 7: Store is present, reader schema is resent, no fingerprint
+| (true, true, false)
+// Row 8: Store present, reader schema present, fingerprint present
+| (true, true, true) => Ok(()),
+// Fingerprint without a store (rows 2 & 4)
+(false, _, true) => Err(ArrowError::InvalidArgumentError(
+"Active fingerprint requires a writer schema store".into(),
+)),
+// Store present but no reader schema (rows 5 & 6)
+(true, false, _) => Err(ArrowError::InvalidArgumentError(
+"Reader schema must be set when writer schema store is
provided".into(),
+)),
+// No schema store or reader schema provided (row 1)
+(false, false, _) => Err(ArrowError::InvalidArgumentError(
Review Comment:
@scovich So you were correct here. I was able to complete remove the
`validate` method and ended up simplifying a lot of the logic. For instance I
managed to slim down `make_decoder` to this:
```rust
fn make_decoder(&self, header: Option<&Header>) -> Result {
if let Some(hdr) = header {
let writer_schema = hdr
.schema()
.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
.ok_or_else(|| {
ArrowError::ParseError("No Avro schema present in file
header".into())
})?;
let record_decoder =
self.make_record_decoder(&writer_schema,
self.reader_schema.as_ref())?;
return Ok(self.make_decoder_with_parts(record_decoder, None,
None, None));
}
let writer_schema_store
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249126125
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +276,389 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+///
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified
`FingerprintAlgorithm`.
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+///
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its
operations.
+/// This ensures consistency when generating fingerprints and looking up
schemas.
+/// All schemas registered will have their fingerprint computed with this
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string
slices
+/// contained within the stored [`Schema`] objects. This means the
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each schema in the slice is registered with the
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249126029
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and
stop.
+total_consumed += n;
+break;
+}
+}
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint::<8>(&buf[SINGLE_OBJECT_MAGIC.len()..],
|bytes| {
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249125735
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +276,389 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+///
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified
`FingerprintAlgorithm`.
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+///
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its
operations.
+/// This ensures consistency when generating fingerprints and looking up
schemas.
+/// All schemas registered will have their fingerprint computed with this
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string
slices
+/// contained within the stored [`Schema`] objects. This means the
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each schema in the slice is registered with the
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249116924
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and
stop.
+total_consumed += n;
+break;
+}
+}
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
Review Comment:
Unit tests covering these corner cases were included in my latest push.
I also improved the `test_decode_stream_with_schema` test to check that the
prefix parsing works correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249115313
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
Review Comment:
@scovich I got this figured out. Currently we can only decode full single
object encoded records via `Decoder::decode`. In my very next PR I'll add a
partial record chunking feature to the `Decoder` which will resolve this.
I have it about 90% complete on my local machine, I just didn't want to grow
this PR any further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2249114536
##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +330,98 @@ impl ReaderBuilder {
/// - `batch_size` = 1024
/// - `strict_mode` = false
/// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fp` = None
+/// - `static_store_mode` = false
pub fn new() -> Self {
Self::default()
}
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) ->
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(),
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let field_builder = match reader_schema {
+Some(rs) if !compare_schemas(writer_schema, rs)? => {
+AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)
+}
+Some(rs) => AvroFieldBuilder::new(rs),
+None => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode);
+let root = field_builder.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
}
-fn build_impl(self, reader: &mut R) -> Result<(Header,
Decoder), ArrowError> {
-let header = read_header(reader)?;
-let record_decoder = if let Some(schema) = &self.schema {
-self.make_record_decoder(schema)?
-} else {
-let avro_schema: Option> = header
-.schema()
-.map_err(|e| ArrowError::ExternalError(Box::new(e)))?;
-let avro_schema = avro_schema.ok_or_else(|| {
-ArrowError::ParseError("No Avro schema present in file
header".to_string())
-})?;
-self.make_record_decoder(&avro_schema)?
-};
-let decoder = Decoder::new(record_decoder, self.batch_size);
-Ok((header, decoder))
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+Decoder {
+batch_size: self.batch_size,
+remaining_capacity: self.batch_size,
+active_fingerprint,
+active_decoder,
+cache: IndexMap::new(),
+max_cache_size: self.decoder_cache_size,
+reader_schema,
+utf8_view: self.utf8_view,
+writer_schema_store,
+strict_mode: self.strict_mode,
+pending_schema: None,
+}
+}
+
+fn make_decoder(&self, header: Option<&Header>) -> Result {
+match header {
+Some(hdr) => {
+let writer_schema = hdr
+.schema()
+.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
+.ok_or_else(|| {
+ArrowError::ParseError("No Avro schema present in file
header".into())
+})?;
+let record_decoder =
+self.make_record_decoder(&writer_schema,
self.reader_schema.as_ref())?;
+Ok(self.make_decoder_with_parts(record_decoder, None, None,
None))
+}
+None => {
+let reader_schema = self.reader_schema.clone().ok_or_else(|| {
+ArrowError::ParseError("Reader schema required for raw
Avro".into())
+})?;
+let (init_fingerprint, initial_decoder) =
+if let (Some(schema_store), Some(fingerprint)) =
+(&self.writer_schema_store, self.active_fingerprint)
+{
+// An initial fingerprint is provided, use it to look
up the first schema.
+let writer_schema =
schema_store.lookup(&fingerprint).ok_or_else(|| {
+ArrowError::ParseError(
+"Active fingerprint not found in schema
store".into(),
+)
+})?;
+let decoder =
+self.make_record_decoder(writer_schema,
Some(&reader_schema))?;
+(Some(fingerprint), decoder)
+} else {
+// No initial fingerprint; the first record must
contain one.
+// A decoder is created from the reader schema only.
+let decoder = self.make_record_decoder(&reader_schema,
None)?;
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248969563
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
Review Comment:
Yeah there's nothing about the wire format that prevents partial reads, the
only rule I came across is the minimum amount that can be read was one "datum"
at a time.
> Sure, but the Some(n) case does exactly the same thing (just for a
different reason). That's why I had suggested to combine them and just use a
comment to explain what's going on.
I'm going with this approach for this PR as it's logical. I may follow-up
later with another PR to revisit some of the complexity of the `Decoder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248969563
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
Review Comment:
Yeah there's nothing about the wire format that prevents partial reads, the
only rule I came across is the minimum amount that can be read was one "datum"
at a time, i.e. a row's field.
> Sure, but the Some(n) case does exactly the same thing (just for a
different reason). That's why I had suggested to combine them and just use a
comment to explain what's going on.
I'm going with this approach for this PR as it's logical. I may follow-up
later with another PR to revisit some of the complexity of the `Decoder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248969563
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
Review Comment:
Yeah there's nothing about the wire format that prevents partial reads, the
only rule I came across is the minimum amount that can be read was one "datum"
at a time, i.e. a row's field.
> Sure, but the Some(n) case does exactly the same thing (just for a
different reason). That's why I had suggested to combine them and just use a
comment to explain what's going on.
I'm going with this approach for this PR as it's logical. I may follow-up
later with another PR to revisit some of the complexity on the `Reader` and
`Decoder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248843889
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
Review Comment:
> Not sure why it should throw an error... is there some special property of
the wire format that prevents partial reads?
I'm researching this pretty deeply right now. I'll report back here soon.
I also think I found a huge simplification opportunity related to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248843889
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
Review Comment:
> Not sure why it should throw an error... is there some special property of
the wire format that prevents partial reads?
I'm researching this pretty deeply right now. I'll report back here soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248778865
##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +365,97 @@ impl ReaderBuilder {
/// - `batch_size` = 1024
/// - `strict_mode` = false
/// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
pub fn new() -> Self {
Self::default()
}
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) ->
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(),
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let field_builder = match reader_schema {
+Some(rs) if !compare_schemas(writer_schema, rs)? => {
+AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)
+}
+Some(rs) => AvroFieldBuilder::new(rs),
+None => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode);
+let root = field_builder.build()?;
Review Comment:
The above sounds good, but I was only commenting on where to put the
`with_xxx` calls above? That seems like a very localized change, orthogonal to
the conditional logic you mention?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248776365
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and
stop.
+total_consumed += n;
+break;
+}
+}
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint::<8>(&buf[SINGLE_OBJECT_MAGIC.len()..],
|bytes| {
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248768996
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
Review Comment:
> use `Some(0)` to cover a scenario where magic bytes were read, but there
wasn't enough remaining bytes in the buffer for the fingerprint. My thought was
to then return so the caller could add more bytes.
Sure, but the `Some(n)` case does exactly the same thing (just for a
different reason). That's why I had suggested to combine them and just use a
comment to explain what's going on.
> digging into the Java implementation's code however I noticed they throw
an error for this scenario
Not sure why it should throw an error... is there some special property of
the wire format that prevents partial reads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248718082
##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
self
}
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) ->
Self {
+self.reader_schema = Some(reader_schema);
self
}
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) ->
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded
data.
+///
+/// This is useful when the data stream does not begin with a schema
definition
+/// or fingerprint, allowing the decoder to start with a known schema from
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may
need to switch
+/// between different decoders. This cache avoids rebuilding them from
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|
Review Comment:
That's another 3am mistake on my part. I'll fix that as well.
The `reader_schema` isn't always required. It's technically the
`writer_schema` that is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248718082
##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
self
}
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) ->
Self {
+self.reader_schema = Some(reader_schema);
self
}
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) ->
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded
data.
+///
+/// This is useful when the data stream does not begin with a schema
definition
+/// or fingerprint, allowing the decoder to start with a known schema from
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may
need to switch
+/// between different decoders. This cache avoids rebuilding them from
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|
Review Comment:
That's another 3am mistake on my part. I'll fix that as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248715880
##
arrow-avro/src/schema.rs:
##
@@ -15,12 +15,27 @@
// specific language governing permissions and limitations
// under the License.
+use arrow_schema::ArrowError;
use serde::{Deserialize, Serialize};
+use serde_json::{json, Value};
+use std::collections::hash_map::Entry;
use std::collections::HashMap;
+use strum_macros::AsRefStr;
/// The metadata key used for storing the JSON encoded [`Schema`]
pub const SCHEMA_METADATA_KEY: &str = "avro.schema";
+/// The Avro single‑object encoding “magic” bytes (`0xC3 0x01`)
+pub const SINGLE_OBJECT_MAGIC: [u8; 2] = [0xC3, 0x01];
Review Comment:
@scovich From the research I was doing it seems what prevents false
positives is having the magic bytes **located at the start** of the message.
Here's a twist to consider though, the `Decoder::decoder` method was
developed to be generic enough to handle both:
1. OCF block encodings (blocks of Avro records without fingerprints, writer
schema is determined from OCF header which has it's own 4 magic byte sequence)
2. Message single object encodings (an encoding of a single Avro record
starting with the 2 byte magic + 8 byte fingerprint OR 1 byte magic + 4 byte id
fingerprint if Confluent variant).
I'm beginning to wonder if we'd benefit from having two decoder methods.
For instance both OCF and the message format support schema evolution via
writer and reader schemas. In message format the writer schema is determined
via a fingerprint located on **each Avro message** that may or may not change
between messages. Meanwhile in OCF format the writer schema is provided in the
header and will remain static for the full file which in turn is a sequence of
block encodings comprised of individual Avro records.
My original plan was to use the `Reader` for OCF and the stand-alone
`Decoder` for Messages (while being re-used in the `Reader`). I'm beginning to
question that approach.
A second alternative is to define a `DecodeMode` enum for the `Decoder` to
used for identifying which decode flow is required.
Either way I need to make some modifications based on this new research.
I'll have something pushed up by this evening. Ty for bringing this up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248632689
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and
stop.
+total_consumed += n;
+break;
+}
+}
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint::<8>(&buf[SINGLE_OBJECT_MAGIC.len()..],
|bytes| {
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+})?,
+};
+// Convert the inner result into a “bytes consumed” count.
+let consumed = match fingerprint_size {
+Some(n) => n + SINGLE_OBJECT_MAGIC.len(), // magic + fingerprint
+None => 0,// incomplete fingerprint
+};
+Ok(Some(consumed))
+}
+
+// Attempts to read and install a new fingerprint of `N` bytes.
+//
+// * Ok(None) – insufficient bytes (`buf.len() < `N`).
+// * Ok(Some(N)) – fingerprint consumed (always `N`).
+fn handle_fingerprint(
+&mut self,
+buf: &[u8],
+fingerprint_from: impl FnOnce([u8; N]) -> Fingerprint,
+) -> Result, ArrowError> {
+// Need enough bytes to get fingerprint (next N bytes)
+let Some(fingerprint_bytes) = buf.get(..N) else {
+return Ok(None); // Get more bytes
+};
+// SAFETY: length checked above.
+let new_fingerprint =
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248589128
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +158,178 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& data.len() >= SINGLE_OBJECT_MAGIC.len()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and
stop.
+total_consumed += n;
+break;
+}
+}
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
Review Comment:
I can add that in. I was planning to bring over much more extensive test
coverage in the follow-up PRs as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248586749
##
arrow-avro/src/schema.rs:
##
@@ -467,37 +465,50 @@ impl<'a> SchemaStore<'a> {
}
}
-#[inline]
fn quote(s: &str) -> Result {
serde_json::to_string(s)
.map_err(|e| ArrowError::ComputeError(format!("Failed to quote string:
{e}")))
}
-fn make_fullname(name: &str, namespace_attr: Option<&str>, enclosing_ns:
Option<&str>) -> String {
-match namespace_attr.or(enclosing_ns) {
-Some(ns) if !name.contains('.') => format!("{ns}.{name}"),
-_ => name.to_string(),
+// Avro names are defined by a `name` and an optional `namespace`.
+// The full name is composed of the namespace and the name, separated by a dot.
+//
+// Avro specification defines two ways to specify a full name:
+// 1. The `name` attribute contains the full name (e.g., "a.b.c.d").
+//In this case, the `namespace` attribute is ignored.
+// 2. The `name` attribute contains the simple name (e.g., "d") and the
+//`namespace` attribute contains the namespace (e.g., "a.b.c").
+//
+// Each part of the name must match the regex `^[A-Za-z_][A-Za-z0-9_]*$`.
+// Complex paths with quotes or backticks like `a."hi".b` are not supported.
+//
+// This function constructs the full name and extracts the namespace,
+// handling both ways of specifying the name. It prioritizes a namespace
+// defined within the `name` attribute itself, then the explicit
`namespace_attr`,
+// and finally the `enclosing_ns`.
+fn make_full_name(
+name: &str,
+namespace_attr: Option<&str>,
+enclosing_ns: Option<&str>,
+) -> Result<(String, Option), ArrowError> {
Review Comment:
I iterated on this function a few times last night and neglected to update
the signature. I'll get that handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248584861
##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +330,98 @@ impl ReaderBuilder {
/// - `batch_size` = 1024
/// - `strict_mode` = false
/// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fp` = None
+/// - `static_store_mode` = false
pub fn new() -> Self {
Self::default()
}
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) ->
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(),
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let field_builder = match reader_schema {
+Some(rs) if !compare_schemas(writer_schema, rs)? => {
+AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)
+}
+Some(rs) => AvroFieldBuilder::new(rs),
+None => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode);
+let root = field_builder.build()?;
+RecordDecoder::try_new_with_options(root.data_type(), self.utf8_view)
}
-fn build_impl(self, reader: &mut R) -> Result<(Header,
Decoder), ArrowError> {
-let header = read_header(reader)?;
-let record_decoder = if let Some(schema) = &self.schema {
-self.make_record_decoder(schema)?
-} else {
-let avro_schema: Option> = header
-.schema()
-.map_err(|e| ArrowError::ExternalError(Box::new(e)))?;
-let avro_schema = avro_schema.ok_or_else(|| {
-ArrowError::ParseError("No Avro schema present in file
header".to_string())
-})?;
-self.make_record_decoder(&avro_schema)?
-};
-let decoder = Decoder::new(record_decoder, self.batch_size);
-Ok((header, decoder))
+fn make_decoder_with_parts(
+&self,
+active_decoder: RecordDecoder,
+active_fingerprint: Option,
+reader_schema: Option>,
+writer_schema_store: Option>,
+) -> Decoder {
+Decoder {
+batch_size: self.batch_size,
+remaining_capacity: self.batch_size,
+active_fingerprint,
+active_decoder,
+cache: IndexMap::new(),
+max_cache_size: self.decoder_cache_size,
+reader_schema,
+utf8_view: self.utf8_view,
+writer_schema_store,
+strict_mode: self.strict_mode,
+pending_schema: None,
+}
+}
+
+fn make_decoder(&self, header: Option<&Header>) -> Result {
+match header {
+Some(hdr) => {
+let writer_schema = hdr
+.schema()
+.map_err(|e| ArrowError::ExternalError(Box::new(e)))?
+.ok_or_else(|| {
+ArrowError::ParseError("No Avro schema present in file
header".into())
+})?;
+let record_decoder =
+self.make_record_decoder(&writer_schema,
self.reader_schema.as_ref())?;
+Ok(self.make_decoder_with_parts(record_decoder, None, None,
None))
+}
+None => {
+let reader_schema = self.reader_schema.clone().ok_or_else(|| {
+ArrowError::ParseError("Reader schema required for raw
Avro".into())
+})?;
+let (init_fingerprint, initial_decoder) =
+if let (Some(schema_store), Some(fingerprint)) =
+(&self.writer_schema_store, self.active_fingerprint)
+{
+// An initial fingerprint is provided, use it to look
up the first schema.
+let writer_schema =
schema_store.lookup(&fingerprint).ok_or_else(|| {
+ArrowError::ParseError(
+"Active fingerprint not found in schema
store".into(),
+)
+})?;
+let decoder =
+self.make_record_decoder(writer_schema,
Some(&reader_schema))?;
+(Some(fingerprint), decoder)
+} else {
+// No initial fingerprint; the first record must
contain one.
+// A decoder is created from the reader schema only.
+let decoder = self.make_record_decoder(&reader_schema,
None)?;
+
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248580794
##
arrow-avro/src/reader/mod.rs:
##
@@ -216,34 +365,97 @@ impl ReaderBuilder {
/// - `batch_size` = 1024
/// - `strict_mode` = false
/// - `utf8_view` = false
-/// - `schema` = None
+/// - `reader_schema` = None
+/// - `writer_schema_store` = None
+/// - `active_fingerprint` = None
pub fn new() -> Self {
Self::default()
}
-fn make_record_decoder(&self, schema: &AvroSchema<'_>) ->
Result {
-let root_field = AvroFieldBuilder::new(schema)
-.with_utf8view(self.utf8_view)
-.with_strict_mode(self.strict_mode)
-.build()?;
-RecordDecoder::try_new_with_options(root_field.data_type(),
self.utf8_view)
+fn make_record_decoder<'a>(
+&self,
+writer_schema: &AvroSchema<'a>,
+reader_schema: Option<&AvroSchema<'a>>,
+) -> Result {
+let field_builder = match reader_schema {
+Some(rs) if !compare_schemas(writer_schema, rs)? => {
+AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)
+}
+Some(rs) => AvroFieldBuilder::new(rs),
+None => AvroFieldBuilder::new(writer_schema),
+}
+.with_utf8view(self.utf8_view)
+.with_strict_mode(self.strict_mode);
+let root = field_builder.build()?;
Review Comment:
@scovich There's some conditional logic that I needed to capture here.
1. If the `reader_schema` is provided and it is different from the
`writer_schema`, then we will have schema resolution. In that scenario we need
to pass in both the `writer_schema` and `reader_schema` into the
`AvroFieldBuilder`, i.e.
`AvroFieldBuilder::new(writer_schema).with_reader_schema(rs)` (I'll rename `rs`
that is pretty bad).
2. If there's a `reader_schema` and it's identical to the `writer_schema`
then we will have no schema resolution, it just becomes added overhead.
3. If there's no `reader_schema` then there's no schema resolution and we
use the `writer_schema`.
That being said I will clean this code up because it's neither clear nor
optimal. For example I could have just merged branches 2 and 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248559975
##
arrow-avro/src/schema.rs:
##
@@ -467,37 +465,50 @@ impl<'a> SchemaStore<'a> {
}
}
-#[inline]
fn quote(s: &str) -> Result {
serde_json::to_string(s)
.map_err(|e| ArrowError::ComputeError(format!("Failed to quote string:
{e}")))
}
-fn make_fullname(name: &str, namespace_attr: Option<&str>, enclosing_ns:
Option<&str>) -> String {
-match namespace_attr.or(enclosing_ns) {
-Some(ns) if !name.contains('.') => format!("{ns}.{name}"),
-_ => name.to_string(),
+// Avro names are defined by a `name` and an optional `namespace`.
+// The full name is composed of the namespace and the name, separated by a dot.
+//
+// Avro specification defines two ways to specify a full name:
+// 1. The `name` attribute contains the full name (e.g., "a.b.c.d").
+//In this case, the `namespace` attribute is ignored.
+// 2. The `name` attribute contains the simple name (e.g., "d") and the
+//`namespace` attribute contains the namespace (e.g., "a.b.c").
+//
+// Each part of the name must match the regex `^[A-Za-z_][A-Za-z0-9_]*$`.
+// Complex paths with quotes or backticks like `a."hi".b` are not supported.
+//
+// This function constructs the full name and extracts the namespace,
+// handling both ways of specifying the name. It prioritizes a namespace
+// defined within the `name` attribute itself, then the explicit
`namespace_attr`,
+// and finally the `enclosing_ns`.
Review Comment:
ty!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248553873
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
+Some(n) => {
+// Consumed a complete prefix (n > 0). Stage flush and
stop.
+total_consumed += n;
+break;
+}
}
-let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
-total_consumed += n;
-self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
+// Attempt to handle a single‑object‑encoding prefix at the current
position.
+//
+// * Ok(None) – buffer does not start with the prefix.
+// * Ok(Some(0)) – prefix detected, but the buffer is too short; caller
should await more bytes.
+// * Ok(Some(n)) – consumed `n > 0` bytes of a complete prefix (magic and
fingerprint).
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+// If there is no schema store, prefixes are unrecognized.
+if self.writer_schema_store.is_none() {
+return Ok(None); // Continue to decode the next record
+}
+// Need at least the magic bytes to decide (2 bytes).
+let Some(magic_bytes) = buf.get(..SINGLE_OBJECT_MAGIC.len()) else {
+return Ok(Some(0)); // Get more bytes
+};
+// Bail out early if the magic does not match.
+if magic_bytes != SINGLE_OBJECT_MAGIC {
+return Ok(None); // Continue to decode the next record
+}
+// Try to parse the fingerprint that follows the magic.
+let fingerprint_size = match hash_type {
+FingerprintAlgorithm::Rabin => self
+.handle_fingerprint::<8>(&buf[SINGLE_OBJECT_MAGIC.len()..],
|bytes| {
Review Comment:
Good call, ty for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248544320
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
Review Comment:
My intention was to use `Some(0)` to cover a scenario where magic bytes were
read, but there wasn't enough remaining bytes in the buffer for the
fingerprint. My thought was to then return so the caller could add more bytes.
I did just realize I'd need to change a mistake on L217 though for that.
After digging into the Java implementation's code however I noticed they
throw an error for this scenario:
https://github.com/apache/avro/blob/0606c33514e084cc0e131ab1f12a1a91e68e07c1/lang/java/avro/src/main/java/org/apache/avro/message/BinaryMessageDecoder.java#L150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248544320
##
arrow-avro/src/reader/mod.rs:
##
@@ -182,21 +174,130 @@ impl Decoder {
FingerprintAlgorithm::Rabin,
SchemaStore::fingerprint_algorithm,
);
+// The loop stops when the batch is full, a schema change is staged,
+// or handle_prefix indicates we need more bytes (Some(0)).
while total_consumed < data.len() && self.remaining_capacity > 0 {
-if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
-// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
-// a schema change is detected or there are insufficient bytes
to read the next prefix.
-// A schema change requires a new batch.
-total_consumed += prefix_bytes;
-break;
+match self.handle_prefix(&data[total_consumed..], hash_type)? {
+None => {
+// No prefix: decode one row.
+let n =
self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
+}
+Some(0) => {
+// Detected start of a prefix but need more bytes.
+break;
+}
Review Comment:
My intention was to use `Some(0)` to cover a scenario where magic bytes were
read, but there wasn't enough remaining bytes in the buffer for the
fingerprint. My thought was to then return so the caller could add more bytes.
After digging into the Java implementation's code however I noticed they
throw an error for this scenario:
https://github.com/apache/avro/blob/0606c33514e084cc0e131ab1f12a1a91e68e07c1/lang/java/avro/src/main/java/org/apache/avro/message/BinaryMessageDecoder.java#L150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248529510
##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +484,92 @@ impl ReaderBuilder {
self
}
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) ->
Self {
+self.reader_schema = Some(reader_schema);
self
}
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) ->
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded
data.
+///
+/// This is useful when the data stream does not begin with a schema
definition
+/// or fingerprint, allowing the decoder to start with a known schema from
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may
need to switch
+/// between different decoders. This cache avoids rebuilding them from
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+// Validate the builder configuration against this truth‑table
+//
+// | writer_schema_store | reader_schema | active_fingerprint | Result |
+// |-|---|||
+// | None| None--| None---| Err|
+// | None| None--| Some---| Err|
+// | None| Some--| None---| Ok-|
+// | None| Some--| Some---| Err|
+// | Some| None--| None---| Err|
+// | Some| None--| Some---| Err|
Review Comment:
Or, if it's correct, and `reader_schema` _is_ always required, then we can
eliminate a dimension from the truth table and greatly simplify the validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248525484
##
arrow-avro/src/codec.rs:
##
@@ -180,15 +182,15 @@ impl<'a> TryFrom<&Schema<'a>> for AvroField {
/// Builder for an [`AvroField`]
#[derive(Debug)]
pub struct AvroFieldBuilder<'a> {
-writer_schema: &'a Schema<'a>,
-reader_schema: Option<&'a Schema<'a>>,
+writer_schema: &'a AvroSchema<'a>,
+reader_schema: Option<&'a AvroSchema<'a>>,
Review Comment:
That sounds great. Reduces noise on this PR, and we can evaluate
`AvroSchema` vs. `Schema` on its own merits in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248512537
##
arrow-avro/src/codec.rs:
##
@@ -180,15 +182,15 @@ impl<'a> TryFrom<&Schema<'a>> for AvroField {
/// Builder for an [`AvroField`]
#[derive(Debug)]
pub struct AvroFieldBuilder<'a> {
-writer_schema: &'a Schema<'a>,
-reader_schema: Option<&'a Schema<'a>>,
+writer_schema: &'a AvroSchema<'a>,
+reader_schema: Option<&'a AvroSchema<'a>>,
Review Comment:
I originally thought the same and my personal preference was to refer to it
as `Schema`.
Then I thought it maybe easier for people who aren't familiar with this
crate to read the code if it was aliased as `AvroSchema`. Due to the size of
this PR though, I can just revert it back for now. It would probably be best to
tackle it explicitly in a dedicated PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248037550
##
arrow-avro/src/codec.rs:
##
@@ -180,15 +182,15 @@ impl<'a> TryFrom<&Schema<'a>> for AvroField {
/// Builder for an [`AvroField`]
#[derive(Debug)]
pub struct AvroFieldBuilder<'a> {
-writer_schema: &'a Schema<'a>,
-reader_schema: Option<&'a Schema<'a>>,
+writer_schema: &'a AvroSchema<'a>,
+reader_schema: Option<&'a AvroSchema<'a>>,
Review Comment:
This seems like a pretty sweeping change (affects lots of other code outside
this PR). Given that we're in the avro crate, maybe it's ok (or even better?)
to let `Schema` be the avro schema, and just use `ArrowSchema[Ref]` for the
exceptional cases where we're not dealing with avro schemas?
Then again, we're surrounded by other `AvroXxx` typenames here, so maybe the
change is just making things more uniform?
What are your thoughts?
##
arrow-avro/src/codec.rs:
##
@@ -474,7 +477,7 @@ impl<'a> Resolver<'a> {
}
}
-/// Parses a [`AvroDataType`] from the provided [`Schema`] and the given
`name` and `namespace`
+/// Parses a [`AvroDataType`] from the provided `Schema` and the given `name`
and `namespace`
Review Comment:
```suggestion
/// Parses a [`AvroDataType`] from the provided `schema` and the given
`name` and `namespace`
```
##
arrow-avro/src/schema.rs:
##
@@ -467,37 +465,50 @@ impl<'a> SchemaStore<'a> {
}
}
-#[inline]
fn quote(s: &str) -> Result {
serde_json::to_string(s)
.map_err(|e| ArrowError::ComputeError(format!("Failed to quote string:
{e}")))
}
-fn make_fullname(name: &str, namespace_attr: Option<&str>, enclosing_ns:
Option<&str>) -> String {
-match namespace_attr.or(enclosing_ns) {
-Some(ns) if !name.contains('.') => format!("{ns}.{name}"),
-_ => name.to_string(),
+// Avro names are defined by a `name` and an optional `namespace`.
+// The full name is composed of the namespace and the name, separated by a dot.
+//
+// Avro specification defines two ways to specify a full name:
+// 1. The `name` attribute contains the full name (e.g., "a.b.c.d").
+//In this case, the `namespace` attribute is ignored.
+// 2. The `name` attribute contains the simple name (e.g., "d") and the
+//`namespace` attribute contains the namespace (e.g., "a.b.c").
+//
+// Each part of the name must match the regex `^[A-Za-z_][A-Za-z0-9_]*$`.
+// Complex paths with quotes or backticks like `a."hi".b` are not supported.
+//
+// This function constructs the full name and extracts the namespace,
+// handling both ways of specifying the name. It prioritizes a namespace
+// defined within the `name` attribute itself, then the explicit
`namespace_attr`,
+// and finally the `enclosing_ns`.
Review Comment:
Really nice comment!
##
arrow-avro/src/schema.rs:
##
@@ -467,37 +465,50 @@ impl<'a> SchemaStore<'a> {
}
}
-#[inline]
fn quote(s: &str) -> Result {
serde_json::to_string(s)
.map_err(|e| ArrowError::ComputeError(format!("Failed to quote string:
{e}")))
}
-fn make_fullname(name: &str, namespace_attr: Option<&str>, enclosing_ns:
Option<&str>) -> String {
-match namespace_attr.or(enclosing_ns) {
-Some(ns) if !name.contains('.') => format!("{ns}.{name}"),
-_ => name.to_string(),
+// Avro names are defined by a `name` and an optional `namespace`.
+// The full name is composed of the namespace and the name, separated by a dot.
+//
+// Avro specification defines two ways to specify a full name:
+// 1. The `name` attribute contains the full name (e.g., "a.b.c.d").
+//In this case, the `namespace` attribute is ignored.
+// 2. The `name` attribute contains the simple name (e.g., "d") and the
+//`namespace` attribute contains the namespace (e.g., "a.b.c").
+//
+// Each part of the name must match the regex `^[A-Za-z_][A-Za-z0-9_]*$`.
+// Complex paths with quotes or backticks like `a."hi".b` are not supported.
+//
+// This function constructs the full name and extracts the namespace,
+// handling both ways of specifying the name. It prioritizes a namespace
+// defined within the `name` attribute itself, then the explicit
`namespace_attr`,
+// and finally the `enclosing_ns`.
+fn make_full_name(
+name: &str,
+namespace_attr: Option<&str>,
+enclosing_ns: Option<&str>,
+) -> Result<(String, Option), ArrowError> {
Review Comment:
I don't see any code in this function that returns Err?
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +276,389 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248380990
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,134 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is
None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
+// a schema change is detected or there are insufficient bytes
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
/// Produce a `RecordBatch` if at least one row is fully decoded, returning
/// `Ok(None)` if no new rows are available.
pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) =
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) =
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder,
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() ||
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let full_len = prefix_len(hash_type);
+if buf.len() < full_len {
+return Ok(Some(0)); // Not enough data to read the full fingerprint
+}
+let fp_bytes = &buf[2..full_len];
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {
+return Err(ArrowError::ParseError(format!(
+"Invalid Rabin fingerprint length, expected 8, got {}",
+fp_bytes.len()
+)));
+};
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+}
+};
Review Comment:
@scovich Actually the approach you suggested below was an extremely clean
and elegant way to implement this. I hadn't fully gone through it when I wrote
the prior comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248375384
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is
None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
+// a schema change is detected or there are insufficient bytes
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
/// Produce a `RecordBatch` if at least one row is fully decoded, returning
/// `Ok(None)` if no new rows are available.
pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) =
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) =
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder,
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() ||
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
Review Comment:
> I think my only opinion is: Not in this PR.
100% agree with you on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248022723
##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +450,70 @@ impl ReaderBuilder {
self
}
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) ->
Self {
+self.reader_schema = Some(reader_schema);
self
}
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) ->
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded
data.
+///
+/// This is useful when the data stream does not begin with a schema
definition
+/// or fingerprint, allowing the decoder to start with a known schema from
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may
need to switch
+/// between different decoders. This cache avoids rebuilding them from
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+fn validate(&self) -> Result<(), ArrowError> {
+match (
+self.writer_schema_store.as_ref(),
+self.reader_schema.as_ref(),
+self.active_fingerprint.as_ref(),
+) {
+(Some(_), None, _) => Err(ArrowError::ParseError(
+"Reader schema must be set when writer schema store is
provided".into(),
+)),
+(None, _, Some(_)) => Err(ArrowError::ParseError(
+"Active fingerprint requires a writer schema store".into(),
+)),
+_ => Ok(()),
+}
+}
+
/// Create a [`Reader`] from this builder and a `BufRead`
pub fn build(self, mut reader: R) -> Result,
ArrowError> {
-let (header, decoder) = self.build_impl(&mut reader)?;
+self.validate()?;
+let header = read_header(&mut reader)?;
+let decoder = self.make_decoder(Some(&header))?;
Ok(Reader {
reader,
header,
Review Comment:
Oh! I missed that `header` is actually used twice and so needs to be a named
value. Sorry about that. So really the suggestion should have been to inline
the `decoder` value that is only used once. But we're definitely in nitpick
territory at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2248016140
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is
None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
+// a schema change is detected or there are insufficient bytes
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
/// Produce a `RecordBatch` if at least one row is fully decoded, returning
/// `Ok(None)` if no new rows are available.
pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) =
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) =
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder,
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() ||
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
Review Comment:
> > From what I can tell, this whole operation should actually be infallible
>
> There's actually one scenario this operation is fallible imo, and that's
if the `Fingerprint` isn't found in the `SchemaStore`, i.e.
>
> ```rust
> let writer_schema = store.lookup(&new_fingerprint).ok_or_else(|| {
> ArrowError::ParseError(format!("Unknown fingerprint:
{new_fingerprint:?}"))
> })?;
> ```
Ah, I somehow totally missed that. You are of course correct.
> I've been toying with the idea of defining a `SchemaStore` trait. This
would enable users to create their own `SchemaStore` implementations... I'm
curious what you think of the idea?
I think my only opinion is: Not in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service,
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#issuecomment-3143694026 @scovich I got those latest changes pushed up. Let me know what you think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2247182724
##
arrow-avro/src/reader/mod.rs:
##
@@ -272,17 +450,70 @@ impl ReaderBuilder {
self
}
-/// Sets the Avro schema.
+/// Sets the Avro reader schema.
///
/// If a schema is not provided, the schema will be read from the Avro
file header.
-pub fn with_schema(mut self, schema: AvroSchema<'static>) -> Self {
-self.schema = Some(schema);
+pub fn with_reader_schema(mut self, reader_schema: AvroSchema<'static>) ->
Self {
+self.reader_schema = Some(reader_schema);
self
}
+/// Sets the `SchemaStore` used for resolving writer schemas.
+///
+/// This is necessary when decoding single-object encoded data that
identifies
+/// schemas by a fingerprint. The store allows the decoder to look up the
+/// full writer schema from a fingerprint embedded in the data.
+///
+/// Defaults to `None`.
+pub fn with_writer_schema_store(mut self, store: SchemaStore<'static>) ->
Self {
+self.writer_schema_store = Some(store);
+self
+}
+
+/// Sets the initial schema fingerprint for decoding single-object encoded
data.
+///
+/// This is useful when the data stream does not begin with a schema
definition
+/// or fingerprint, allowing the decoder to start with a known schema from
the
+/// `SchemaStore`.
+///
+/// Defaults to `None`.
+pub fn with_active_fingerprint(mut self, fp: Fingerprint) -> Self {
+self.active_fingerprint = Some(fp);
+self
+}
+
+/// Set the maximum number of decoders to cache.
+///
+/// When dealing with Avro files that contain multiple schemas, we may
need to switch
+/// between different decoders. This cache avoids rebuilding them from
scratch every time.
+///
+/// Defaults to `20`.
+pub fn with_max_decoder_cache_size(mut self, n: usize) -> Self {
+self.decoder_cache_size = n;
+self
+}
+
+fn validate(&self) -> Result<(), ArrowError> {
+match (
+self.writer_schema_store.as_ref(),
+self.reader_schema.as_ref(),
+self.active_fingerprint.as_ref(),
+) {
+(Some(_), None, _) => Err(ArrowError::ParseError(
+"Reader schema must be set when writer schema store is
provided".into(),
+)),
+(None, _, Some(_)) => Err(ArrowError::ParseError(
+"Active fingerprint requires a writer schema store".into(),
+)),
+_ => Ok(()),
+}
+}
+
/// Create a [`Reader`] from this builder and a `BufRead`
pub fn build(self, mut reader: R) -> Result,
ArrowError> {
-let (header, decoder) = self.build_impl(&mut reader)?;
+self.validate()?;
+let header = read_header(&mut reader)?;
+let decoder = self.make_decoder(Some(&header))?;
Ok(Reader {
reader,
header,
Review Comment:
@scovich If you're alright with it, I'd like to follow up on this one in a
future PR. There's some trickiness to this one (Header has a `Schema` with a
lifetime in it, I can't push it down too far or it turns into a rabbit hole).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2246957540
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is
None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
+// a schema change is detected or there are insufficient bytes
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
/// Produce a `RecordBatch` if at least one row is fully decoded, returning
/// `Ok(None)` if no new rows are available.
pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) =
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) =
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder,
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() ||
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
Review Comment:
@scovich I'm finishing these changes now and getting close to pushing up.
> From what I can tell, this whole operation should actually be infallible
There's actually one scenario this operation is fallible imo, and that's if
the `Fingerprint` isn't found in the `SchemaStore`, i.e.
```rust
let writer_schema = store.lookup(&new_fingerprint).ok_or_else(|| {
ArrowError::ParseError(format!("Unknown fingerprint:
{new_fingerprint:?}"))
})?;
```
The soonest we could get the new `Fingerprint` is in either
`handle_fingerprint` or `handle_prefix` (not as clean imo). A `SchemaStore`
missing an otherwise valid `Fingerprint` should propagate an error back to the
caller. This will allow the caller to then recreate the `Decoder` with an
updated `SchemaStore`, etc.
As a side note, I've been toying with the idea of defining a `SchemaStore`
trait. This would enable users to create their own `SchemaStore`
implementations, including ones which could
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2245962100
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+///
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+///
+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its
operations.
+/// This ensures consistency when generating fingerprints and looking up
schemas.
+/// All schemas registered will have their fingerprint computed with this
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string
slices
+/// contained within the stored [`Schema`] objects. This means the
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2245932943
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+///
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+///
+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its
operations.
+/// This ensures consistency when generating fingerprints and looking up
schemas.
+/// All schemas registered will have their fingerprint computed with this
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string
slices
+/// contained within the stored [`Schema`] objects. This means the
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2245920424
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,134 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is
None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
+// a schema change is detected or there are insufficient bytes
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
/// Produce a `RecordBatch` if at least one row is fully decoded, returning
/// `Ok(None)` if no new rows are available.
pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) =
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) =
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder,
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() ||
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let full_len = prefix_len(hash_type);
+if buf.len() < full_len {
+return Ok(Some(0)); // Not enough data to read the full fingerprint
+}
+let fp_bytes = &buf[2..full_len];
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {
+return Err(ArrowError::ParseError(format!(
+"Invalid Rabin fingerprint length, expected 8, got {}",
+fp_bytes.len()
+)));
+};
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+}
+};
Review Comment:
My main concern is the combination of split logic and magic constants
produces a large bug surface. Hard to get the code correct, hard for others to
convince themselves it's correct, and easy for future changes to break things
if one part of the code ever gets out of sync.
It's less clear to me whether that needs
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
scovich commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2245909595
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+///
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+///
+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its
operations.
+/// This ensures consistency when generating fingerprints and looking up
schemas.
+/// All schemas registered will have their fingerprint computed with this
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string
slices
+/// contained within the stored [`Schema`] objects. This means the
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+/// Each
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244570973
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is
None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
+// a schema change is detected or there are insufficient bytes
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
/// Produce a `RecordBatch` if at least one row is fully decoded, returning
/// `Ok(None)` if no new rows are available.
pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) =
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) =
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder,
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() ||
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
Review Comment:
> suppose there really is a schema change, but the buffer has only 1 byte
left. Then by returning None we would tell our caller that no schema change is
coming and the caller would attempt to consume the byte as part of a (possibly
single-byte) record instead.
Ah, I think I see what you mean. Also I really like your suggested approach.
I'll pick this back up in the morning and finish incorporating these
recommendations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244531096
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,130 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is
None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
+// a schema change is detected or there are insufficient bytes
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
/// Produce a `RecordBatch` if at least one row is fully decoded, returning
/// `Ok(None)` if no new rows are available.
pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) =
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) =
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder,
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() ||
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let fp_bytes = &buf[2..]; // safe thanks to the `starts_with` check
above
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {
Review Comment:
This is extremely useful information. I'm going through this now.
> but I don't know if that "extract an array from a slice at a given offset"
pattern arises often enough in this crate to be worth creating a similar helper
here?
I haven't come across that yet in this crate, but if I do again I may as
well make a helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244485742
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+///
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+///
+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its
operations.
+/// This ensures consistency when generating fingerprints and looking up
schemas.
+/// All schemas registered will have their fingerprint computed with this
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string
slices
+/// contained within the stored [`Schema`] objects. This means the
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+///
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244483679
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+///
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+///
+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its
operations.
+/// This ensures consistency when generating fingerprints and looking up
schemas.
+/// All schemas registered will have their fingerprint computed with this
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string
slices
+/// contained within the stored [`Schema`] objects. This means the
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+///
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244482440
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+///
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+///
+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its
operations.
+/// This ensures consistency when generating fingerprints and looking up
schemas.
+/// All schemas registered will have their fingerprint computed with this
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string
slices
+/// contained within the stored [`Schema`] objects. This means the
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+///
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244481107
##
arrow-avro/src/reader/mod.rs:
##
@@ -520,20 +498,13 @@ impl ReaderBuilder {
self.writer_schema_store.as_ref(),
self.reader_schema.as_ref(),
self.active_fingerprint.as_ref(),
-self.static_store_mode,
) {
-(Some(_), None, _, _) => Err(ArrowError::ParseError(
+(Some(_), None, _) => Err(ArrowError::ParseError(
"Reader schema must be set when writer schema store is
provided".into(),
)),
-(None, _, Some(_), _) => Err(ArrowError::ParseError(
+(None, _, Some(_)) => Err(ArrowError::ParseError(
"Active fingerprint requires a writer schema store".into(),
)),
-(None, _, _, true) => Err(ArrowError::ParseError(
-"static_store_mode=true requires a writer schema store".into(),
-)),
-(Some(_), _, None, true) => Err(ArrowError::ParseError(
-"static_store_mode=true requires an active fingerprint".into(),
-)),
_ => Ok(()),
Review Comment:
I pushed up those improvements and included a truth table in the comments.
Really good idea.
Also I removed the `self.validate()?;` call from the `ReaderBuilder::build`
method since it's only really needed in `ReaderBuilder::build_decoder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244476447
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,134 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is
None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
+// a schema change is detected or there are insufficient bytes
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
/// Produce a `RecordBatch` if at least one row is fully decoded, returning
/// `Ok(None)` if no new rows are available.
pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) =
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) =
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder,
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() ||
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let full_len = prefix_len(hash_type);
+if buf.len() < full_len {
+return Ok(Some(0)); // Not enough data to read the full fingerprint
+}
+let fp_bytes = &buf[2..full_len];
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {
+return Err(ArrowError::ParseError(format!(
+"Invalid Rabin fingerprint length, expected 8, got {}",
+fp_bytes.len()
+)));
+};
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+}
+};
+// If the fingerprint indicates a schema change, prepare to switch
decoders.
+if self.active_fingerprint != Some(new_fp) {
+self.prepare_schema_switch(new_fp)?;
+// If there are already decoded rows, we must flush them first.
+// Forcing the batch to be full ensures `flush` is called
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244475539
##
arrow-avro/src/reader/mod.rs:
##
@@ -154,39 +167,134 @@ impl Decoder {
///
/// Returns the number of bytes consumed.
pub fn decode(&mut self, data: &[u8]) -> Result {
+if self.active_fingerprint.is_none()
+&& self.writer_schema_store.is_some()
+&& !data.starts_with(&SINGLE_OBJECT_MAGIC)
+{
+return Err(ArrowError::ParseError(
+"Expected single‑object encoding fingerprint prefix for first
message \
+ (writer_schema_store is set but active_fingerprint is
None)"
+.into(),
+));
+}
let mut total_consumed = 0usize;
-while total_consumed < data.len() && self.decoded_rows <
self.batch_size {
-let consumed = self.record_decoder.decode(&data[total_consumed..],
1)?;
-// A successful call to record_decoder.decode means one row was
decoded.
-// If `consumed` is 0 on a non-empty buffer, it implies a valid
zero-byte record.
-// We increment `decoded_rows` to mark progress and avoid an
infinite loop.
-// We add `consumed` (which can be 0) to `total_consumed`.
-total_consumed += consumed;
-self.decoded_rows += 1;
+let hash_type = self.writer_schema_store.as_ref().map_or(
+FingerprintAlgorithm::Rabin,
+SchemaStore::fingerprint_algorithm,
+);
+while total_consumed < data.len() && self.remaining_capacity > 0 {
+if let Some(prefix_bytes) =
self.handle_prefix(&data[total_consumed..], hash_type)? {
+// A batch is complete when its `remaining_capacity` is 0. It
may be completed early if
+// a schema change is detected or there are insufficient bytes
to read the next prefix.
+// A schema change requires a new batch.
+total_consumed += prefix_bytes;
+break;
+}
+let n = self.active_decoder.decode(&data[total_consumed..], 1)?;
+total_consumed += n;
+self.remaining_capacity -= 1;
}
Ok(total_consumed)
}
/// Produce a `RecordBatch` if at least one row is fully decoded, returning
/// `Ok(None)` if no new rows are available.
pub fn flush(&mut self) -> Result, ArrowError> {
-if self.decoded_rows == 0 {
-Ok(None)
-} else {
-let batch = self.record_decoder.flush()?;
-self.decoded_rows = 0;
-Ok(Some(batch))
+if self.remaining_capacity == self.batch_size {
+return Ok(None);
+}
+let batch = self.active_decoder.flush()?;
+self.remaining_capacity = self.batch_size;
+// Apply a pending schema switch if one is staged
+if let Some((new_fingerprint, new_decoder)) =
self.pending_schema.take() {
+// Cache the old decoder before replacing it
+if let Some(old_fingerprint) =
self.active_fingerprint.replace(new_fingerprint) {
+let old_decoder = std::mem::replace(&mut self.active_decoder,
new_decoder);
+self.cache.shift_remove(&old_fingerprint);
+self.cache.insert(old_fingerprint, old_decoder);
+if self.cache.len() > self.max_cache_size {
+self.cache.shift_remove_index(0);
+}
+} else {
+self.active_decoder = new_decoder;
+}
+}
+Ok(Some(batch))
+}
+
+#[inline]
+fn handle_prefix(
+&mut self,
+buf: &[u8],
+hash_type: FingerprintAlgorithm,
+) -> Result, ArrowError> {
+if self.writer_schema_store.is_none() ||
!buf.starts_with(&SINGLE_OBJECT_MAGIC) {
+return Ok(None);
+}
+let full_len = prefix_len(hash_type);
+if buf.len() < full_len {
+return Ok(Some(0)); // Not enough data to read the full fingerprint
+}
+let fp_bytes = &buf[2..full_len];
+let new_fp = match hash_type {
+FingerprintAlgorithm::Rabin => {
+let Ok(bytes) = <[u8; 8]>::try_from(fp_bytes) else {
+return Err(ArrowError::ParseError(format!(
+"Invalid Rabin fingerprint length, expected 8, got {}",
+fp_bytes.len()
+)));
+};
+Fingerprint::Rabin(u64::from_le_bytes(bytes))
+}
+};
Review Comment:
@scovich
I just saw your other newest comments and I'm about to address the `<[u8;
8]>::try_from(fingerprint_bytes)` issue you raised.
I left `prefix_len` in and moved `let fingerprint_bytes = &buf[2..];` to
inside of the `FingerprintAlgorithm::Rabin` branch because I planned to fast
follow on getting
Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]
jecsand838 commented on code in PR #8006:
URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r226409
##
arrow-avro/src/schema.rs:
##
@@ -260,13 +274,369 @@ pub struct Fixed<'a> {
pub attributes: Attributes<'a>,
}
+/// Supported fingerprint algorithms for Avro schema identification.
+/// Currently only `Rabin` is supported, `SHA256` and `MD5` support will come
in a future update
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum FingerprintAlgorithm {
+/// 64‑bit CRC‑64‑AVRO Rabin fingerprint.
+Rabin,
+}
+
+/// A schema fingerprint in one of the supported formats.
+///
+/// This is used as the key inside `SchemaStore` `HashMap`. Each `SchemaStore`
+/// instance always stores only one variant, matching its configured
+/// `FingerprintAlgorithm`, but the enum makes the API uniform.
+/// Currently only `Rabin` is supported
+///
+///
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
+pub enum Fingerprint {
+/// A 64-bit Rabin fingerprint.
+Rabin(u64),
+}
+
+/// Allow easy extraction of the algorithm used to create a fingerprint.
+impl From<&Fingerprint> for FingerprintAlgorithm {
+#[inline]
+fn from(fp: &Fingerprint) -> Self {
+match fp {
+Fingerprint::Rabin(_) => FingerprintAlgorithm::Rabin,
+}
+}
+}
+
+/// Generates a fingerprint for the given `Schema` using the specified
`FingerprintAlgorithm`.
+#[inline]
+pub(crate) fn generate_fingerprint(
+schema: &Schema,
+hash_type: FingerprintAlgorithm,
+) -> Result {
+let canonical = generate_canonical_form(schema).map_err(|e| {
+ArrowError::ComputeError(format!("Failed to generate canonical form
for schema: {e}"))
+})?;
+match hash_type {
+FingerprintAlgorithm::Rabin => {
+Ok(Fingerprint::Rabin(compute_fingerprint_rabin(&canonical)))
+}
+}
+}
+
+/// Generates the 64-bit Rabin fingerprint for the given `Schema`.
+///
+/// The fingerprint is computed from the canonical form of the schema.
+/// This is also known as `CRC-64-AVRO`.
+///
+/// # Returns
+/// A `Fingerprint::Rabin` variant containing the 64-bit fingerprint.
+#[inline]
+pub fn generate_fingerprint_rabin(schema: &Schema) -> Result {
+generate_fingerprint(schema, FingerprintAlgorithm::Rabin)
+}
+
+/// Generates the Parsed Canonical Form for the given [`Schema`].
+///
+/// The canonical form is a standardized JSON representation of the schema,
+/// primarily used for generating a schema fingerprint for equality checking.
+///
+/// This form strips attributes that do not affect the schema's identity,
+/// such as `doc` fields, `aliases`, and any properties not defined in the
+/// Avro specification.
+///
+///
+#[inline]
+pub fn generate_canonical_form(schema: &Schema) -> Result {
+build_canonical(schema, None)
+}
+
+/// An in-memory cache of Avro schemas, indexed by their fingerprint.
+///
+/// `SchemaStore` provides a mechanism to store and retrieve Avro schemas
efficiently.
+/// Each schema is associated with a unique [`Fingerprint`], which is
generated based
+/// on the schema's canonical form and a specific hashing algorithm.
+///
+/// A `SchemaStore` instance is configured to use a single
[`FingerprintAlgorithm`] such as Rabin,
+/// MD5 (not yet supported), or SHA256 (not yet supported) for all its
operations.
+/// This ensures consistency when generating fingerprints and looking up
schemas.
+/// All schemas registered will have their fingerprint computed with this
algorithm, and
+/// lookups must use a matching fingerprint.
+///
+/// The lifetime parameter `'a` corresponds to the lifetime of the string
slices
+/// contained within the stored [`Schema`] objects. This means the
`SchemaStore`
+/// cannot outlive the data referenced by the schemas it contains.
+///
+/// # Examples
+///
+/// ```no_run
+/// // Create a new store with the default Rabin fingerprinting.
+/// use arrow_avro::schema::{PrimitiveType, Schema, SchemaStore, TypeName};
+///
+/// let mut store = SchemaStore::new();
+/// let schema = Schema::TypeName(TypeName::Primitive(PrimitiveType::String));
+/// // Register the schema to get its fingerprint.
+/// let fingerprint = store.register(schema.clone()).unwrap();
+/// // Use the fingerprint to look up the schema.
+/// let retrieved_schema = store.lookup(&fingerprint).cloned();
+/// assert_eq!(retrieved_schema, Some(schema));
+/// ```
+#[derive(Debug, Clone)]
+pub struct SchemaStore<'a> {
+/// The hashing algorithm used for generating fingerprints.
+fingerprint_algorithm: FingerprintAlgorithm,
+/// A map from a schema's fingerprint to the schema itself.
+schemas: HashMap>,
+}
+
+impl<'a> TryFrom<&'a [Schema<'a>]> for SchemaStore<'a> {
+type Error = ArrowError;
+
+/// Creates a `SchemaStore` from a slice of schemas.
+///
