[
https://issues.apache.org/jira/browse/AVRO-3240?focusedWorklogId=670519&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-670519
]
ASF GitHub Bot logged work on AVRO-3240:
----------------------------------------
Author: ASF GitHub Bot
Created on: 27/Oct/21 07:34
Start Date: 27/Oct/21 07:34
Worklog Time Spent: 10m
Work Description: martin-g commented on a change in pull request #1379:
URL: https://github.com/apache/avro/pull/1379#discussion_r737182833
##########
File path: lang/rust/src/reader.rs
##########
@@ -341,6 +343,40 @@ mod tests {
6u8, 102u8, 111u8, 111u8, 84u8, 6u8, 98u8, 97u8, 114u8, 94u8, 61u8,
54u8, 221u8, 190u8,
207u8, 108u8, 180u8, 158u8, 57u8, 114u8, 40u8, 173u8, 199u8, 228u8,
239u8,
];
+ const TEST_RECORD_SCHEMA: &str = r#"
Review comment:
The name of the schema and the struct below are too generic.
Let's add `_3240` (the JIRA number) to them
##########
File path: lang/rust/src/decode.rs
##########
@@ -23,7 +23,7 @@ use crate::{
util::{safe_len, zag_i32, zag_i64},
AvroResult, Error,
};
-use std::{collections::HashMap, convert::TryFrom, io::Read, str::FromStr};
+use std::{collections::HashMap, convert::TryFrom, io::{ErrorKind,Read},
str::FromStr};
Review comment:
Add an empty space between `ErrorKind` and `Read`.
I expect clippy to complain once the checks are enabled.
##########
File path: lang/rust/src/reader.rs
##########
@@ -341,6 +343,40 @@ mod tests {
6u8, 102u8, 111u8, 111u8, 84u8, 6u8, 98u8, 97u8, 114u8, 94u8, 61u8,
54u8, 221u8, 190u8,
207u8, 108u8, 180u8, 158u8, 57u8, 114u8, 40u8, 173u8, 199u8, 228u8,
239u8,
];
+ const TEST_RECORD_SCHEMA: &str = r#"
+ {
+ "type": "record",
+ "name": "test",
+ "fields": [
+ {
+ "name": "a",
+ "type": "long",
+ "default": 42
+ },
+ {
+ "name": "b",
+ "type": "string"
+ },
+ {
+ "name": "a_nullable_array",
+ "type": ["null", {"type": "array", "items": {"type": "string"}}],
+ "default": null
+ },
+ {
+ "name": "a_nullable_boolean",
+ "type": ["null", {"type": "boolean"}],
+ "default": null
+ }
+ ]
+ }
+ "#;
+ #[derive(Default, Debug, Deserialize, PartialEq)]
Review comment:
Is `PartialEq` really needed ?
##########
File path: lang/rust/src/reader.rs
##########
@@ -358,6 +394,23 @@ mod tests {
);
}
+ #[test]
+ fn test_from_avro_datum_with_union_to_struct() {
+ let schema = Schema::parse_str(TEST_RECORD_SCHEMA).unwrap();
+ let mut encoded: &'static [u8] = &[54, 6, 102, 111, 111];
+
+ let avro_datum = from_avro_datum(&schema, &mut encoded, None).unwrap();
+ let test_record: TestRecord = match &avro_datum {
+ Value::Record(_) => from_value::<TestRecord>(&avro_datum).unwrap(),
+ _ => panic!("could not map avro data to struct"),
+ };
+
+ assert_eq!(
+ test_record.a_nullable_array,
+ None
+ );
Review comment:
Please add assertions for all record fields.
##########
File path: lang/rust/src/reader.rs
##########
@@ -358,6 +394,23 @@ mod tests {
);
}
+ #[test]
+ fn test_from_avro_datum_with_union_to_struct() {
+ let schema = Schema::parse_str(TEST_RECORD_SCHEMA).unwrap();
+ let mut encoded: &'static [u8] = &[54, 6, 102, 111, 111];
+
+ let avro_datum = from_avro_datum(&schema, &mut encoded, None).unwrap();
+ let test_record: TestRecord = match &avro_datum {
+ Value::Record(_) => from_value::<TestRecord>(&avro_datum).unwrap(),
+ _ => panic!("could not map avro data to struct"),
Review comment:
Let's use a proper name instead of `_`, e.g. `unexpected` and print it
in the panic message. It will help for debugging regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 670519)
Time Spent: 1h 50m (was: 1h 40m)
> Schema deserialization is not backwards compatible
> --------------------------------------------------
>
> Key: AVRO-3240
> URL: https://issues.apache.org/jira/browse/AVRO-3240
> Project: Apache Avro
> Issue Type: Bug
> Components: rust
> Reporter: Ultrabug
> Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Hello,
>
> When providing your own schema to `from_avro_datum`, the deserialization is
> not backward compatible with messages containing a previous schema even if
> the schemas are created to be backward compatible.
> This is due to the `decode_variable` function in `utils` returning
> `Error::ReadVariableIntegerBytes` when the reader object is smaller than
> expected when reading a message with previous schema version (which is indeed
> smaller) and trying to decode it with a backward compatible schema.
> I have fixed the issue locally with the following patch which I will submit
> as a PR
>
> ```
> diff --git a/lang/rust/src/util.rs b/lang/rust/src/util.rs
> index f9daf285..3538399e 100644
> --- a/lang/rust/src/util.rs
> +++ b/lang/rust/src/util.rs
> @@ -96,7 +96,7 @@ fn decode_variable<R: Read>(reader: &mut R) ->
> AvroResult<u64> {
> }
> reader
> .read_exact(&mut buf[..])
> - .map_err(Error::ReadVariableIntegerBytes)?;
> + .or(Ok(()))?; // return nullable to support avro schemas backward
> compatibility
> i |= (u64::from(buf[0] & 0x7F)) << (j * 7);
> if (buf[0] >> 7) == 0 {
> break;
> ```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
