Rich-T-kid commented on code in PR #11144:
URL: https://github.com/apache/arrow-rs/pull/11144#discussion_r4564861640


##########
parquet/src/arrow/arrow_reader/tests/schema.rs:
##########
@@ -852,30 +924,92 @@ fn test_plain_binary_dictionary_rejects_invalid_utf8() {
     assert!(error.to_string().contains("non UTF-8 data"), "{error}");
 }
 
+/// A supplied schema that reads a `Binary` column as a string type must still
+/// validate UTF-8.
 #[test]
-#[should_panic(expected = "Invalid UTF8 sequence at")]
 fn test_read_non_utf8_binary_as_utf8() {
-    let file = write_parquet_from_iter(vec![(
+    // Invalid UTF-8 after an empty value and a null, so that when the writer
+    // abandons the dictionary after the first value, the invalid values are in
+    // the PLAIN pages that follow.
+    let batch = RecordBatch::try_from_iter(vec![(
         "non_utf8_binary",
-        Arc::new(BinaryArray::from(vec![
-            b"\xDE\x00\xFF".as_ref(),
-            b"\xDE\x01\xAA".as_ref(),
-            b"\xDE\x02\xFF".as_ref(),
+        Arc::new(BinaryArray::from_opt_vec(vec![
+            Some(b"".as_ref()),
+            None,
+            Some(b"\xDE\x00\xFF".as_ref()),

Review Comment:
   Could we add `[b"\xC2", b"\xA2"]` as two separate values here? Each value is 
invalid UTF-8, but together they form a valid sequence. With those values, 
`DELTA_LENGTH_BYTE_ARRAY` read as `Utf8View` returns an array that fails 
`validate_full`, while `Utf8` panics; the `Utf8View` `DELTA_BYTE_ARRAY` path 
also returns an invalid array. These paths need the per-value boundary check 
used by `OffsetBuffer::try_push`.



##########
parquet/src/arrow/record_reader/mod.rs:
##########
@@ -217,6 +222,11 @@ where
         self.padding_threshold = Some(threshold);
     }
 
+    /// Validate decoded values as UTF-8 even if the column is not annotated 
as a string.
+    pub fn set_validate_utf8(&mut self, validate_utf8: bool) {

Review Comment:
   Could we add a two-row-group test for this? A small reproducer is 
`[b"valid", b"\xFF"]` written with `set_max_row_group_row_count(Some(1))`, then 
read as `Utf8` with batch size 1. The first `next()` should succeed and the 
second should return the UTF-8 error. That exercises the new decoder created by 
the second `set_page_reader` call.



##########
parquet/src/arrow/arrow_reader/tests/schema.rs:
##########
@@ -550,34 +552,98 @@ fn test_schema_error_bad_nullability() {
     )
 }
 
+/// The ways a byte array column is written, with the encodings each produces:
+/// dictionary pages (the default), dictionary pages that fall back to PLAIN
+/// part way through the column chunk, and PLAIN, DELTA_LENGTH_BYTE_ARRAY and
+/// DELTA_BYTE_ARRAY pages without a dictionary.
+fn byte_array_encodings() -> Vec<(Vec<Encoding>, WriterProperties)> {
+    let without_dictionary = |encoding| {
+        WriterProperties::builder()
+            .set_dictionary_enabled(false)
+            .set_encoding(encoding)
+            .build()
+    };
+    // A one-byte limit makes the writer abandon the dictionary after the
+    // first value.
+    let dictionary_fallback = WriterProperties::builder()
+        .set_dictionary_page_size_limit(1)
+        .set_write_batch_size(1)
+        .build();
+    vec![
+        (vec![Encoding::RLE_DICTIONARY], WriterProperties::default()),
+        (
+            vec![Encoding::RLE_DICTIONARY, Encoding::PLAIN],
+            dictionary_fallback,
+        ),
+        (vec![Encoding::PLAIN], without_dictionary(Encoding::PLAIN)),
+        (
+            vec![Encoding::DELTA_LENGTH_BYTE_ARRAY],
+            without_dictionary(Encoding::DELTA_LENGTH_BYTE_ARRAY),
+        ),
+        (
+            vec![Encoding::DELTA_BYTE_ARRAY],
+            without_dictionary(Encoding::DELTA_BYTE_ARRAY),
+        ),
+    ]
+}
+
+/// Writes `batch` with `props`, checking that every column used `encodings`.
+fn write_with_encodings(
+    batch: &RecordBatch,
+    encodings: &[Encoding],
+    props: WriterProperties,
+) -> File {
+    let file = tempfile().unwrap();
+    let mut writer =
+        ArrowWriter::try_new(file.try_clone().unwrap(), batch.schema(), 
Some(props)).unwrap();
+    writer.write(batch).unwrap();
+    writer.close().unwrap();
+
+    let builder = 
ParquetRecordBatchReaderBuilder::try_new(file.try_clone().unwrap()).unwrap();
+    for column in builder.metadata().row_group(0).columns() {
+        let written: Vec<_> = column.encodings().collect();

Review Comment:
   Could this inspect the data-page encodings instead of `column.encodings()`? 
To reproduce, write the same batch once with default properties and once with 
`set_dictionary_page_size_limit(1)`, then print 
`column.encodings().collect::<Vec<_>>()`. Both report `[PLAIN, RLE, 
RLE_DICTIONARY]` because the dictionary page itself is PLAIN, even though only 
the second file has a PLAIN data page.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to