karellincoln opened a new issue, #5162:
URL: https://github.com/apache/arrow-rs/issues/5162
**Describe the bug**
when I used below WriterProperties to close column Statistics
```
let w_opt = WriterProperties::builder()
.set_compression(Compression::SNAPPY)
.set_max_row_group_size(1024 * 1024)
.set_data_page_row_count_limit(1024)
.set_statistics_enabled(EnabledStatistics::None)
.set_column_statistics_enabled("taskId".into(), EnabledStatistics::Page) //
the only column I want to build ColumnIndex, in Page level
.build();
```
**actual behavior**
I find all column which type is string build ColumnIndex, but I set it did
not build.
```
let mut file = tokio::fs::File::open(file_path).await.unwrap();
let metadata =
ArrowReaderMetadata::load_async(&mut file,
ArrowReaderOptions::new().with_page_index(true))
.await
.unwrap();
use this get metadata like below:
}), NONE, BYTE_ARRAY(NativeIndex { indexes: [PageIndex {
min: Some(ByteArray { data: "ok"
}), max: Some(ByteArray { data: "ok"
}), null_count: Some(0)
}, PageIndex { min: Some(ByteArray { data: "ok"
}), max: Some(ByteArray { data: "ok"
}), null_count: Some(0)
}
], boundary_order: BoundaryOrder(0)
}), BYTE_ARRAY(NativeIndex { indexes: [PageIndex { min:
Some(ByteArray { data: "ok"
}), max: Some(ByteArray { data: "ok"
}), null_count: Some(0)
}, PageIndex { min: Some(ByteArray { data: "ok"
}), max: Some(ByteArray { data: "ok"
}), null_count: Some(0)
}
], boundary_order: BoundaryOrder(0)
}), BYTE_ARRAY(NativeIndex { indexes: [PageIndex { min:
Some(ByteArray { data: "60b6a5dd99c75"
}), max: Some(ByteArray { data: "60b6a5dd99c75"
}), null_count: Some(0)
}, PageIndex { min: Some(ByteArray { data:
"60b6a5dd99c75"
}), max: Some(ByteArray { data: "60b6a5dd99c75"
}), null_count: Some(0)
}
], boundary_order: BoundaryOrder(0)
```
**Expected behavior**
I found
[ByteArrayEncoder](https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/arrow_writer/byte_array.rs#L501)
not use EnabledStatistics value to control that,
like
[ColumnValueEncoderImpl](https://github.com/apache/arrow-rs/blob/master/parquet/src/column/writer/encoder.rs#L139)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
