JingsongLi commented on code in PR #764:
URL: https://github.com/apache/paimon-rust/pull/764#discussion_r3893255395
##########
crates/paimon/src/file_index/file_index_format.rs:
##########
@@ -329,12 +334,74 @@ fn calculate_head_length(
Ok(total_length)
}
+fn resolve_index_data_type(
+ fields_by_name: &HashMap<&str, &DataType>,
+ column_name: &str,
+) -> crate::Result<DataType> {
+ let nested_start = column_name.find('[').filter(|_|
column_name.ends_with(']'));
+ let field_name = nested_start
+ .map(|index| &column_name[..index])
+ .unwrap_or(column_name);
+ let data_type = fields_by_name.get(field_name).copied().ok_or_else(|| {
+ format_invalid(format!(
+ "Column '{field_name}' for file index '{column_name}' was not
found in schema"
+ ))
+ })?;
+
+ match (nested_start, data_type) {
+ (Some(_), DataType::Map(map_type)) =>
Ok(map_type.value_type().clone()),
+ (Some(_), data_type) => Err(format_invalid(format!(
+ "Nested file index '{column_name}' requires Map column
'{field_name}', but found {data_type:?}"
+ ))),
+ (None, data_type) => Ok(data_type.clone()),
+ }
+}
+
pub struct FileIndex {
reader: Box<dyn FileRead>,
header: HashMap<String, HashMap<String, IndexInfo>>,
}
impl FileIndex {
+ /// Constructs the concrete readers described by this outer-format file.
+ #[allow(dead_code)]
+ pub(crate) async fn create_index_readers(
+ &self,
+ fields: &[DataField],
+ ) -> crate::Result<HashMap<String, Vec<Box<dyn FileIndexReader>>>> {
+ let fields_by_name = fields
+ .iter()
+ .map(|field| (field.name(), field.data_type()))
+ .collect::<HashMap<&str, &DataType>>();
+ let mut readers = HashMap::with_capacity(self.header.len());
+
+ for (column_name, index_info) in &self.header {
Review Comment:
[P2] Please avoid eagerly materializing every index payload here.
`create_index_readers` walks the complete header, serially reads each full
range, and returns concrete readers that retain those `Bytes`, even when the
query predicate uses only one column. A legal payload is already 100 MB in
`test_large_data_set`, so concurrent scans can multiply unnecessary remote
reads and pinned memory before evaluation. The Java `FileIndexPredicate`
collects predicate field names and calls `readColumnIndex` only for those
columns. Could this API accept the required columns (or keep `FileRead` plus
ranges and load lazily), with a counting-reader test proving unrelated ranges
are not fetched?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
