comphead commented on code in PR #7082:
URL: https://github.com/apache/datafusion-comet/pull/7082#discussion_r4338820980
##########
native/core/src/parquet/parquet_exec.rs:
##########
@@ -2111,4 +2118,131 @@ mod tests {
);
}
}
+
+ /// A parquet file of one `a` column holding `values`.
+ fn parquet_of(values: Vec<i32>) -> Bytes {
+ let schema = Arc::new(Schema::new(vec![Field::new("a",
DataType::Int32, false)]));
+ let batch = RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![Arc::new(Int32Array::from(values))],
+ )
+ .unwrap();
+ let mut buffer = Vec::new();
+ let mut writer = ArrowWriter::try_new(&mut buffer, schema,
None).unwrap();
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+ Bytes::from(buffer)
+ }
+
+ /// Two stores that hold a file at the same path with the same size but
different contents
+ /// must not hand each other's footer to a scan, whether the metadata
cache is shared by the
+ /// tasks of a stage or private to one task. The path DataFusion keys the
cache by has no
+ /// bucket, so the cache a scan uses is keyed by the scan's object store
as well.
+ #[tokio::test]
+ async fn equal_paths_in_two_stores_do_not_share_footers() {
+ use crate::parquet::shared_metadata_cache::{SharedMetadataCaches,
StageKey};
+ use datafusion::execution::cache::cache_manager::CacheManagerConfig;
+ use datafusion::execution::runtime_env::RuntimeEnvBuilder;
+
+ let location =
object_store::path::Path::from("table/part-00000.parquet");
+ let contents = [vec![1, 2, 3], vec![101, 102, 103]];
+ let files: Vec<Bytes> =
contents.iter().cloned().map(parquet_of).collect();
+ assert_eq!(
+ files[0].len(),
+ files[1].len(),
+ "the files must have equal sizes for the cache to accept one for
the other"
+ );
+ let size = files[0].len() as u64;
+ let urls = [
+ "s3+comet-0000000000000001-native://bucket-a",
+ "s3+comet-0000000000000001-native://bucket-b",
+ ];
+
+ for shared in [true, false] {
Review Comment:
With `shared == false` each scan builds its own `RuntimeEnv` inside the
inner loop, so the two scans never see the same cache and this half passes
without `scope_to_store`. With sharing off, the collision the view guards
against needs two scans in one plan, so one `RuntimeEnv`. Could the unshared
iteration build one `RuntimeEnv` for both scans? Then the "or private to one
task" part of the doc comment is tested too. The `registry` and `stage_cache`
built for that iteration are also unused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
