JingsongLi commented on code in PR #1014:
URL: https://github.com/apache/paimon-rust/pull/1014#discussion_r4479106694
##########
crates/paimon/src/io/file_io.rs:
##########
@@ -494,13 +579,135 @@ impl FileIO {
}
}
- pub(crate) fn create_static(&self, path: &str) -> crate::Result<(Operator,
String)> {
+ /// A view of this FileIO that sends every request to origin, as copies
require.
+ pub(crate) fn origin_only(&self) -> Self {
+ let mut file_io = self.clone();
+ file_io.origin_only = true;
+ file_io.schema_file_prefixes = None;
+ file_io
+ }
+
+ /// A view of this FileIO whose io-cache routing also knows the prefixes
`schemas` loads,
+ /// on the first request that may use io-cache.
+ pub(crate) fn with_schema_file_prefixes(&self, schemas:
SchemaFilePrefixes) -> Self {
+ let mut file_io = self.clone();
+ if !self.origin_only {
+ file_io.schema_file_prefixes = Some(Arc::new(schemas));
+ }
+ file_io
+ }
+
+ /// Whether routing `path` needs the persisted prefixes, which are not
loaded yet.
+ fn awaits_schema_file_prefixes(&self, path: &str) -> bool {
+ self.schema_file_prefixes
+ .as_ref()
+ .is_some_and(|schemas| schemas.loaded.get().is_none())
+ && has_other_data_file_prefix(path, &self.file_prefixes)
+ }
+
+ /// Prefixes for io-cache routing, with the persisted ones loaded so far.
+ fn loaded_file_prefixes(&self) -> Cow<'_, [String]> {
+ match self
+ .schema_file_prefixes
+ .as_ref()
+ .and_then(|s| s.loaded.get())
+ {
+ Some(persisted) => with_persisted_prefixes(&self.file_prefixes,
persisted),
+ None => Cow::Borrowed(&self.file_prefixes),
+ }
+ }
+
+ /// Prefixes for io-cache routing of `path`, loading the persisted ones
first when only they
+ /// may make it routable.
+ async fn routing_file_prefixes(&self, path: &str) -> crate::Result<Cow<'_,
[String]>> {
+ if self.awaits_schema_file_prefixes(path) &&
self.routes_to_io_cache().await? {
+ if let Some(schemas) = &self.schema_file_prefixes {
+ schemas.load().await;
+ }
+ }
+ Ok(self.loaded_file_prefixes())
+ }
+
+ /// A view of this FileIO whose io-cache routing also knows data files
named with `prefixes`.
+ pub(crate) fn with_file_prefixes<'a>(
+ &self,
+ prefixes: impl IntoIterator<Item = &'a str>,
+ ) -> Self {
+ let mut file_prefixes = self.file_prefixes.to_vec();
+ for prefix in prefixes {
+ if !prefix.trim().is_empty() && !file_prefixes.iter().any(|known|
known == prefix) {
+ file_prefixes.push(prefix.to_string());
+ }
+ }
+ let mut file_io = self.clone();
+ file_io.file_prefixes = file_prefixes.into();
+ file_io
+ }
+
+ /// A view of this FileIO serving the table at `location`.
+ pub(crate) fn with_table_location(&self, location: &str) -> Self {
+ let mut file_io = self.clone();
+ file_io.table_location = Some(location.trim_end_matches('/').into());
+ file_io
+ }
+
+ /// The view to read URIs taken from row data through. A FileIO that
serves no table cannot
+ /// tell who owns them, and other writers may replace what they own, so it
uses origin.
+ pub(crate) fn for_row_data(&self) -> Self {
+ if self.table_location.is_some() {
+ self.clone()
+ } else {
+ self.origin_only()
+ }
+ }
+
+ /// Whether requests on `path` must use origin: a table's view routes only
its own files.
+ fn origin_only_for(&self, path: &str) -> bool {
+ self.origin_only
+ || self.table_location.as_deref().is_some_and(|location| {
+ !path
+ .strip_prefix(location)
Review Comment:
[P1] Keep normalized external OSS paths on origin
The table boundary is checked on the raw URI. A descriptor path under
`oss://bkt/db.db/t/../outside/` passes this prefix check for table
`oss://bkt/db.db/t`, but the HTTP request normalizes the dot segment and
addresses `bkt/db.db/outside/`.
I reproduced this with two independent local OSS endpoints. The plain origin
view read the current external object. The table's row-data view sent a GET for
that same external key to the target and returned its stale bytes, with no
request to origin. This bypasses the rule that externally owned BLOB sources
stay on origin.
Please base cache eligibility on the effective OSS bucket and object path,
or route ambiguous dot-segment paths to origin before file classification.
Cover an outside object and a stale target copy in the row-data routing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]