JingsongLi commented on code in PR #1014:
URL: https://github.com/apache/paimon-rust/pull/1014#discussion_r4261249465
##########
crates/paimon/src/io/file_io.rs:
##########
@@ -344,13 +404,34 @@ impl FileIO {
Ok((op, relative_path, cache_path))
}
+ async fn create_routed(&self, path: &str) ->
crate::Result<(RoutedOperator, String)> {
+ match &self.backend {
+ FileIOBackend::Provider(provider) => {
+ let (op, relative_path, _) = resolve_provider_routed(
+ provider.as_ref(),
+ path,
+ self.origin_only,
+ &self.file_prefixes,
+ )
+ .await?;
+ Ok((op, relative_path))
+ }
+ FileIOBackend::Storage(_) => self.create_routed_static(path,
&self.file_prefixes),
+ }
+ }
+
fn file_source(&self, path: &str) -> crate::Result<FileSource> {
match &self.backend {
- FileIOBackend::Provider(provider) =>
Ok(FileSource::Provider(provider.clone())),
+ FileIOBackend::Provider(provider) => Ok(FileSource::Provider {
+ provider: provider.clone(),
+ origin_only: self.origin_only,
+ file_prefixes: self.file_prefixes.clone(),
+ }),
FileIOBackend::Storage(storage) => {
- let (op, relative_path) = self.create_static(path)?;
+ let (op, relative_path) = self.create_routed_static(path,
&self.file_prefixes)?;
Review Comment:
[P2] Read independently owned BLOB ingestion sources from origin
The new routing also reaches BlobFormatWriter's external descriptor source
reads through UriInput, using only the source basename to infer immutability. I
reproduced with policy=read / whitelist=data: external
oss://bkt/import/data-<uuid>-1.png contains NEW on origin and retained OLD on
target; pass BlobDescriptor(uri,0,3) as a PK BLOB input and perform normal
TableWrite / prepare / commit. The source GET reaches target only, and the
newly committed managed pack permanently contains OLD. Reading that pack
through an origin-only table still returns OLD, so this is incorrect persisted
payload. External descriptor URIs are supported inputs (see
primary_key_blob_copies_input_descriptor_into_managed_pack) and have no
immutable-source contract; the Format/Object Table guards do not protect this
source owned outside the normal table. Please use origin for independently
owned descriptor-source ingestion in both BLOB writer paths, or establish
trusted source ownership before routing. Prese
rve routing for the new immutable outputs. An origin-only input FileIO control
copies NEW and passes the actual write/read regression.
##########
crates/paimon/src/table/mod.rs:
##########
@@ -294,6 +296,16 @@ pub struct Table {
applied_dynamic_option_keys: HashSet<String>,
}
+// Format Table files may be replaced in place under the same name, so they
never use io-cache;
+// other tables tell io-cache routing the prefixes their data files are named
with.
+fn table_file_io(file_io: &FileIO, schema: &TableSchema) -> FileIO {
+ let options = CoreOptions::new(schema.options());
+ if options.is_format_table() {
+ return file_io.origin_only();
+ }
+ file_io.with_file_prefixes([options.data_file_prefix(),
options.changelog_file_prefix()])
Review Comment:
[P2] Preserve retained data-file routing across table reloads
Only the latest loaded schema's prefixes reach this view. I reproduced with
a real FileSystemCatalog and HTTP OSS endpoints under meta,read,write /
whitelist=data: create data-file.prefix=older-, write/commit/read through the
target, ALTER that option to newer-, then load, append, and load again. The
fresh scan returns both rows correctly, but the old committed
older-<uuid>-0.parquet GET goes only to origin; target sees no request for it.
These files remain immutable Paimon data, so a supported prefix change/restart
silently loses the configured cache/accelerator traffic allocation for existing
data. The same risk exists for dynamic prefixes whose write options are not
persisted. The current test retains one FileIO clone that remembers old
prefixes, which does not exercise fresh loads. Please recover trusted routing
context for committed files across reloads, rather than only the latest
in-memory schema prefixes, and add the persisted ALTER/reload regression.
Retaining older- in t
he reloaded context makes the actual endpoint assertion pass. No row-data loss
was observed in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
