sundapeng commented on code in PR #1014:
URL: https://github.com/apache/paimon-rust/pull/1014#discussion_r4368192540
##########
crates/paimon/src/io/file_io.rs:
##########
@@ -344,13 +404,34 @@ impl FileIO {
Ok((op, relative_path, cache_path))
}
+ async fn create_routed(&self, path: &str) ->
crate::Result<(RoutedOperator, String)> {
+ match &self.backend {
+ FileIOBackend::Provider(provider) => {
+ let (op, relative_path, _) = resolve_provider_routed(
+ provider.as_ref(),
+ path,
+ self.origin_only,
+ &self.file_prefixes,
+ )
+ .await?;
+ Ok((op, relative_path))
+ }
+ FileIOBackend::Storage(_) => self.create_routed_static(path,
&self.file_prefixes),
+ }
+ }
+
fn file_source(&self, path: &str) -> crate::Result<FileSource> {
match &self.backend {
- FileIOBackend::Provider(provider) =>
Ok(FileSource::Provider(provider.clone())),
+ FileIOBackend::Provider(provider) => Ok(FileSource::Provider {
+ provider: provider.clone(),
+ origin_only: self.origin_only,
+ file_prefixes: self.file_prefixes.clone(),
+ }),
FileIOBackend::Storage(storage) => {
- let (op, relative_path) = self.create_static(path)?;
+ let (op, relative_path) = self.create_routed_static(path,
&self.file_prefixes)?;
Review Comment:
Thanks, 869f7be reads descriptor sources through the origin-only FileIO view
in `BlobFormatWriter`, which both the managed-pack and the `.blob` writer paths
share; the packs and `.blob` files the table writes keep their routing.
`blob_descriptor_sources_are_read_from_origin` mirrors your probe:
meta,read,write / whitelist=data, origin holds NEW and the target retains OLD
at `oss://bkt/import/<table>/data-<uuid>-1.png`, then `BlobDescriptor(uri, 0,
3)` goes through TableWrite / prepare_commit / TableCommit for a PK BLOB table
and an append BLOB table. The source GET reaches origin only, the `.blob` PUT
still goes to the target, and both the routed table and an origin-only catalog
read NEW (both read OLD without the fix).
d686bfa applies the same rule to the other places that read objects a
descriptor points to: `blob-descriptor-field` columns on read, and ROW sidecar
writes with `data-evolution.row-sidecar.enabled`.
`blob_descriptor_fields_read_their_sources_from_origin` covers both, with and
without the sidecar.
##########
crates/paimon/src/table/mod.rs:
##########
@@ -294,6 +296,16 @@ pub struct Table {
applied_dynamic_option_keys: HashSet<String>,
}
+// Format Table files may be replaced in place under the same name, so they
never use io-cache;
+// other tables tell io-cache routing the prefixes their data files are named
with.
+fn table_file_io(file_io: &FileIO, schema: &TableSchema) -> FileIO {
+ let options = CoreOptions::new(schema.options());
+ if options.is_format_table() {
+ return file_io.origin_only();
+ }
+ file_io.with_file_prefixes([options.data_file_prefix(),
options.changelog_file_prefix()])
Review Comment:
Thanks, e871d80 gives each table's FileIO view a shared, run-once loader of
the `data-file.prefix` / `changelog-file.prefix` of all persisted schemas. It
is attached in `Table::new` / `from_resolved_schema`, so every catalog and the
bindings get it, and copies share it (a copy on another branch gets one for
that branch's schemas). It runs lazily, on the first request whose file is
named like a data file with none of the current prefixes, and only once routing
is known to be on; it reads `schema/` through an origin-only view, so loading a
table does no extra I/O, and a table without routing never lists its schemas.
`altered_data_file_prefix_keeps_routing_after_reloads` mirrors your probe:
real FileSystemCatalog, meta,read,write / whitelist=data, create with
`data-file.prefix=older-`, write/commit, ALTER to `newer-`, `load_table`,
append, then `get_table` from a new catalog; the scan returns both rows and the
GETs of the `older-<uuid>-0.parquet` and `newer-` files reach the target, not
origin. `rest_tables_keep_routing_for_prefixes_of_older_schemas` covers the
REST load path with only the newer schema in the response: building the table
sends no OSS request, and reading the older file reaches the target without
another token request. Dynamic prefixes that were never persisted still go to
origin, which only loses cache benefit.
The branch now also merges main; `io-cache.whitelist` keeps the documented
types after main split data files into dedicated and structured data (11c8613):
`data` covers both, and other values are ignored as before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
