lxy-9602 commented on code in PR #447:
URL: https://github.com/apache/paimon-cpp/pull/447#discussion_r4374372099


##########
docs/source/user_guide/format_table.rst:
##########


Review Comment:
   I’m a bit confused—now that expect for a `data table`, shouldn’t all the 
other table types use the `table->NewScan()`-style interfaces?



##########
include/paimon/read_context.h:
##########
@@ -246,7 +246,7 @@ class PAIMON_EXPORT ReadContextBuilder {
     /// `SetTableSchema()`, `WithFileSystem()`, 
`WithFileSystemSchemeToIdentifierMap()` and a
     /// branch are refused here rather than ignored.
     ///
-    /// @param table The format table to read, as `Catalog::GetFormatTable()` 
hands it back.
+    /// @param table The format table to read, as `Catalog::GetTable()` hands 
it back.
     explicit ReadContextBuilder(const std::shared_ptr<FormatTable>& table);
 

Review Comment:
   I’m thinking about removing `FormatTable` from `xxxContextBuilder`. Wouldn’t 
it be better if all special tables also go through `table` (i.e., create the 
behavior via `table`)? Only the `data table` might be allowed to be created 
directly via the context.
   
   Otherwise, the current design feels a bit too coupled between `context` and 
`table`.



##########
src/paimon/core/catalog/file_system_catalog_test.cpp:
##########
@@ -223,10 +230,15 @@ TEST(FileSystemCatalogTest, 
TestOptionsSystemTableCatalog) {
     ASSERT_FALSE(loaded_schema->field(1)->nullable());
 
     ASSERT_OK_AND_ASSIGN(auto system_table, 
catalog.GetTable(options_identifier));
-    ASSERT_EQ(system_table->Name(), "tbl1$options");
-    // A system table reads the files of the data table it belongs to, so the 
table opened
-    // for it carries that table's root path.
+    // A system table comes back as the `SystemTable` it is: it answers for 
itself with its own
+    // read-only schema and the root path of the data table it reads.
+    ASSERT_TRUE(std::dynamic_pointer_cast<SystemTable>(system_table) != 
nullptr);
+    ASSERT_TRUE(std::dynamic_pointer_cast<FormatTable>(system_table) == 
nullptr);
+    ASSERT_EQ("options", system_table->Name());
+    ASSERT_EQ("options", system_table->FullName());

Review Comment:
   For system tables, should the full name and name be `tbl1$options`?
   



##########
src/paimon/core/table/system/system_table.h:
##########
@@ -49,19 +50,48 @@ struct SystemTablePath {
     bool is_global = false;
 };
 
-/// Base interface for table-scoped system tables such as `T$options` and 
`T$snapshots`.
+/// Base class for table-scoped system tables such as `T$options` and 
`T$snapshots`.
 ///
-/// Implementations expose a read-only schema and create their own scan/read 
objects.
-class SystemTable : public std::enable_shared_from_this<SystemTable> {
+/// A system table is the read-only view a `T$system` identifier names: it 
exposes a schema of
+/// its own and serves scans and reads from the data table it belongs to, 
through the same
+/// `Table` interface as every other kind of table. `Name()` - which each kind 
of system table
+/// answers for itself - is its full name as well: the identifier it is opened 
by belongs to the
+/// data table underneath and would name that one.
+class SystemTable : public Table, public 
std::enable_shared_from_this<SystemTable> {

Review Comment:
   Why is this designed this way here? In Java it just returns `"tbl$options"`.



##########
include/paimon/table/source/table_scan.h:
##########
@@ -36,7 +36,8 @@ class ScanContext;
 /// A scanner interface for reading table's meta and create a plan.
 class PAIMON_EXPORT TableScan {
  public:
-    /// Create an instance of `TableScan`.
+    /// Create an instance of `TableScan`: the table the context names is 
loaded first, then the

Review Comment:
   "the table the context names" -> "the table created by the context"?



##########
src/paimon/core/table/format/format_table.cpp:
##########
@@ -177,35 +193,196 @@ Result<std::shared_ptr<FormatTable>> FormatTable::Create(
         core_options.FormatTablePartitionOnlyValueInPath(), 
location_carries_paimon_metadata));
 }
 
-Result<std::shared_ptr<FormatTable>> FormatTable::Copy(
-    const std::shared_ptr<FormatTable>& table,
-    const std::map<std::string, std::string>& dynamic_options) {
-    if (table == nullptr) {
-        return Status::Invalid("format table requires a table to copy");
-    }
+Result<std::shared_ptr<FormatTable>> FormatTable::CopySelf(
+    const std::map<std::string, std::string>& dynamic_options) const {
     if (dynamic_options.empty()) {
-        return table;
+        return SharedSelf();
     }
     // The options this table already carries go back in as dynamic ones, so 
an option an earlier
     // call set is not lost to the schema's own value underneath it.
-    std::map<std::string, std::string> options = table->options_;
+    std::map<std::string, std::string> options = options_;
     for (const auto& [key, value] : dynamic_options) {
         options[key] = value;
     }
-    return Create(table->file_system_, table->location_, table->identifier_, 
table->schema_,
-                  table->location_carries_paimon_metadata_, options);
+    return Create(file_system_, root_path_, identifier_,
+                  std::static_pointer_cast<DataSchema>(schema_), 
location_carries_paimon_metadata_,
+                  options);
 }
 
-const std::vector<std::string>& FormatTable::PartitionKeys() const {
-    return schema_->PartitionKeys();
+Result<std::shared_ptr<Table>> FormatTable::Copy(
+    const std::map<std::string, std::string>& dynamic_options) const {
+    PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<FormatTable> copied, 
CopySelf(dynamic_options));
+    return std::shared_ptr<Table>(std::move(copied));
 }
 
-std::string FormatTable::FullName() const {
-    return identifier_.GetFullName();
+const std::vector<std::string>& FormatTable::PartitionKeys() const {
+    return LatestSchema()->PartitionKeys();
 }
 
 Result<std::unique_ptr<::ArrowSchema>> FormatTable::GetArrowSchema() const {
     return schema_->GetArrowSchema();
 }
 
+Result<std::unique_ptr<TableScan>> FormatTable::NewScan(
+    std::unique_ptr<ScanContext> context) const {
+    std::shared_ptr<ScanContext> shared_context = std::move(context);
+    if (shared_context == nullptr) {
+        return Status::Invalid("scan context is null pointer");
+    }
+    if (shared_context->GetMemoryPool() == nullptr) {
+        return Status::Invalid("memory pool is null pointer");
+    }
+    if (shared_context->GetExecutor() == nullptr) {
+        return Status::Invalid("executor is null pointer");
+    }
+    PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<FormatTable> table,
+                           CopySelf(shared_context->GetOptions()));
+    // Anything a `ScanContext` carries that a format table cannot honour is 
refused rather than
+    // silently dropped.
+    if (shared_context->IsStreamingMode()) {
+        return Status::NotImplemented(
+            "a format table has no snapshots, so there is nothing for a 
streaming scan to follow");
+    }
+    if (shared_context->GetRealtimeContext() != nullptr) {
+        return Status::NotImplemented(
+            "a format table has no real-time store to union with what is on 
disk");
+    }
+    if (shared_context->GetGlobalIndexResult() != nullptr) {
+        return Status::NotImplemented("a format table carries no global 
index");
+    }
+    std::map<std::string, std::string> partition_filter;
+    std::shared_ptr<ScanFilter> filters = shared_context->GetScanFilters();
+    if (filters != nullptr) {
+        if (filters->GetPredicate() != nullptr) {
+            return Status::NotImplemented(
+                "a format table scan does not take a predicate: its files 
carry no statistics to "
+                "skip by, and pruning by the partition columns a predicate 
names is not "
+                "implemented yet; give the partition values as a partition 
filter instead");
+        }
+        if (filters->GetBucketFilter()) {
+            return Status::NotImplemented("a format table has no buckets to 
filter by");
+        }
+        const std::vector<std::map<std::string, std::string>>& 
partition_filters =
+            filters->GetPartitionFilters();
+        if (partition_filters.size() > 1) {
+            return Status::NotImplemented("a format table scan takes at most 
one partition filter");
+        }
+        if (partition_filters.size() == 1) {
+            partition_filter = partition_filters.front();
+        }
+    }
+    PAIMON_ASSIGN_OR_RAISE(
+        std::unique_ptr<FormatTableScan> scan,
+        FormatTableScan::Create(table, partition_filter, 
shared_context->GetLimit()));
+    return std::unique_ptr<TableScan>(std::move(scan));
+}
+
+Result<std::unique_ptr<TableRead>> FormatTable::NewRead(
+    std::unique_ptr<ReadContext> context) const {
+    std::shared_ptr<ReadContext> read_context = std::move(context);
+    if (read_context == nullptr) {
+        return Status::Invalid("read context is null pointer");
+    }
+    if (read_context->GetMemoryPool() == nullptr) {
+        return Status::Invalid("memory pool is null pointer");
+    }
+    if (read_context->GetExecutor() == nullptr) {
+        return Status::Invalid("executor is null pointer");
+    }
+    if (read_context->HasVectorOrFullTextSearch()) {
+        return Status::NotImplemented(
+            "VectorSearch or FullTextSearch is not supported for format 
tables");
+    }
+    if (PredicateUtils::ContainsFullTextMatch(read_context->GetPredicate())) {
+        return Status::NotImplemented("FullTextMatch predicate execution is 
not implemented yet");
+    }
+    PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<FormatTable> table,
+                           CopySelf(read_context->GetOptions()));
+    PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<FormatTableRead> read,
+                           FormatTableRead::Create(table, read_context));
+    return std::unique_ptr<TableRead>(std::move(read));
+}
+
+Result<std::unique_ptr<FileStoreWrite>> FormatTable::NewWrite(
+    std::unique_ptr<WriteContext> context) const {
+    if (context == nullptr) {
+        return Status::Invalid("write context is null pointer");
+    }
+    if (context->GetMemoryPool() == nullptr) {
+        return Status::Invalid("memory pool is null pointer");
+    }
+    if (context->GetExecutor() == nullptr) {
+        return Status::Invalid("executor is null pointer");
+    }
+    PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<FormatTable> table, 
CopySelf(context->GetOptions()));
+    if (context->GetCatalog() != nullptr) {
+        return Status::Invalid(
+            "WithCatalog() requires a native table; use 
WriteContextBuilder(FormatTable) for a "
+            "format table");
+    }
+    if (context->IsStreamingMode()) {
+        return Status::NotImplemented(
+            "a format table has no snapshots, so there is nothing a streaming 
write could "
+            "commit against");
+    }

Review Comment:
   Are these actually “NotImplemented”, or are some of them “Invalid”?
   



##########
src/paimon/core/table/system/global_system_tables.cpp:
##########
@@ -300,7 +300,13 @@ Result<std::shared_ptr<SystemTable>> 
GlobalSystemTableLoader::Load(
                 return Status::NotImplemented("global system table requires 
catalog context: ",
                                               table_name);
             }
-            return entry.factory(context);
+            PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<SystemTable> system_table,
+                                   entry.factory(context));
+            // The table is handed out through the `Table` interface, which 
serves the schema
+            // the base holds, so its own read-only schema is adopted before 
it leaves the
+            // loader.
+            PAIMON_RETURN_NOT_OK(system_table->AdoptOwnSchema());
+            return system_table;

Review Comment:
   This design is a bit coupled—similar to an init function.
   Do we need `SystemTable::GetSchema` to override with its own logic?



##########
test/inte/realtime_write_inte_test.cpp:
##########
@@ -559,6 +560,41 @@ class RealtimeWriteInteTest : public ::testing::Test {
         return FileStoreWrite::Create(std::move(context));
     }
 
+    // A writer that means to change an option the table was created with 
writes through a copy
+    // carrying it, which the context then agrees with.
+    Result<std::shared_ptr<Table>> CopiedTable(
+        const std::map<std::string, std::string>& overrides) const {
+        PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<Catalog> catalog, 
Catalog::Create(dir_->Str(), {}));
+        PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<Table> table,
+                               catalog->GetTable(Identifier("foo", "bar")));
+        return table->Copy(overrides);
+    }
+
+    // Evolves the table schema to the current options with its fields 
unchanged, the way an
+    // ALTER TABLE changing an option would: the table itself then serves the 
changed option.
+    // The fields are carried over from the latest schema so their attributes 
survive: rebuilt
+    // raw fields would lose the non-nullable primary keys the table schema 
enforces.
+    Status EvolveSchemaWithOptions() const {
+        SchemaManager schema_manager(dir_->GetFileSystem(), table_path_);
+        PAIMON_ASSIGN_OR_RAISE(std::optional<std::shared_ptr<TableSchema>> 
latest,
+                               schema_manager.Latest());
+        if (!latest.has_value()) {
+            return Status::Invalid("table schema does not exist");
+        }
+        return TestHelper::WriteNextSchema(dir_->GetFileSystem(), table_path_,
+                                           latest.value()->Fields(),
+                                           latest.value()->HighestFieldId(), 
options_);
+    }
+
+    Result<std::unique_ptr<FileStoreWrite>> CreateRealtimeWriter(
+        const std::shared_ptr<RealtimeContext>& realtime_context,
+        const std::shared_ptr<Table>& table) const {
+        WriteContextBuilder builder(table_path_, commit_user_);
+        
builder.SetOptions(options_).WithStreamingMode(true).WithRealtimeContext(realtime_context);
+        PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<WriteContext> context, 
builder.Finish());
+        return table->NewWrite(std::move(context));
+    }

Review Comment:
   Is this function used anywhere?
   



##########
src/paimon/core/table/system/global_system_tables.h:
##########
@@ -33,7 +33,14 @@ class FileSystem;
 /// Context passed to global system table constructors, providing catalog-level
 /// access for enumerating databases, tables, and reading metadata.
 struct GlobalSystemTableContext {
-    const Catalog* catalog = nullptr;  // non-owning pointer
+    /// The catalog the table reads its rows through, non-owning: the rows are 
built lazily,
+    /// so a table that outlives the call that built it must be handed the 
shared ownership
+    /// below too, or this pointer dangles at the first `BuildRows`.
+    const Catalog* catalog = nullptr;
+    /// The shared ownership a returned table keeps of its catalog, so its 
lazy `BuildRows`
+    /// cannot read through one already destroyed. Empty where the table is 
transient - used
+    /// within the call, before the catalog can go away.
+    std::shared_ptr<const Catalog> catalog_retainer;

Review Comment:
   This design is a bit weird—why can’t we just use `std::shared_ptr<Catalog>` 
directly?
   



##########
test/inte/realtime_write_inte_test.cpp:
##########
@@ -2084,6 +2124,9 @@ TEST_F(RealtimeWriteInteTest, 
TestRealtimeDeduplicateAcrossMultipleFileIndexStor
     ASSERT_OK(embedded_writer->Close());
 
     options_[Options::FILE_INDEX_IN_MANIFEST_THRESHOLD] = "1B";
+    // The tighter threshold is a change the table itself carries, the way an 
ALTER TABLE would:
+    // the schema is evolved to it, so every later write, commit, and read 
serves it.
+    ASSERT_OK(EvolveSchemaWithOptions());

Review Comment:
   Why do we need real schema evolution and a new schema file here? In 
practice, users might also use some session-level settings.



##########
src/paimon/core/operation/internal_read_context.h:
##########
@@ -56,10 +58,27 @@ class InternalReadContext {
         std::string field_name;
     };
 
+    /// Builds the context of a read a table serves: the options are the 
table's own, already
+    /// parsed - the store's `options()` - and the cache the context carries 
is armed on them
+    /// here. `path` names the root path the table was opened with, which the 
read reads from in
+    /// place of the one named by the context.
+    static Result<std::unique_ptr<InternalReadContext>> Create(
+        const std::shared_ptr<ReadContext>& read_context,
+        const std::shared_ptr<TableSchema>& table_schema, CoreOptions 
core_options,
+        const std::string* path = nullptr);

Review Comment:
   Prefer not to use default parameters in the production code if 
possible—otherwise we can’t be sure every call site has been updated. Also, why 
is `CoreOptions` not `const &`?
   And adding an extra `const std::string* path` sounds a bit odd.



##########
src/paimon/core/operation/internal_read_context.h:
##########
@@ -164,7 +183,15 @@ class InternalReadContext {
                         const std::shared_ptr<TableSchema>& table_schema,
                         const std::shared_ptr<arrow::Schema>& read_schema,
                         const CoreOptions& options,
-                        const std::shared_ptr<arrow::MemoryPool>& arrow_pool);
+                        const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
+                        std::optional<std::string> path = std::nullopt);

Review Comment:
   Please keep the `pool` parameter at the end.
   



##########
src/paimon/core/table/system/audit_log_system_table.cpp:
##########
@@ -393,7 +395,12 @@ Result<std::unique_ptr<TableScan>> 
AuditLogSystemTable::NewScan(
         .WithStreamingMode(context->IsStreamingMode())
         .WithMemoryPool(context->GetMemoryPool())
         .WithExecutor(context->GetExecutor())
-        .WithFileSystem(context->GetSpecificFileSystem())
+        // The data table behind this view is served through the file system 
the view itself was
+        // opened with, so a context without one of its own falls back to it 
rather than the
+        // default backend.
+        .WithFileSystem(context->GetSpecificFileSystem() != nullptr
+                            ? context->GetSpecificFileSystem()
+                            : GetFileSystem())

Review Comment:
   Why doesn’t this consider `context->GetFileSystemSchemeToIdentifierMap()`?
   It seems a bit asymmetric with `read`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to