lxy-9602 commented on code in PR #447:
URL: https://github.com/apache/paimon-cpp/pull/447#discussion_r4374372099
##########
docs/source/user_guide/format_table.rst:
##########
Review Comment:
I’m a bit confused—now that expect for a `data table`, shouldn’t all the
other table types use the `table->NewScan()`-style interfaces?
##########
include/paimon/read_context.h:
##########
@@ -246,7 +246,7 @@ class PAIMON_EXPORT ReadContextBuilder {
/// `SetTableSchema()`, `WithFileSystem()`,
`WithFileSystemSchemeToIdentifierMap()` and a
/// branch are refused here rather than ignored.
///
- /// @param table The format table to read, as `Catalog::GetFormatTable()`
hands it back.
+ /// @param table The format table to read, as `Catalog::GetTable()` hands
it back.
explicit ReadContextBuilder(const std::shared_ptr<FormatTable>& table);
Review Comment:
I’m thinking about removing `FormatTable` from `xxxContextBuilder`. Wouldn’t
it be better if all special tables also go through `table` (i.e., create the
behavior via `table`)? Only the `data table` might be allowed to be created
directly via the context.
Otherwise, the current design feels a bit too coupled between `context` and
`table`.
##########
src/paimon/core/catalog/file_system_catalog_test.cpp:
##########
@@ -223,10 +230,15 @@ TEST(FileSystemCatalogTest,
TestOptionsSystemTableCatalog) {
ASSERT_FALSE(loaded_schema->field(1)->nullable());
ASSERT_OK_AND_ASSIGN(auto system_table,
catalog.GetTable(options_identifier));
- ASSERT_EQ(system_table->Name(), "tbl1$options");
- // A system table reads the files of the data table it belongs to, so the
table opened
- // for it carries that table's root path.
+ // A system table comes back as the `SystemTable` it is: it answers for
itself with its own
+ // read-only schema and the root path of the data table it reads.
+ ASSERT_TRUE(std::dynamic_pointer_cast<SystemTable>(system_table) !=
nullptr);
+ ASSERT_TRUE(std::dynamic_pointer_cast<FormatTable>(system_table) ==
nullptr);
+ ASSERT_EQ("options", system_table->Name());
+ ASSERT_EQ("options", system_table->FullName());
Review Comment:
For system tables, should the full name and name be `tbl1$options`?
##########
src/paimon/core/table/system/system_table.h:
##########
@@ -49,19 +50,48 @@ struct SystemTablePath {
bool is_global = false;
};
-/// Base interface for table-scoped system tables such as `T$options` and
`T$snapshots`.
+/// Base class for table-scoped system tables such as `T$options` and
`T$snapshots`.
///
-/// Implementations expose a read-only schema and create their own scan/read
objects.
-class SystemTable : public std::enable_shared_from_this<SystemTable> {
+/// A system table is the read-only view a `T$system` identifier names: it
exposes a schema of
+/// its own and serves scans and reads from the data table it belongs to,
through the same
+/// `Table` interface as every other kind of table. `Name()` - which each kind
of system table
+/// answers for itself - is its full name as well: the identifier it is opened
by belongs to the
+/// data table underneath and would name that one.
+class SystemTable : public Table, public
std::enable_shared_from_this<SystemTable> {
Review Comment:
Why is this designed this way here? In Java it just returns `"tbl$options"`.
##########
include/paimon/table/source/table_scan.h:
##########
@@ -36,7 +36,8 @@ class ScanContext;
/// A scanner interface for reading table's meta and create a plan.
class PAIMON_EXPORT TableScan {
public:
- /// Create an instance of `TableScan`.
+ /// Create an instance of `TableScan`: the table the context names is
loaded first, then the
Review Comment:
"the table the context names" -> "the table created by the context"?
##########
src/paimon/core/table/format/format_table.cpp:
##########
@@ -177,35 +193,196 @@ Result<std::shared_ptr<FormatTable>> FormatTable::Create(
core_options.FormatTablePartitionOnlyValueInPath(),
location_carries_paimon_metadata));
}
-Result<std::shared_ptr<FormatTable>> FormatTable::Copy(
- const std::shared_ptr<FormatTable>& table,
- const std::map<std::string, std::string>& dynamic_options) {
- if (table == nullptr) {
- return Status::Invalid("format table requires a table to copy");
- }
+Result<std::shared_ptr<FormatTable>> FormatTable::CopySelf(
+ const std::map<std::string, std::string>& dynamic_options) const {
if (dynamic_options.empty()) {
- return table;
+ return SharedSelf();
}
// The options this table already carries go back in as dynamic ones, so
an option an earlier
// call set is not lost to the schema's own value underneath it.
- std::map<std::string, std::string> options = table->options_;
+ std::map<std::string, std::string> options = options_;
for (const auto& [key, value] : dynamic_options) {
options[key] = value;
}
- return Create(table->file_system_, table->location_, table->identifier_,
table->schema_,
- table->location_carries_paimon_metadata_, options);
+ return Create(file_system_, root_path_, identifier_,
+ std::static_pointer_cast<DataSchema>(schema_),
location_carries_paimon_metadata_,
+ options);
}
-const std::vector<std::string>& FormatTable::PartitionKeys() const {
- return schema_->PartitionKeys();
+Result<std::shared_ptr<Table>> FormatTable::Copy(
+ const std::map<std::string, std::string>& dynamic_options) const {
+ PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<FormatTable> copied,
CopySelf(dynamic_options));
+ return std::shared_ptr<Table>(std::move(copied));
}
-std::string FormatTable::FullName() const {
- return identifier_.GetFullName();
+const std::vector<std::string>& FormatTable::PartitionKeys() const {
+ return LatestSchema()->PartitionKeys();
}
Result<std::unique_ptr<::ArrowSchema>> FormatTable::GetArrowSchema() const {
return schema_->GetArrowSchema();
}
+Result<std::unique_ptr<TableScan>> FormatTable::NewScan(
+ std::unique_ptr<ScanContext> context) const {
+ std::shared_ptr<ScanContext> shared_context = std::move(context);
+ if (shared_context == nullptr) {
+ return Status::Invalid("scan context is null pointer");
+ }
+ if (shared_context->GetMemoryPool() == nullptr) {
+ return Status::Invalid("memory pool is null pointer");
+ }
+ if (shared_context->GetExecutor() == nullptr) {
+ return Status::Invalid("executor is null pointer");
+ }
+ PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<FormatTable> table,
+ CopySelf(shared_context->GetOptions()));
+ // Anything a `ScanContext` carries that a format table cannot honour is
refused rather than
+ // silently dropped.
+ if (shared_context->IsStreamingMode()) {
+ return Status::NotImplemented(
+ "a format table has no snapshots, so there is nothing for a
streaming scan to follow");
+ }
+ if (shared_context->GetRealtimeContext() != nullptr) {
+ return Status::NotImplemented(
+ "a format table has no real-time store to union with what is on
disk");
+ }
+ if (shared_context->GetGlobalIndexResult() != nullptr) {
+ return Status::NotImplemented("a format table carries no global
index");
+ }
+ std::map<std::string, std::string> partition_filter;
+ std::shared_ptr<ScanFilter> filters = shared_context->GetScanFilters();
+ if (filters != nullptr) {
+ if (filters->GetPredicate() != nullptr) {
+ return Status::NotImplemented(
+ "a format table scan does not take a predicate: its files
carry no statistics to "
+ "skip by, and pruning by the partition columns a predicate
names is not "
+ "implemented yet; give the partition values as a partition
filter instead");
+ }
+ if (filters->GetBucketFilter()) {
+ return Status::NotImplemented("a format table has no buckets to
filter by");
+ }
+ const std::vector<std::map<std::string, std::string>>&
partition_filters =
+ filters->GetPartitionFilters();
+ if (partition_filters.size() > 1) {
+ return Status::NotImplemented("a format table scan takes at most
one partition filter");
+ }
+ if (partition_filters.size() == 1) {
+ partition_filter = partition_filters.front();
+ }
+ }
+ PAIMON_ASSIGN_OR_RAISE(
+ std::unique_ptr<FormatTableScan> scan,
+ FormatTableScan::Create(table, partition_filter,
shared_context->GetLimit()));
+ return std::unique_ptr<TableScan>(std::move(scan));
+}
+
+Result<std::unique_ptr<TableRead>> FormatTable::NewRead(
+ std::unique_ptr<ReadContext> context) const {
+ std::shared_ptr<ReadContext> read_context = std::move(context);
+ if (read_context == nullptr) {
+ return Status::Invalid("read context is null pointer");
+ }
+ if (read_context->GetMemoryPool() == nullptr) {
+ return Status::Invalid("memory pool is null pointer");
+ }
+ if (read_context->GetExecutor() == nullptr) {
+ return Status::Invalid("executor is null pointer");
+ }
+ if (read_context->HasVectorOrFullTextSearch()) {
+ return Status::NotImplemented(
+ "VectorSearch or FullTextSearch is not supported for format
tables");
+ }
+ if (PredicateUtils::ContainsFullTextMatch(read_context->GetPredicate())) {
+ return Status::NotImplemented("FullTextMatch predicate execution is
not implemented yet");
+ }
+ PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<FormatTable> table,
+ CopySelf(read_context->GetOptions()));
+ PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<FormatTableRead> read,
+ FormatTableRead::Create(table, read_context));
+ return std::unique_ptr<TableRead>(std::move(read));
+}
+
+Result<std::unique_ptr<FileStoreWrite>> FormatTable::NewWrite(
+ std::unique_ptr<WriteContext> context) const {
+ if (context == nullptr) {
+ return Status::Invalid("write context is null pointer");
+ }
+ if (context->GetMemoryPool() == nullptr) {
+ return Status::Invalid("memory pool is null pointer");
+ }
+ if (context->GetExecutor() == nullptr) {
+ return Status::Invalid("executor is null pointer");
+ }
+ PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<FormatTable> table,
CopySelf(context->GetOptions()));
+ if (context->GetCatalog() != nullptr) {
+ return Status::Invalid(
+ "WithCatalog() requires a native table; use
WriteContextBuilder(FormatTable) for a "
+ "format table");
+ }
+ if (context->IsStreamingMode()) {
+ return Status::NotImplemented(
+ "a format table has no snapshots, so there is nothing a streaming
write could "
+ "commit against");
+ }
Review Comment:
Are these actually “NotImplemented”, or are some of them “Invalid”?
##########
src/paimon/core/table/system/global_system_tables.cpp:
##########
@@ -300,7 +300,13 @@ Result<std::shared_ptr<SystemTable>>
GlobalSystemTableLoader::Load(
return Status::NotImplemented("global system table requires
catalog context: ",
table_name);
}
- return entry.factory(context);
+ PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<SystemTable> system_table,
+ entry.factory(context));
+ // The table is handed out through the `Table` interface, which
serves the schema
+ // the base holds, so its own read-only schema is adopted before
it leaves the
+ // loader.
+ PAIMON_RETURN_NOT_OK(system_table->AdoptOwnSchema());
+ return system_table;
Review Comment:
This design is a bit coupled—similar to an init function.
Do we need `SystemTable::GetSchema` to override with its own logic?
##########
test/inte/realtime_write_inte_test.cpp:
##########
@@ -559,6 +560,41 @@ class RealtimeWriteInteTest : public ::testing::Test {
return FileStoreWrite::Create(std::move(context));
}
+ // A writer that means to change an option the table was created with
writes through a copy
+ // carrying it, which the context then agrees with.
+ Result<std::shared_ptr<Table>> CopiedTable(
+ const std::map<std::string, std::string>& overrides) const {
+ PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<Catalog> catalog,
Catalog::Create(dir_->Str(), {}));
+ PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<Table> table,
+ catalog->GetTable(Identifier("foo", "bar")));
+ return table->Copy(overrides);
+ }
+
+ // Evolves the table schema to the current options with its fields
unchanged, the way an
+ // ALTER TABLE changing an option would: the table itself then serves the
changed option.
+ // The fields are carried over from the latest schema so their attributes
survive: rebuilt
+ // raw fields would lose the non-nullable primary keys the table schema
enforces.
+ Status EvolveSchemaWithOptions() const {
+ SchemaManager schema_manager(dir_->GetFileSystem(), table_path_);
+ PAIMON_ASSIGN_OR_RAISE(std::optional<std::shared_ptr<TableSchema>>
latest,
+ schema_manager.Latest());
+ if (!latest.has_value()) {
+ return Status::Invalid("table schema does not exist");
+ }
+ return TestHelper::WriteNextSchema(dir_->GetFileSystem(), table_path_,
+ latest.value()->Fields(),
+ latest.value()->HighestFieldId(),
options_);
+ }
+
+ Result<std::unique_ptr<FileStoreWrite>> CreateRealtimeWriter(
+ const std::shared_ptr<RealtimeContext>& realtime_context,
+ const std::shared_ptr<Table>& table) const {
+ WriteContextBuilder builder(table_path_, commit_user_);
+
builder.SetOptions(options_).WithStreamingMode(true).WithRealtimeContext(realtime_context);
+ PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<WriteContext> context,
builder.Finish());
+ return table->NewWrite(std::move(context));
+ }
Review Comment:
Is this function used anywhere?
##########
src/paimon/core/table/system/global_system_tables.h:
##########
@@ -33,7 +33,14 @@ class FileSystem;
/// Context passed to global system table constructors, providing catalog-level
/// access for enumerating databases, tables, and reading metadata.
struct GlobalSystemTableContext {
- const Catalog* catalog = nullptr; // non-owning pointer
+ /// The catalog the table reads its rows through, non-owning: the rows are
built lazily,
+ /// so a table that outlives the call that built it must be handed the
shared ownership
+ /// below too, or this pointer dangles at the first `BuildRows`.
+ const Catalog* catalog = nullptr;
+ /// The shared ownership a returned table keeps of its catalog, so its
lazy `BuildRows`
+ /// cannot read through one already destroyed. Empty where the table is
transient - used
+ /// within the call, before the catalog can go away.
+ std::shared_ptr<const Catalog> catalog_retainer;
Review Comment:
This design is a bit weird—why can’t we just use `std::shared_ptr<Catalog>`
directly?
##########
test/inte/realtime_write_inte_test.cpp:
##########
@@ -2084,6 +2124,9 @@ TEST_F(RealtimeWriteInteTest,
TestRealtimeDeduplicateAcrossMultipleFileIndexStor
ASSERT_OK(embedded_writer->Close());
options_[Options::FILE_INDEX_IN_MANIFEST_THRESHOLD] = "1B";
+ // The tighter threshold is a change the table itself carries, the way an
ALTER TABLE would:
+ // the schema is evolved to it, so every later write, commit, and read
serves it.
+ ASSERT_OK(EvolveSchemaWithOptions());
Review Comment:
Why do we need real schema evolution and a new schema file here? In
practice, users might also use some session-level settings.
##########
src/paimon/core/operation/internal_read_context.h:
##########
@@ -56,10 +58,27 @@ class InternalReadContext {
std::string field_name;
};
+ /// Builds the context of a read a table serves: the options are the
table's own, already
+ /// parsed - the store's `options()` - and the cache the context carries
is armed on them
+ /// here. `path` names the root path the table was opened with, which the
read reads from in
+ /// place of the one named by the context.
+ static Result<std::unique_ptr<InternalReadContext>> Create(
+ const std::shared_ptr<ReadContext>& read_context,
+ const std::shared_ptr<TableSchema>& table_schema, CoreOptions
core_options,
+ const std::string* path = nullptr);
Review Comment:
Prefer not to use default parameters in the production code if
possible—otherwise we can’t be sure every call site has been updated. Also, why
is `CoreOptions` not `const &`?
And adding an extra `const std::string* path` sounds a bit odd.
##########
src/paimon/core/operation/internal_read_context.h:
##########
@@ -164,7 +183,15 @@ class InternalReadContext {
const std::shared_ptr<TableSchema>& table_schema,
const std::shared_ptr<arrow::Schema>& read_schema,
const CoreOptions& options,
- const std::shared_ptr<arrow::MemoryPool>& arrow_pool);
+ const std::shared_ptr<arrow::MemoryPool>& arrow_pool,
+ std::optional<std::string> path = std::nullopt);
Review Comment:
Please keep the `pool` parameter at the end.
##########
src/paimon/core/table/system/audit_log_system_table.cpp:
##########
@@ -393,7 +395,12 @@ Result<std::unique_ptr<TableScan>>
AuditLogSystemTable::NewScan(
.WithStreamingMode(context->IsStreamingMode())
.WithMemoryPool(context->GetMemoryPool())
.WithExecutor(context->GetExecutor())
- .WithFileSystem(context->GetSpecificFileSystem())
+ // The data table behind this view is served through the file system
the view itself was
+ // opened with, so a context without one of its own falls back to it
rather than the
+ // default backend.
+ .WithFileSystem(context->GetSpecificFileSystem() != nullptr
+ ? context->GetSpecificFileSystem()
+ : GetFileSystem())
Review Comment:
Why doesn’t this consider `context->GetFileSystemSchemeToIdentifierMap()`?
It seems a bit asymmetric with `read`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]