SteNicholas commented on code in PR #534:
URL: https://github.com/apache/paimon-cpp/pull/534#discussion_r4512423392
##########
src/paimon/core/table/table.cpp:
##########
@@ -720,13 +778,21 @@ Result<std::unique_ptr<TableScan>>
TableScan::Create(std::unique_ptr<ScanContext
if (shared_context->GetRealtimeContext()) {
PAIMON_RETURN_NOT_OK(RealtimeUtils::ValidateBranch(tmp_options.GetBranch()));
}
+ std::map<std::string, std::string> read_options;
+ PAIMON_ASSIGN_OR_RAISE(
+ read_options,
+ ResolveQueryAuthReadOptions(shared_context->GetCatalog(),
shared_context->GetIdentifier(),
+ shared_context->GetOptions(),
+
BranchManager::NormalizeBranch(tmp_options.GetBranch()),
Review Comment:
Fixed in 3206711. The effective branch is now resolved once from the system
table path along with the identifier, the `branch` option and `WithBranch()`:
- `ScanContextBuilder::Finish()` and `ReadContextBuilder::Finish()` resolve
the branch `tbl$branch_dev$files` names before requesting the table's file
system, carry it into the identifier (`t$branch_dev`, or `t$branch_dev$files`)
and refuse conflicting branches before the catalog is asked for anything.
- `ResolveQueryAuthReadOptions()` resolves the path's branch the way
`SystemTableLoader::LoadFromPath()` does and loads the catalog schema of that
branch, so the policy check and the system table load always use the same
branch.
`TableTest.TestQueryAuthSystemTablesValidateTheBranchRead` uses a catalog
that enables query auth only for branch `restricted`, whose on-disk schema
lacks the option. It checks that `$branch_restricted$` plus `files`,
`manifests`, `binlog`, `audit_log` and `ro` are refused for both scans and
reads, that the catalog was asked for `bar$branch_restricted`, and that
`$files` on main still works.
`ScanContextTest`/`ReadContextTest.TestSystemTablePathResolvesBranchBeforeCatalogRequest`
check the identifier the credentials are requested for, the conflict errors
and that a refused branch makes no catalog request.
##########
src/paimon/core/table/source/query_auth_table_scan.cpp:
##########
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "paimon/core/table/source/query_auth_table_scan.h"
+
+#include <algorithm>
+#include <chrono>
+#include <map>
+#include <optional>
+#include <utility>
+#include <vector>
+
+#include "arrow/c/abi.h"
+#include "arrow/c/bridge.h"
+#include "fmt/format.h"
+#include "paimon/common/utils/arrow/status_utils.h"
+#include "paimon/common/utils/object_utils.h"
+#include "paimon/common/utils/path_util.h"
+#include "paimon/common/utils/string_utils.h"
+#include "paimon/core/catalog/catalog_utils.h"
+#include "paimon/core/catalog/query_auth_catalog.h"
+#include "paimon/core/schema/schema_manager.h"
+#include "paimon/core/table/source/plan_impl.h"
+#include "paimon/core/table/source/query_auth_split.h"
+#include "paimon/core/utils/branch_manager.h"
+#include "paimon/defs.h"
+#include "paimon/predicate/predicate.h"
+#include "paimon/predicate/predicate_builder.h"
+#include "paimon/predicate/predicate_utils.h"
+#include "paimon/schema/schema.h"
+#include "paimon/status.h"
+
+namespace paimon {
+namespace {
+
+std::set<std::string> Intersect(const std::set<std::string>& left,
+ const std::set<std::string>& right) {
+ std::set<std::string> result;
+ for (const std::string& value : left) {
+ if (right.count(value) > 0) {
+ result.insert(value);
+ }
+ }
+ return result;
+}
+
+Status CheckNoMaskedPartitionFilter(const std::set<std::string>&
partition_filter_fields,
+ const std::set<std::string>&
masked_fields) {
+ const std::set<std::string> masked = Intersect(partition_filter_fields,
masked_fields);
+ if (!masked.empty()) {
+ return Status::NotImplemented(fmt::format(
+ "A partition filter cannot be enforced on masked partition key
'{}': it is matched "
+ "against the raw partition value. Filter on the column through the
predicate instead.",
+ *masked.begin()));
+ }
+ return Status::OK();
+}
+
+const DataField* FindField(const std::vector<DataField>& fields, const
std::string& name) {
+ auto iter = std::find_if(fields.begin(), fields.end(),
+ [&](const DataField& field) { return field.Name()
== name; });
+ return iter != fields.end() ? &*iter : nullptr;
+}
+
+std::string FieldIdOrAbsent(const DataField* field) {
+ return field != nullptr ? std::to_string(field->Id()) :
std::string("absent");
+}
+
+bool SameLocation(const Path& left, const Path& right) {
+ auto scheme = [](const Path& path) {
+ std::string lower = StringUtils::ToLowerCase(path.scheme);
+ return lower.empty() ? std::string("file") : lower;
+ };
+ return scheme(left) == scheme(right) && left.authority == right.authority
&&
+ left.path == right.path;
+}
+
+/// Fails unless `path` is where the catalog keeps `identifier`, so the rules
the catalog serves
+/// for one table are never applied to the files of another.
+Status CheckPathIsTableLocation(const Catalog& catalog, const Identifier&
identifier,
+ const std::string& path) {
+ Result<std::string> location = catalog.GetTableLocation(identifier);
+ if (!location.ok()) {
+ return location.status().WithMessage(fmt::format(
+ "Cannot check that {} is the location of table {}, whose query
auth rules the scan "
+ "applies: {}",
+ path, identifier.GetFullName(), location.status().message()));
+ }
+ PAIMON_ASSIGN_OR_RAISE(Path expected, PathUtil::ToPath(location.value()));
+ PAIMON_ASSIGN_OR_RAISE(Path actual, PathUtil::ToPath(path));
+ if (!SameLocation(expected, actual)) {
+ return Status::Invalid(fmt::format(
+ "The scan reads {}, but the catalog keeps table {}, whose query
auth rules it would "
+ "apply, at {}. Scan the location the catalog returns for the
table.",
+ path, identifier.GetFullName(), location.value()));
+ }
+ return Status::OK();
+}
+
+} // namespace
+
+Result<std::unique_ptr<TableScan>> QueryAuthTableScan::Create(
+ const std::shared_ptr<ScanContext>& context, const
std::shared_ptr<TableSchema>& table_schema,
+ const std::shared_ptr<FileSystem>& file_system, const std::string& branch,
+ InnerScanFactory create_inner) {
+ if (context->GetCatalog() == nullptr || !context->GetIdentifier()) {
+ return Status::NotImplemented(fmt::format(
+ "Cannot read a table with '{}=true' without the catalog that
serves its row filters "
+ "and column masks: build the scan with
ScanContextBuilder::WithCatalog().",
+ Options::QUERY_AUTH_ENABLED));
+ }
+ const QueryAuthCatalog* auth_catalog =
AsQueryAuthCatalog(context->GetCatalog());
+ if (auth_catalog == nullptr) {
+ return Status::NotImplemented(fmt::format(
+ "Cannot read a table with '{}=true': its catalog does not support
the query auth "
+ "interface.",
+ Options::QUERY_AUTH_ENABLED));
+ }
+ if (context->GetGlobalIndexResult() != nullptr) {
+ return Status::NotImplemented(
+ "Search is not supported on a query-auth table: the index ranks
raw values, which a "
+ "column mask invalidates.");
+ }
+ if (context->GetRealtimeContext() != nullptr) {
+ return Status::NotImplemented("Real-time reads are not supported on a
query-auth table.");
+ }
+ // The rules are asked for the branch planned, even when only the options
name it.
+ PAIMON_ASSIGN_OR_RAISE(std::string resolved_branch,
+
BranchManager::ResolveBranch(context->GetIdentifier(),
+ context->GetOptions(),
branch, "scan"));
+ PAIMON_ASSIGN_OR_RAISE(
+ Identifier auth_identifier,
+
CatalogUtils::ResolvedBranchIdentifier(context->GetIdentifier().value(),
resolved_branch));
+ PAIMON_RETURN_NOT_OK(
+ CheckPathIsTableLocation(*context->GetCatalog(), auth_identifier,
context->GetPath()));
+ std::set<std::string> user_filter_fields;
+ std::set<std::string> partition_filter_fields;
+ const std::shared_ptr<ScanFilter>& scan_filter = context->GetScanFilters();
+ if (scan_filter != nullptr) {
+ if (scan_filter->GetPredicate() != nullptr) {
+ PAIMON_RETURN_NOT_OK(
+ PredicateUtils::GetAllNames(scan_filter->GetPredicate(),
&user_filter_fields));
+ }
+ for (const auto& partition : scan_filter->GetPartitionFilters()) {
+ for (const auto& [key, value] : partition) {
+ partition_filter_fields.insert(key);
+ }
+ }
+ }
+ // As in Java, the operands of the filters are authorized along with the
projected columns.
+ const std::optional<std::vector<std::string>>& read_field_names =
context->GetReadFieldNames();
+ std::vector<std::string> requested =
read_field_names.value_or(std::vector<std::string>());
+ requested.insert(requested.end(), user_filter_fields.begin(),
user_filter_fields.end());
+ requested.insert(requested.end(), partition_filter_fields.begin(),
+ partition_filter_fields.end());
+ std::vector<std::string> columns;
+ std::set<std::string> seen;
+ for (const std::string& name : requested) {
+ if (!table_schema->GetField(name).ok()) {
+ return Status::NotImplemented(fmt::format(
+ "Column '{}' is not a column of the table schema, which is all
a query-auth scan "
+ "can authorize.",
+ name));
+ }
+ if (seen.insert(name).second) {
+ columns.push_back(name);
+ }
+ }
+ std::optional<std::vector<std::string>> authorization_columns;
+ if (read_field_names) {
+ authorization_columns = std::move(columns);
+ }
+ return std::unique_ptr<TableScan>(new QueryAuthTableScan(
+ context, auth_catalog, std::move(auth_identifier), table_schema,
file_system,
+ resolved_branch, std::move(create_inner),
std::move(authorization_columns),
+ std::move(user_filter_fields), std::move(partition_filter_fields)));
+}
+
+QueryAuthTableScan::QueryAuthTableScan(
+ const std::shared_ptr<ScanContext>& context, const QueryAuthCatalog*
auth_catalog,
+ Identifier auth_identifier, const std::shared_ptr<TableSchema>&
table_schema,
+ const std::shared_ptr<FileSystem>& file_system, const std::string& branch,
+ InnerScanFactory create_inner, std::optional<std::vector<std::string>>
authorization_columns,
+ std::set<std::string> user_filter_fields, std::set<std::string>
partition_filter_fields)
+ : context_(context),
+ auth_catalog_(auth_catalog),
+ auth_identifier_(std::move(auth_identifier)),
+ table_schema_(table_schema),
+ file_system_(file_system),
+ branch_(branch),
+ create_inner_(std::move(create_inner)),
+ authorization_columns_(std::move(authorization_columns)),
+ user_filter_fields_(std::move(user_filter_fields)),
+ partition_filter_fields_(std::move(partition_filter_fields)) {}
+
+std::shared_ptr<Metrics> QueryAuthTableScan::GetMetrics() const {
+ return inner_ != nullptr ? inner_->GetMetrics() : TableScan::GetMetrics();
+}
+
+Status QueryAuthTableScan::AddErrorContext(const Status& status) const {
+ return status.WithMessage(fmt::format("Query auth of {} (branch {}, schema
{}): {}",
+ auth_identifier_.ToString(),
branch_, table_schema_->Id(),
+ status.message()));
+}
+
+Status QueryAuthTableScan::ValidateAgainstCurrentSchemas(const
TableQueryAuthResult& rules) const {
+ SchemaManager schema_manager(file_system_, context_->GetPath(), branch_);
+ PAIMON_ASSIGN_OR_RAISE(std::optional<std::shared_ptr<TableSchema>> latest,
+ schema_manager.Latest());
+ const std::shared_ptr<TableSchema>& latest_schema = latest ?
latest.value() : table_schema_;
+ const std::vector<DataField>& planned_fields = table_schema_->Fields();
+ const std::vector<DataField>& latest_fields = latest_schema->Fields();
+ // The catalog authorizes the columns by their names in the latest schema,
so each must still
+ // be the field the scan plans under that name, whether or not the rules
mention it.
+ std::vector<std::string> columns;
+ if (authorization_columns_) {
+ columns = authorization_columns_.value();
+ } else {
+ for (const DataField& field : planned_fields) {
+ columns.push_back(field.Name());
+ }
+ }
+ auto validate_columns = [&](const std::vector<DataField>& fields, int64_t
schema_id,
+ const char* source) -> Status {
+ for (const std::string& column : columns) {
+ const DataField* planned = FindField(planned_fields, column);
+ const DataField* current = FindField(fields, column);
+ if (planned == nullptr || current == nullptr || planned->Id() !=
current->Id()) {
+ return Status::Invalid(fmt::format(
+ "Column '{}' is field {} in schema {} the scan plans with,
but field {} in {} "
+ "{}, under which the catalog authorized it. Create the
scan again.",
+ column, FieldIdOrAbsent(planned), table_schema_->Id(),
FieldIdOrAbsent(current),
+ source, schema_id));
+ }
+ }
+ return Status::OK();
+ };
+ PAIMON_RETURN_NOT_OK(validate_columns(latest_fields, latest_schema->Id(),
"the latest schema"));
+ PAIMON_RETURN_NOT_OK(rules.ValidateAgainstSchema(latest_fields));
+
PAIMON_RETURN_NOT_OK(rules.ValidateFieldIds(rules.BindFieldIds(latest_fields),
planned_fields));
+ // A rule column the planned schema does not hold would leave every split
unreadable.
+ PAIMON_RETURN_NOT_OK(
+ rules.ValidateFieldIds(rules.BindFieldIds(planned_fields),
planned_fields));
+
+ // The catalog can advance its schema before publishing it under the table
path. It
+ // authorizes current names, so binding those names only to local field
ids could expose an
+ // old column that the catalog has dropped and replaced with another of
the same name.
+ PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<Schema> catalog_schema,
+
context_->GetCatalog()->LoadTableSchema(auth_identifier_));
+ auto catalog_data_schema =
std::dynamic_pointer_cast<DataSchema>(catalog_schema);
+ if (catalog_data_schema == nullptr) {
+ return Status::Invalid("Query auth requires a data table schema from
the catalog.");
+ }
+ PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<::ArrowSchema> c_schema,
+ catalog_data_schema->GetArrowSchema());
+ PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr<arrow::Schema>
arrow_schema,
+ arrow::ImportSchema(c_schema.get()));
+ PAIMON_ASSIGN_OR_RAISE(std::vector<DataField> catalog_fields,
+
DataField::ConvertArrowSchemaToDataFields(arrow_schema));
+ PAIMON_RETURN_NOT_OK(
+ validate_columns(catalog_fields, catalog_data_schema->Id(), "catalog
schema"));
+ PAIMON_RETURN_NOT_OK(rules.ValidateAgainstSchema(catalog_fields));
+ return rules.ValidateFieldIds(rules.BindFieldIds(catalog_fields),
planned_fields);
+}
+
+Result<std::shared_ptr<TableQueryAuthResult>>
QueryAuthTableScan::LoadAndValidateRules() const {
+ const auto start = std::chrono::steady_clock::now();
+ Result<std::shared_ptr<TableQueryAuthResult>> rules =
+ auth_catalog_->AuthTableQuery(auth_identifier_,
authorization_columns_);
Review Comment:
Fixed in 3206711. On every `CreatePlan()`, after fetching the rules, the
scan reloads the catalog schema and location from one response through
`QueryAuthCatalog::LoadTableSchemaAndLocation()`. If the identifier is no
longer bound to the path being scanned, the plan fails with `Invalid` before
any split is returned. The same catalog schema is still used to check that the
selected and rule columns keep their field ids. The splits are now bound as
well: `QueryAuthScope` records the authorized table location and branch with
the field ids and is serialized with the split. `QueryAuthTableRead` refuses a
split authorized for another location or branch, even with matching schemas and
data file names, including on a worker without the catalog.
`TableTest.TestQueryAuthScanReadsOnlyTheLocationOfTheAuthorizedTable`
rebinds the identifier to another table between `TableScan::Create()` and the
first `CreatePlan()` and checks that the plan is refused.
`TableTest.TestQueryAuthReadRejectsSplitsAuthorizedForAnotherTable` and
`...AnotherBranch` check the read-side binding through serialized splits.
`sys.partitions` uses the same one-response location
(`RestCatalogTest.GlobalPartitionsProtectsQueryAuthPartitionValues`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]