[
https://issues.apache.org/jira/browse/DRILL-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123401#comment-18123401
]
ASF GitHub Bot commented on DRILL-8548:
---------------------------------------
cgivre commented on code in PR #3056:
URL: https://github.com/apache/drill/pull/3056#discussion_r4185239588
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/SqlConverter.java:
##########
@@ -245,6 +246,19 @@ public RelRoot toRel(final SqlNode validatedNode) {
RelNode project = LogicalProject.create(rel.rel,
Collections.emptyList(), expressions, rel.validatedRowType);
rel = RelRoot.of(project, rel.validatedRowType, rel.kind);
}
+
+ // Column-level SELECT authorization check. Done after SqlToRelConverter
has
+ // resolved all column references (so we can trace each to its TableScan)
+ // and before flattenTypes/optimization (so column references are intact).
+ //
+ // Skip the whole check when authorization is disabled: the tree walk plus
+ // a getColumnOrigins() metadata query per output column is measurable
+ // planning cost that deployments not using the feature must not pay on
+ // every query.
+ if (AccessAuthorizerManager.isEnabled(drillConfig)) {
+ new ColumnAccessChecker(session, drillConfig,
cluster.getMetadataQuery()).check(rel.rel);
Review Comment:
`DrillViewExpander` calls `toRel` on the view SQL, so this also runs during
view expansion and checks every column the view outputs. With `v = SELECT pub,
secret FROM t`, `SELECT pub FROM v` is denied, which breaks views as a
column-hiding mechanism.
Fix: skip the check when `toRel` is invoked for view expansion and run it
once on the top-level plan only.
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/ColumnAccessChecker.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.conversion;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelShuttleImpl;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexSubQuery;
+import org.apache.calcite.rex.RexVisitor;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.security.AccessAuthorizerManager;
+import org.apache.drill.exec.security.TableAccessResource;
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Visitor that traverses a RelNode tree and enforces column-level SELECT
authorization
+ * via the configured {@link AccessAuthorizer} (Ranger by default).
+ *
+ * <p><b>Design:</b> For every RelNode that carries {@link RexNode} expressions
+ * (Project, Filter, Join, Aggregate, Sort), the visitor collects all {@link
RexInputRef}s
+ * and uses Calcite's {@link RelMetadataQuery#getColumnOrigins(RelNode, int)}
to trace each
+ * referenced column back to its originating {@link TableScan} column index.
This correctly
+ * handles multi-hop projections, filters, joins, and aggregations.</p>
+ *
+ * <p>For a {@link TableScan} that has NO traced column references (e.g.
+ * {@code SELECT * FROM t} with no intervening Project), ALL columns of that
table
+ * are checked.</p>
+ *
+ * <p>Correlated subqueries are handled as well: a reference to an outer query
+ * column from inside a correlated subquery is represented by Calcite as a
+ * {@link RexFieldAccess} over a {@link RexCorrelVariable} (e.g. {@code
$cor0.id}),
+ * not as a {@link RexInputRef}. Each correlation variable carries a unique
+ * name {@code $corN} assigned by {@code SqlToRelConverter}; we keep an exact
+ * map from variable name to the enclosing {@link RelNode} row scope it refers
+ * to. The map entry is populated the FIRST time we see a given {@code $corN}
+ * during Rex traversal: we use {@code putIfAbsent(name, enclosingScope)} so
+ * that subsequent encounters (including cross-nested skip-level references
+ * from deeper subqueries to an outer {@code $cor0}) never overwrite the
+ * original, correct binding. This avoids the ambiguity of row-type-based
+ * heuristics: multiple enclosing scopes with identical schemas no longer
+ * cause false positives or missed detections, and no scope stack is needed —
+ * a single field tracks the innermost enclosing scope and is saved/restored
+ * around each subquery entry via try/finally.</p>
+ *
+ * <p>System schemas (INFORMATION_SCHEMA, sys) are bypassed inside the
authorizer
+ * implementation. When authorization is disabled, the visitor is a no-op
(fail-open).</p>
+ */
+class ColumnAccessChecker extends RelShuttleImpl {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ColumnAccessChecker.class);
+
+ private final UserSession session;
+ private final DrillConfig drillConfig;
+ private final RelMetadataQuery mq;
+
+ // Records each table's referenced column indices. Uses IdentityHashMap
because
+ // RelOptTable equals/hashCode may be expensive or not identity-based.
+ private final Map<RelOptTable, Set<Integer>> tableToReferencedCols = new
IdentityHashMap<>();
+
+ // The innermost enclosing row scope for correlated references. When we are
+ // about to descend into a RexSubQuery's Rel tree, we save the previous value
+ // and set this field to the outer-query inputNode that the subquery's
+ // $cor* variables are relative to; the saved value is restored in a finally
+ // block. Because variable → scope registration uses putIfAbsent in the Rex
+ // collector, a single field is sufficient even for deep nesting: skip-level
+ // references to an outer $corN reuse the binding that was established the
+ // first time that variable was seen (at a shallower enclosing scope).
+ private RelNode enclosingScope;
+
+ // Exact, name-keyed scope mapping: $cor0 → outer row scope, $cor1 → outer
+ // row scope, etc. Populated with putIfAbsent during Rex traversal so the
+ // first occurrence of each $corN establishes a permanent binding. Lookup is
+ // pure get(); a miss falls back (safely, in the over-approximate direction)
+ // to enclosingScope and emits a warning so unusual plans remain diagnosable.
+ private final Map<String, RelNode> correlationScopeByVar = new HashMap<>();
+
+ ColumnAccessChecker(UserSession session, DrillConfig drillConfig,
RelMetadataQuery mq) {
+ this.session = session;
+ this.drillConfig = drillConfig;
+ this.mq = mq;
+ }
+
+ /**
+ * Entry point: traverse the tree and enforce column-level access.
+ */
+ void check(RelNode root) {
+ // Also trace the root node's output columns (covers bare TableScan root or
+ // top-level Project output).
+ traceOutputColumns(root);
+
+ // Walk the tree to collect RexInputRef origins from all
expression-bearing nodes.
+ root.accept(this);
+ }
+
+ // ------------------------------------------------------------------
+ // RelShuttle overrides — collect RexInputRefs from expression-bearing nodes
+ // ------------------------------------------------------------------
+
+ @Override
+ public RelNode visit(LogicalProject project) {
+ collectRefs(project.getProjects(), project.getInput());
+ return super.visit(project);
+ }
+
+ @Override
+ public RelNode visit(LogicalFilter filter) {
+ if (filter.getCondition() != null) {
+ analyzeRex(filter.getCondition(), filter.getInput(), -1, null);
+ }
+ return super.visit(filter);
+ }
+
+ @Override
+ public RelNode visit(LogicalJoin join) {
+ if (join.getCondition() != null) {
+ int leftCount = join.getLeft().getRowType().getFieldCount();
+ analyzeRex(join.getCondition(), join.getRight(), leftCount,
join.getLeft());
Review Comment:
Subqueries in the join condition get only the right input as their
correlation scope, but their correlated field indexes are into the combined
left+right row. For `a JOIN b ON a.x IN (SELECT y FROM c WHERE c.z =
a.secret)`, `a.secret` is resolved against `b`.
Fix: pass the join itself as the correlation scope so indexes resolve across
both inputs.
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/ColumnAccessChecker.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.conversion;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelShuttleImpl;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexSubQuery;
+import org.apache.calcite.rex.RexVisitor;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.security.AccessAuthorizerManager;
+import org.apache.drill.exec.security.TableAccessResource;
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Visitor that traverses a RelNode tree and enforces column-level SELECT
authorization
+ * via the configured {@link AccessAuthorizer} (Ranger by default).
+ *
+ * <p><b>Design:</b> For every RelNode that carries {@link RexNode} expressions
+ * (Project, Filter, Join, Aggregate, Sort), the visitor collects all {@link
RexInputRef}s
+ * and uses Calcite's {@link RelMetadataQuery#getColumnOrigins(RelNode, int)}
to trace each
+ * referenced column back to its originating {@link TableScan} column index.
This correctly
+ * handles multi-hop projections, filters, joins, and aggregations.</p>
+ *
+ * <p>For a {@link TableScan} that has NO traced column references (e.g.
+ * {@code SELECT * FROM t} with no intervening Project), ALL columns of that
table
+ * are checked.</p>
+ *
+ * <p>Correlated subqueries are handled as well: a reference to an outer query
+ * column from inside a correlated subquery is represented by Calcite as a
+ * {@link RexFieldAccess} over a {@link RexCorrelVariable} (e.g. {@code
$cor0.id}),
+ * not as a {@link RexInputRef}. Each correlation variable carries a unique
+ * name {@code $corN} assigned by {@code SqlToRelConverter}; we keep an exact
+ * map from variable name to the enclosing {@link RelNode} row scope it refers
+ * to. The map entry is populated the FIRST time we see a given {@code $corN}
+ * during Rex traversal: we use {@code putIfAbsent(name, enclosingScope)} so
+ * that subsequent encounters (including cross-nested skip-level references
+ * from deeper subqueries to an outer {@code $cor0}) never overwrite the
+ * original, correct binding. This avoids the ambiguity of row-type-based
+ * heuristics: multiple enclosing scopes with identical schemas no longer
+ * cause false positives or missed detections, and no scope stack is needed —
+ * a single field tracks the innermost enclosing scope and is saved/restored
+ * around each subquery entry via try/finally.</p>
+ *
+ * <p>System schemas (INFORMATION_SCHEMA, sys) are bypassed inside the
authorizer
+ * implementation. When authorization is disabled, the visitor is a no-op
(fail-open).</p>
+ */
+class ColumnAccessChecker extends RelShuttleImpl {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ColumnAccessChecker.class);
+
+ private final UserSession session;
+ private final DrillConfig drillConfig;
+ private final RelMetadataQuery mq;
+
+ // Records each table's referenced column indices. Uses IdentityHashMap
because
+ // RelOptTable equals/hashCode may be expensive or not identity-based.
+ private final Map<RelOptTable, Set<Integer>> tableToReferencedCols = new
IdentityHashMap<>();
+
+ // The innermost enclosing row scope for correlated references. When we are
+ // about to descend into a RexSubQuery's Rel tree, we save the previous value
+ // and set this field to the outer-query inputNode that the subquery's
+ // $cor* variables are relative to; the saved value is restored in a finally
+ // block. Because variable → scope registration uses putIfAbsent in the Rex
+ // collector, a single field is sufficient even for deep nesting: skip-level
+ // references to an outer $corN reuse the binding that was established the
+ // first time that variable was seen (at a shallower enclosing scope).
+ private RelNode enclosingScope;
+
+ // Exact, name-keyed scope mapping: $cor0 → outer row scope, $cor1 → outer
+ // row scope, etc. Populated with putIfAbsent during Rex traversal so the
+ // first occurrence of each $corN establishes a permanent binding. Lookup is
+ // pure get(); a miss falls back (safely, in the over-approximate direction)
+ // to enclosingScope and emits a warning so unusual plans remain diagnosable.
+ private final Map<String, RelNode> correlationScopeByVar = new HashMap<>();
+
+ ColumnAccessChecker(UserSession session, DrillConfig drillConfig,
RelMetadataQuery mq) {
+ this.session = session;
+ this.drillConfig = drillConfig;
+ this.mq = mq;
+ }
+
+ /**
+ * Entry point: traverse the tree and enforce column-level access.
+ */
+ void check(RelNode root) {
+ // Also trace the root node's output columns (covers bare TableScan root or
+ // top-level Project output).
+ traceOutputColumns(root);
+
+ // Walk the tree to collect RexInputRef origins from all
expression-bearing nodes.
+ root.accept(this);
+ }
+
+ // ------------------------------------------------------------------
+ // RelShuttle overrides — collect RexInputRefs from expression-bearing nodes
+ // ------------------------------------------------------------------
+
+ @Override
+ public RelNode visit(LogicalProject project) {
+ collectRefs(project.getProjects(), project.getInput());
+ return super.visit(project);
+ }
+
+ @Override
+ public RelNode visit(LogicalFilter filter) {
+ if (filter.getCondition() != null) {
+ analyzeRex(filter.getCondition(), filter.getInput(), -1, null);
+ }
+ return super.visit(filter);
+ }
+
+ @Override
+ public RelNode visit(LogicalJoin join) {
+ if (join.getCondition() != null) {
+ int leftCount = join.getLeft().getRowType().getFieldCount();
+ analyzeRex(join.getCondition(), join.getRight(), leftCount,
join.getLeft());
+ }
+ return super.visit(join);
+ }
+
+ @Override
+ public RelNode visit(LogicalAggregate aggregate) {
+ for (int i : aggregate.getGroupSet()) {
+ traceColumnOrigin(aggregate.getInput(), i);
+ }
+ // AggregateCall arguments (the x in SUM(x)), FILTER (WHERE ...) columns,
+ // and WITHIN GROUP collation columns are not covered by getGroupSet().
+ // Trace them explicitly rather than relying on an implicit Project beneath
+ // the aggregate (plan shape can change across Calcite upgrades).
+ for (AggregateCall call : aggregate.getAggCallList()) {
+ for (int arg : call.getArgList()) {
+ traceColumnOrigin(aggregate.getInput(), arg);
+ }
+ if (call.filterArg >= 0) {
+ traceColumnOrigin(aggregate.getInput(), call.filterArg);
+ }
+ if (call.getCollation() != null) {
+ for (RelFieldCollation fc : call.getCollation().getFieldCollations()) {
+ traceColumnOrigin(aggregate.getInput(), fc.getFieldIndex());
+ }
+ }
+ }
+ return super.visit(aggregate);
+ }
+
+ @Override
+ public RelNode visit(LogicalSort sort) {
+ if (sort.getCollation() != null) {
+ sort.getCollation().getFieldCollations().forEach(fc ->
+ traceColumnOrigin(sort.getInput(), fc.getFieldIndex()));
+ }
+ // LIMIT/OFFSET are RexNode expressions that may carry RexSubQuery or
+ // RexInputRef (e.g. LIMIT (SELECT MAX(amount) FROM ...)). analyzeRex
+ // descends into them; without it, subqueries reachable only from Sort
+ // would never be column-checked.
+ if (sort.fetch != null) {
+ analyzeRex(sort.fetch, sort.getInput(), -1, null);
+ }
+ if (sort.offset != null) {
+ analyzeRex(sort.offset, sort.getInput(), -1, null);
+ }
+ return super.visit(sort);
+ }
+
+ @Override
+ public RelNode visit(TableScan scan) {
+ RelOptTable table = scan.getTable();
+
+ // Determine which columns to check: traced columns, or ALL if none were
traced
+ // (SELECT * FROM t case).
+ Set<Integer> referencedColIndices = tableToReferencedCols.get(table);
+ List<String> allColumnNames = scan.getRowType().getFieldNames();
+
+ Set<String> columnsToCheck;
+ if (referencedColIndices == null || referencedColIndices.isEmpty()) {
Review Comment:
Two issues with this branch:
1. Schemaless sources (JSON/CSV, Parquet w/o metadata): the row type is just
the dynamic star `**`, so that's the only name sent to Ranger. A policy
allowing `column=*` and denying `secret` matches `**`, and `SELECT *` returns
`secret`. Fix: if any checked column is a dynamic star, require access to all
columns (e.g. check `*` and fail if any column-level deny exists for the table).
2. "None referenced" is treated as `SELECT *`, so `COUNT(*)`, `SELECT 1`,
and `EXISTS (SELECT 1 ...)` are denied for anyone restricted on a single
column. Fix: distinguish "no columns read" from "star expanded"; for the
former, do a table-level check only.
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/ColumnAccessChecker.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.conversion;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelShuttleImpl;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexSubQuery;
+import org.apache.calcite.rex.RexVisitor;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.security.AccessAuthorizerManager;
+import org.apache.drill.exec.security.TableAccessResource;
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Visitor that traverses a RelNode tree and enforces column-level SELECT
authorization
+ * via the configured {@link AccessAuthorizer} (Ranger by default).
+ *
+ * <p><b>Design:</b> For every RelNode that carries {@link RexNode} expressions
+ * (Project, Filter, Join, Aggregate, Sort), the visitor collects all {@link
RexInputRef}s
+ * and uses Calcite's {@link RelMetadataQuery#getColumnOrigins(RelNode, int)}
to trace each
+ * referenced column back to its originating {@link TableScan} column index.
This correctly
+ * handles multi-hop projections, filters, joins, and aggregations.</p>
+ *
+ * <p>For a {@link TableScan} that has NO traced column references (e.g.
+ * {@code SELECT * FROM t} with no intervening Project), ALL columns of that
table
+ * are checked.</p>
+ *
+ * <p>Correlated subqueries are handled as well: a reference to an outer query
+ * column from inside a correlated subquery is represented by Calcite as a
+ * {@link RexFieldAccess} over a {@link RexCorrelVariable} (e.g. {@code
$cor0.id}),
+ * not as a {@link RexInputRef}. Each correlation variable carries a unique
+ * name {@code $corN} assigned by {@code SqlToRelConverter}; we keep an exact
+ * map from variable name to the enclosing {@link RelNode} row scope it refers
+ * to. The map entry is populated the FIRST time we see a given {@code $corN}
+ * during Rex traversal: we use {@code putIfAbsent(name, enclosingScope)} so
+ * that subsequent encounters (including cross-nested skip-level references
+ * from deeper subqueries to an outer {@code $cor0}) never overwrite the
+ * original, correct binding. This avoids the ambiguity of row-type-based
+ * heuristics: multiple enclosing scopes with identical schemas no longer
+ * cause false positives or missed detections, and no scope stack is needed —
+ * a single field tracks the innermost enclosing scope and is saved/restored
+ * around each subquery entry via try/finally.</p>
+ *
+ * <p>System schemas (INFORMATION_SCHEMA, sys) are bypassed inside the
authorizer
+ * implementation. When authorization is disabled, the visitor is a no-op
(fail-open).</p>
+ */
+class ColumnAccessChecker extends RelShuttleImpl {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ColumnAccessChecker.class);
+
+ private final UserSession session;
+ private final DrillConfig drillConfig;
+ private final RelMetadataQuery mq;
+
+ // Records each table's referenced column indices. Uses IdentityHashMap
because
+ // RelOptTable equals/hashCode may be expensive or not identity-based.
+ private final Map<RelOptTable, Set<Integer>> tableToReferencedCols = new
IdentityHashMap<>();
+
+ // The innermost enclosing row scope for correlated references. When we are
+ // about to descend into a RexSubQuery's Rel tree, we save the previous value
+ // and set this field to the outer-query inputNode that the subquery's
+ // $cor* variables are relative to; the saved value is restored in a finally
+ // block. Because variable → scope registration uses putIfAbsent in the Rex
+ // collector, a single field is sufficient even for deep nesting: skip-level
+ // references to an outer $corN reuse the binding that was established the
+ // first time that variable was seen (at a shallower enclosing scope).
+ private RelNode enclosingScope;
+
+ // Exact, name-keyed scope mapping: $cor0 → outer row scope, $cor1 → outer
+ // row scope, etc. Populated with putIfAbsent during Rex traversal so the
+ // first occurrence of each $corN establishes a permanent binding. Lookup is
+ // pure get(); a miss falls back (safely, in the over-approximate direction)
+ // to enclosingScope and emits a warning so unusual plans remain diagnosable.
+ private final Map<String, RelNode> correlationScopeByVar = new HashMap<>();
+
+ ColumnAccessChecker(UserSession session, DrillConfig drillConfig,
RelMetadataQuery mq) {
+ this.session = session;
+ this.drillConfig = drillConfig;
+ this.mq = mq;
+ }
+
+ /**
+ * Entry point: traverse the tree and enforce column-level access.
+ */
+ void check(RelNode root) {
+ // Also trace the root node's output columns (covers bare TableScan root or
+ // top-level Project output).
+ traceOutputColumns(root);
+
+ // Walk the tree to collect RexInputRef origins from all
expression-bearing nodes.
+ root.accept(this);
+ }
+
+ // ------------------------------------------------------------------
+ // RelShuttle overrides — collect RexInputRefs from expression-bearing nodes
+ // ------------------------------------------------------------------
+
+ @Override
+ public RelNode visit(LogicalProject project) {
+ collectRefs(project.getProjects(), project.getInput());
+ return super.visit(project);
+ }
+
+ @Override
+ public RelNode visit(LogicalFilter filter) {
+ if (filter.getCondition() != null) {
+ analyzeRex(filter.getCondition(), filter.getInput(), -1, null);
+ }
+ return super.visit(filter);
+ }
+
+ @Override
+ public RelNode visit(LogicalJoin join) {
+ if (join.getCondition() != null) {
+ int leftCount = join.getLeft().getRowType().getFieldCount();
+ analyzeRex(join.getCondition(), join.getRight(), leftCount,
join.getLeft());
+ }
+ return super.visit(join);
+ }
+
+ @Override
+ public RelNode visit(LogicalAggregate aggregate) {
+ for (int i : aggregate.getGroupSet()) {
+ traceColumnOrigin(aggregate.getInput(), i);
+ }
+ // AggregateCall arguments (the x in SUM(x)), FILTER (WHERE ...) columns,
+ // and WITHIN GROUP collation columns are not covered by getGroupSet().
+ // Trace them explicitly rather than relying on an implicit Project beneath
+ // the aggregate (plan shape can change across Calcite upgrades).
+ for (AggregateCall call : aggregate.getAggCallList()) {
+ for (int arg : call.getArgList()) {
+ traceColumnOrigin(aggregate.getInput(), arg);
+ }
+ if (call.filterArg >= 0) {
+ traceColumnOrigin(aggregate.getInput(), call.filterArg);
+ }
+ if (call.getCollation() != null) {
+ for (RelFieldCollation fc : call.getCollation().getFieldCollations()) {
+ traceColumnOrigin(aggregate.getInput(), fc.getFieldIndex());
+ }
+ }
+ }
+ return super.visit(aggregate);
+ }
+
+ @Override
+ public RelNode visit(LogicalSort sort) {
+ if (sort.getCollation() != null) {
+ sort.getCollation().getFieldCollations().forEach(fc ->
+ traceColumnOrigin(sort.getInput(), fc.getFieldIndex()));
+ }
+ // LIMIT/OFFSET are RexNode expressions that may carry RexSubQuery or
+ // RexInputRef (e.g. LIMIT (SELECT MAX(amount) FROM ...)). analyzeRex
+ // descends into them; without it, subqueries reachable only from Sort
+ // would never be column-checked.
+ if (sort.fetch != null) {
+ analyzeRex(sort.fetch, sort.getInput(), -1, null);
+ }
+ if (sort.offset != null) {
+ analyzeRex(sort.offset, sort.getInput(), -1, null);
+ }
+ return super.visit(sort);
+ }
+
+ @Override
+ public RelNode visit(TableScan scan) {
+ RelOptTable table = scan.getTable();
+
+ // Determine which columns to check: traced columns, or ALL if none were
traced
+ // (SELECT * FROM t case).
+ Set<Integer> referencedColIndices = tableToReferencedCols.get(table);
+ List<String> allColumnNames = scan.getRowType().getFieldNames();
+
+ Set<String> columnsToCheck;
+ if (referencedColIndices == null || referencedColIndices.isEmpty()) {
+ // SELECT * — check all columns
+ columnsToCheck = new HashSet<>(allColumnNames);
+ } else {
+ columnsToCheck = new HashSet<>();
+ for (int idx : referencedColIndices) {
+ if (idx >= 0 && idx < allColumnNames.size()) {
+ columnsToCheck.add(allColumnNames.get(idx));
+ }
+ }
+ }
+
+ if (columnsToCheck.isEmpty()) {
+ return scan;
+ }
+
+ enforceColumnAccess(table, columnsToCheck);
+ return scan;
+ }
+
+ /**
+ * Traces the output columns of a RelNode back to their table-scan origins.
+ */
+ private void traceOutputColumns(RelNode node) {
+ if (node == null) {
+ return;
+ }
+ int fieldCount = node.getRowType().getFieldCount();
+ for (int i = 0; i < fieldCount; i++) {
+ traceColumnOrigin(node, i);
+ }
+ }
+
+ /**
+ * Traces a single output column of {@code node} at index {@code
columnIndex} back to
+ * table-scan origins, recording them in {@link #tableToReferencedCols}.
+ */
+ private void traceColumnOrigin(RelNode node, int columnIndex) {
+ if (node == null || mq == null) {
+ return;
+ }
+ Set<RelColumnOrigin> origins;
+ try {
+ origins = mq.getColumnOrigins(node, columnIndex);
+ } catch (Exception e) {
+ logger.debug("getColumnOrigins failed for {} column {}", node,
columnIndex, e);
+ return;
+ }
+ if (origins == null) {
+ return;
+ }
+ for (RelColumnOrigin origin : origins) {
+ RelOptTable originTable = origin.getOriginTable();
+ if (originTable != null) {
+ // Record origins for ALL table types (DrillTable, JdbcTable, etc.).
+ // Previously this only recorded DrillTable origins, which caused
+ // JDBC storage plugin tables (JdbcTable) to be skipped entirely.
+ tableToReferencedCols
+ .computeIfAbsent(originTable, k -> new HashSet<>())
+ .add(origin.getOriginColumnOrdinal());
+ }
+ }
+ }
+
+ /**
+ * Collects RexInputRefs from a list of RexNodes and traces each to its
+ * table-scan origin via the input node's metadata. Also processes any
+ * {@link RexSubQuery} found in the expressions (scalar/IN/EXISTS subqueries)
+ * so that column references inside subqueries are authorized.
+ */
+ private void collectRefs(List<RexNode> rexNodes, RelNode inputNode) {
+ if (rexNodes == null || inputNode == null) {
+ return;
+ }
+ for (RexNode rex : rexNodes) {
+ analyzeRex(rex, inputNode, -1, null);
+ }
+ }
+
+ /**
+ * Analyzes a {@link RexNode} expression, collecting {@link RexInputRef}s and
+ * {@link RexSubQuery}s, tracing each input ref to its table-scan origin and
+ * recursively visiting each subquery's {@link RelNode} tree.
+ *
+ * @param rex the expression to analyze
+ * @param inputNode the input RelNode that RexInputRefs resolve against
+ * @param leftCount if {@code >= 0}, indicates a join condition: refs with
+ * index {@code < leftCount} resolve against {@code
leftInput},
+ * others resolve against {@code inputNode} (the right
input)
+ * with offset {@code leftCount}. If {@code < 0}, all refs
+ * resolve against {@code inputNode}.
+ * @param leftInput the left input of a join, or {@code null} when
+ * {@code leftCount < 0}.
+ */
+ private void analyzeRex(RexNode rex, RelNode inputNode, int leftCount,
RelNode leftInput) {
+ if (rex == null) {
+ return;
+ }
+ Set<Integer> refs = new HashSet<>();
+ List<RexSubQuery> subQueries = new ArrayList<>();
+ List<CorrelFieldAccess> correlAccesses = new ArrayList<>();
+ rex.accept(new RexRefCollector(refs, subQueries, correlAccesses));
+ for (int refIndex : refs) {
+ if (leftCount >= 0 && refIndex < leftCount) {
+ traceColumnOrigin(leftInput, refIndex);
+ } else if (leftCount >= 0) {
+ traceColumnOrigin(inputNode, refIndex - leftCount);
+ } else {
+ traceColumnOrigin(inputNode, refIndex);
+ }
+ }
+ // Resolve correlated outer-column references. A RexFieldAccess over a
+ // RexCorrelVariable (e.g. $cor0.id inside a correlated subquery) always
+ // refers to an ENCLOSING subquery's row scope, never to inputNode itself
+ // (a reference to inputNode would be a plain RexInputRef).
+ //
+ // Registration happens inside RexRefCollector.visitFieldAccess with
+ // putIfAbsent(varName, enclosingScope), so by the time we reach here the
+ // scope for every $corN we just collected has been bound if possible.
+ // Lookup is a direct map get(); the uncommon case of an unbound variable
+ // falls back (safely over-approximating) to enclosingScope before giving
+ // up — tracing an extra scope is harmless and keeps us on the secure side
+ // of the check.
+ for (CorrelFieldAccess cfa : correlAccesses) {
+ RelNode scope = correlationScopeByVar.get(cfa.varName);
+ if (scope != null) {
+ traceColumnOrigin(scope, cfa.fieldIndex);
+ continue;
+ }
+ if (enclosingScope != null) {
+ logger.debug("Correlated reference '{}' was not pre-bound; "
+ + "falling back to current enclosingScope for a safe
over-approximation.",
+ cfa.varName);
+ traceColumnOrigin(enclosingScope, cfa.fieldIndex);
+ } else {
+ logger.warn("Unresolved correlated column reference '{}' (field index
{}): "
+ + "no enclosing correlation scope could be determined.
Column-level "
+ + "authorization may be incomplete for this reference.",
+ cfa.varName, cfa.fieldIndex);
+ }
+ }
+ for (RexSubQuery sq : subQueries) {
+ // Trace the subquery's output columns to their table-scan origins.
+ // For scalar subqueries (e.g. SELECT sum(user_id) FROM t), this traces
+ // the aggregate output back to the underlying table column.
+ traceOutputColumns(sq.rel);
+ // The subquery's $cor* variables reference the rows produced by
+ // inputNode (the input of the node whose expression contained this
+ // RexSubQuery). Make inputNode the active enclosing scope while we
+ // descend into sq.rel; save/restore the previous value so nesting and
+ // skip-level references behave correctly. Registration of new $corN
+ // names in correlationScopeByVar uses putIfAbsent, so a variable whose
+ // binding was established at a shallower scope is never overwritten.
+ RelNode prevEnclosing = enclosingScope;
+ enclosingScope = inputNode;
+ try {
+ // Recursively visit the subquery's RelNode tree so that RexInputRefs
+ // and correlated references inside the subquery are also collected
+ // and traced.
+ sq.rel.accept(this);
+ } finally {
+ enclosingScope = prevEnclosing;
+ }
+ }
+ }
+
+ /**
+ * Enforces column-level access for the given table and column set.
+ */
+ private void enforceColumnAccess(RelOptTable table, Set<String> columns) {
+ AccessAuthorizer authorizer =
AccessAuthorizerManager.getAuthorizer(drillConfig);
+ // When authorization is disabled the manager returns the NoOp authorizer,
+ // which allows all access — no enabled/disabled branching needed here.
+
+ // Resolve datasource / schema / table from the qualified name via the
+ // shared resolver, so column-level checks address exactly the same
+ // resource as the table-level checks in DrillCalciteCatalogReader. This
+ // works for ALL table types (DrillTable, JdbcTable, etc.) — previously
+ // this method required a DrillTable and skipped JdbcTable, leaving JDBC
+ // storage plugin tables without column-level authorization.
+ TableAccessResource resource =
TableAccessResource.resolve(table.getQualifiedName());
+ String userName = session.getCredentials().getUserName();
+
+ if (!authorizer.checkColumnAccess(UserIdentity.of(userName),
resource.getDataSource(),
+ resource.getSchemaPath(), resource.getTable(), columns,
AccessType.SELECT)) {
+ throw UserException.permissionError()
+ .message("Access denied: user '%s' lacks SELECT privilege on one or
more columns " +
+ "(%s) of table %s", userName, columns, resource)
+ .build(logger);
+ }
+ }
+
+ /**
+ * Holder for a correlated outer-column reference captured during Rex
+ * traversal: the variable name of the {@link RexCorrelVariable}
+ * (e.g. {@code "$cor0"}, used to do an exact lookup of the enclosing row
+ * scope via {@link #correlationScopeByVar}) and the index of the referenced
+ * field within that row scope.
+ */
+ private static final class CorrelFieldAccess {
+ final String varName;
+ final int fieldIndex;
+
+ CorrelFieldAccess(String varName, int fieldIndex) {
+ this.varName = varName;
+ this.fieldIndex = fieldIndex;
+ }
+ }
+
+ /**
+ * RexVisitor that collects all {@link RexInputRef} indices,
+ * {@link RexSubQuery} instances, and correlated outer-column references
+ * encountered in a {@link RexNode} tree. A correlated reference to an outer
+ * query column appears as a {@link RexFieldAccess} over a
+ * {@link RexCorrelVariable} (e.g. {@code $cor0.id}); such references are not
+ * {@link RexInputRef}s and would be silently skipped by a plain
+ * {@code RexInputRef}-only visitor, bypassing column-level authorization for
+ * the referenced outer column.
+ *
+ * <p>This collector is non-static so it can eagerly register each {@code
+ * $corN} in {@link #correlationScopeByVar} at the exact moment we first
+ * encounter it during Rex traversal. Registration is
+ * {@code putIfAbsent(name, enclosingScope)}, which guarantees we never
+ * overwrite a variable that was already bound at a shallower nesting level
+ * (which is exactly how skip-level references to outer variables stay
+ * correctly bound).
+ *
+ * <p>{@link RexSubQuery#accept(RexVisitor)} dispatches to
+ * {@link #visitSubQuery(RexSubQuery)} (not {@code visitCall}), so a plain
+ * {@code RexInputRef}-only visitor silently skips over subqueries. This
+ * collector overrides {@code visitSubQuery} to capture the subquery and then
+ * continues traversing its operands so that nested {@link RexInputRef}s
+ * (e.g. the left side of {@code x IN (SELECT ...)}) and nested subqueries
+ * are also collected.</p>
+ */
+ private final class RexRefCollector extends RexVisitorImpl<Void> {
+ private final Set<Integer> refs;
+ private final List<RexSubQuery> subQueries;
+ private final List<CorrelFieldAccess> correlAccesses;
+
+ RexRefCollector(Set<Integer> refs, List<RexSubQuery> subQueries,
+ List<CorrelFieldAccess> correlAccesses) {
+ super(true);
+ this.refs = refs;
+ this.subQueries = subQueries;
+ this.correlAccesses = correlAccesses;
+ }
+
+ @Override
+ public Void visitInputRef(RexInputRef ref) {
+ refs.add(ref.getIndex());
+ return null;
+ }
+
+ @Override
+ public Void visitSubQuery(RexSubQuery subQuery) {
+ subQueries.add(subQuery);
+ // Continue traversing operands (e.g. the left expression of `x IN
(...)`)
+ // to collect any RexInputRefs and nested RexSubQueries within them.
+ for (RexNode operand : subQuery.getOperands()) {
+ operand.accept(this);
+ }
+ return null;
+ }
+
+ @Override
+ public Void visitFieldAccess(RexFieldAccess fieldAccess) {
+ // A RexFieldAccess over a RexCorrelVariable is how Calcite represents a
+ // correlated reference to an outer query's column (e.g. $cor0.id inside
+ // a subquery). Such references are not RexInputRefs and would otherwise
+ // be silently skipped by this collector, bypassing column-level
+ // authorization for the referenced outer column.
+ RexNode refExpr = fieldAccess.getReferenceExpr();
+ if (refExpr instanceof RexCorrelVariable) {
+ RexCorrelVariable corVar = (RexCorrelVariable) refExpr;
+ String varName = corVar.getName();
+ // Eager, idempotent registration: putIfAbsent so the first
+ // enclosing scope we saw this variable under wins, and any later
+ // occurrences (possibly from a deeper nested scope after
+ // enclosingScope has been overwritten) do not clobber the binding.
+ if (enclosingScope != null) {
+ correlationScopeByVar.putIfAbsent(varName, enclosingScope);
Review Comment:
`putIfAbsent` binds the correlation variable to whichever scope is active
when it's first seen, which is only correct if each intermediate level also
references it.
```sql
SELECT a.pub FROM a WHERE EXISTS (
SELECT 1 FROM b WHERE EXISTS (
SELECT 1 FROM c WHERE c.k = a.secret AND c.j = b.x))
```
`$cor0` gets bound to `b`, so `a.secret`'s index is resolved against `b` and
never checked on `a`.
Fix: bind by `CorrelationId` at the node that defines it (the
`LogicalFilter`/`LogicalProject`/`LogicalJoin` whose `getVariablesSet()`
contains it, or the `LogicalCorrelate`), not on first use. Worth adding this
query as a test.
##########
auth/drill-ranger-plugin/src/main/java/org/apache/ranger/authorization/drill/authorizer/DrillAccessControl.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file distributed
with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ranger.authorization.drill.authorizer;
+
+
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.ranger.authorization.drill.resource.DrillAccessType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Drill-facing authorization facade: implementation of the Drill
+ * {@link AccessAuthorizer} SPI backed by Ranger.
+ *
+ * <p>The class is instantiated once by the shim (drill-ranger-plugin-shim)
+ * via {@code DrillAccessControl(String serviceName)} — the constructor
+ * completes all initialization (or throws, fail-closed), mirroring Presto's
+ * {@code RangerPrestoAccessControl}. All access checks are instance methods
+ * invoked directly through the SPI interface.</p>
+ *
+ * <p>Group resolution: if the engine-supplied {@link UserIdentity} carries
+ * groups (resolved at authentication time), they are used as-is; otherwise
+ * the groups are resolved via Hadoop UGI ({@link #getUserGroups}).</p>
+ *
+ * <p>On Ranger evaluation error, checks return {@code false} (fail-closed).
+ * The service name identifies the Ranger service instance whose policies
+ * are evaluated.</p>
+ */
+public class DrillAccessControl implements AccessAuthorizer {
+
+ private static final Logger logger =
LoggerFactory.getLogger(DrillAccessControl.class);
+
+ private final DrillAuthorizer authorizer;
+
+ // Set of system schemas that bypass authorization (information_schema, sys,
etc.)
+ // Stored in uppercase; isSystemSchema() uppercases input before lookup so
the
+ // bypass is case-insensitive (e.g. "information_schema",
"INFORMATION_SCHEMA",
+ // "Sys", "SYS" all match).
+ private static final Set<String> SYSTEM_SCHEMAS = new
HashSet<>(Arrays.asList(
+ "INFORMATION_SCHEMA", "SYS"
+ ));
+
+ /**
+ * Creates and initializes the Ranger Drill plugin. Completes all
+ * initialization or throws (fail-closed) — the caller (the shim) must not
+ * receive a half-initialized authorizer.
+ *
+ * @param serviceName the Ranger service instance name (must match a service
created in Ranger Admin)
+ */
+ public DrillAccessControl(String serviceName) {
+ logger.info("Initializing Ranger Drill authorization plugin for service:
{}", serviceName);
+ try {
+ this.authorizer = new DrillAuthorizer(serviceName);
+ logger.info("Ranger Drill authorization plugin initialized
successfully");
+ } catch (Exception e) {
+ logger.error("Failed to initialize Ranger Drill plugin for service {}",
serviceName, e);
+ throw new RuntimeException(
+ "Failed to initialize Ranger Drill plugin — authorization disabled "
+ serviceName
+ + " with exception: " + e);
+ }
+ }
+
+ /**
+ * Package-private constructor for unit tests: injects a (mock) authorizer
+ * directly, bypassing Ranger Admin connectivity.
+ */
+ DrillAccessControl(DrillAuthorizer authorizer) {
+ this.authorizer = authorizer;
+ }
+
+ /**
+ * Resolves the OS-level groups for a given user via Hadoop UGI. Used only
+ * as a fallback when the engine-supplied {@link UserIdentity} carries no
+ * groups.
+ *
+ * @param user the username
+ * @return a set of group names (never null, empty on failure)
+ */
+ public static Set<String> getUserGroups(String user) {
+ if (user == null || user.trim().isEmpty()) {
+ return Collections.emptySet();
+ }
+ try {
+ UserGroupInformation ugi = UserGroupInformation.createRemoteUser(user);
+ String[] groups = ugi.getGroupNames();
+ return groups == null ? Collections.emptySet() : new
HashSet<>(Arrays.asList(groups));
+ } catch (Exception e) {
+ logger.warn("Failed to determine groups for user={}", user, e);
+ return Collections.emptySet();
+ }
+ }
+
+ /**
+ * Checks table-level access. The SPI {@link AccessType} is mapped to
+ * {@link DrillAccessType} by name; if the mapping fails (e.g. the SPI
+ * added a type this plugin does not know yet) access is denied
+ * (fail-closed).
+ *
+ * <p>System schemas ({@code INFORMATION_SCHEMA}, {@code sys}) bypass
+ * authorization. A null/empty schema is invalid input and fails closed
+ * rather than silently bypassing Ranger.</p>
+ *
+ * @param user the querying user identity
+ * @param dataSource the Drill storage plugin name (e.g. "dfs", "hbase")
+ * @param schema the schema path (e.g. "dfs.tmp")
+ * @param table the table name
+ * @param accessType the access type (e.g. SELECT, CREATE)
+ * @return {@code true} if access is allowed
+ */
+ @Override
+ public boolean checkTableAccess(UserIdentity user, String dataSource, String
schema,
+ String table, AccessType accessType) {
+ DrillAccessType operator = parseAccessType(accessType, user, schema,
table);
+ if (operator == null) {
+ return false;
+ }
+ Boolean bypass = systemSchemaBypass(user, schema, table);
+ if (bypass != null) {
+ return bypass;
+ }
+ try {
+ return authorizer.checkTableAccess(resolveIdentity(user), dataSource,
schema, table, operator);
+ } catch (Exception e) {
+ logger.error("Checking table access for user={}, schema={}, table={}.
with exception:{}",
+ user.getUser(), schema, table, e.toString());
+ return false; // fail-closed on error
+ }
+ }
+
+ /**
+ * Checks column-level access for a set of columns. Returns {@code true}
+ * only if the user has the specified access type on ALL given columns.
+ *
+ * <p>System schemas ({@code INFORMATION_SCHEMA}, {@code sys}) bypass
+ * authorization. A null/empty schema is invalid input and fails closed
+ * rather than silently bypassing Ranger.</p>
+ *
+ * @param user the querying user identity
+ * @param dataSource the Drill storage plugin name
+ * @param schema the schema path
+ * @param table the table name
+ * @param columns the set of column names to check
+ * @param accessType the access type (e.g. SELECT)
+ * @return {@code true} if access is allowed for every column
+ */
+ @Override
+ public boolean checkColumnAccess(UserIdentity user, String dataSource,
String schema,
+ String table, Set<String> columns, AccessType accessType) {
+ DrillAccessType operator = parseAccessType(accessType, user, schema,
table);
+ if (operator == null) {
+ return false;
+ }
+ Boolean bypass = systemSchemaBypass(user, schema, table);
+ if (bypass != null) {
+ return bypass;
+ }
+ try {
+ return authorizer.checkColumnAccess(resolveIdentity(user), dataSource,
schema, table,
+ columns, operator);
+ } catch (Exception e) {
+ logger.error("Error checking column access for user={}, schema={},
table={}",
+ user.getUser(), schema, table, e);
+ return false; // fail-closed on error
+ }
+ }
+
+ /**
+ * Maps the SPI {@link AccessType} to a {@link DrillAccessType} by name.
+ * Returns {@code null} (and logs) when the SPI enum carries a type this
+ * plugin does not know yet — callers deny access in that case
+ * (fail-closed). This guards against drift when a newer SPI adds access
+ * types before the Ranger service-def does.
+ */
+ private DrillAccessType parseAccessType(AccessType accessType, UserIdentity
user,
+ String schema, String table) {
+ try {
+ return DrillAccessType.valueOf(accessType.name());
+ } catch (Exception e) {
+ logger.error("Unsupported access type '{}', denied access for user={},
schema={}, table={}",
+ accessType, user.getUser(), schema, table);
+ return null;
+ }
+ }
+
+ /**
+ * Resolves the effective identity for an access check: engine-supplied
+ * groups are used as-is; when the identity carries none (e.g. mount points
+ * that only have the authenticated user name), the groups are resolved via
+ * Hadoop UGI ({@link #getUserGroups}).
+ */
+ private UserIdentity resolveIdentity(UserIdentity user) {
+ Set<String> groups = user.getGroups();
+ if (groups != null && !groups.isEmpty()) {
+ return user;
+ }
+ return UserIdentity.builder()
+ .setUser(user.getUser())
+ .setGroups(getUserGroups(user.getUser()))
+ .build();
+ }
+
+ /**
+ * Returns whether the given schema is a system schema that should bypass
+ * authorization.
+ *
+ * <p>Comparison is case-insensitive so that SQL like
+ * {@code SELECT * FROM information_schema.tables} (lowercase) or
+ * {@code SELECT * FROM SYS.DRILLBITS} (uppercase) both bypass authorization,
+ * matching Drill's own case-insensitive schema resolution.
+ *
+ * <p>For compound schema paths like {@code dfs.tmp}, only the top-level
segment
+ * (the storage plugin name) is checked — that is intentional, because system
+ * schemas ({@code INFORMATION_SCHEMA}, {@code sys}) are always top-level.
+ */
+ private static boolean isSystemSchema(String schema) {
+ if (schema == null || schema.trim().isEmpty()) {
+ throw new IllegalArgumentException(
+ "Schema must not be null or empty for authorization check; refusing
to treat as system schema");
+ }
+ // Use only the top-level segment of a compound schema path
+ // (e.g. "dfs.tmp" -> "dfs", "INFORMATION_SCHEMA" -> "INFORMATION_SCHEMA")
+ String topLevel = schema;
+ int dot = schema.indexOf('.');
+ if (dot > 0) {
+ topLevel = schema.substring(0, dot);
+ }
+ return SYSTEM_SCHEMAS.contains(topLevel.toUpperCase());
Review Comment:
The schema passed here has the datasource stripped (`TableAccessResource`),
so `mysql.sys.*`, `pg.information_schema.*`, or a dfs workspace named `sys` all
bypass Ranger. MySQL ships a `sys` database by default.
Fix: pass the datasource into the bypass and match on it
(`SYSTEM_SCHEMAS.contains(dataSource.toUpperCase())`), not the schema.
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/ColumnAccessChecker.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.conversion;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelShuttleImpl;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexSubQuery;
+import org.apache.calcite.rex.RexVisitor;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.security.AccessAuthorizerManager;
+import org.apache.drill.exec.security.TableAccessResource;
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Visitor that traverses a RelNode tree and enforces column-level SELECT
authorization
+ * via the configured {@link AccessAuthorizer} (Ranger by default).
+ *
+ * <p><b>Design:</b> For every RelNode that carries {@link RexNode} expressions
+ * (Project, Filter, Join, Aggregate, Sort), the visitor collects all {@link
RexInputRef}s
+ * and uses Calcite's {@link RelMetadataQuery#getColumnOrigins(RelNode, int)}
to trace each
+ * referenced column back to its originating {@link TableScan} column index.
This correctly
+ * handles multi-hop projections, filters, joins, and aggregations.</p>
+ *
+ * <p>For a {@link TableScan} that has NO traced column references (e.g.
+ * {@code SELECT * FROM t} with no intervening Project), ALL columns of that
table
+ * are checked.</p>
+ *
+ * <p>Correlated subqueries are handled as well: a reference to an outer query
+ * column from inside a correlated subquery is represented by Calcite as a
+ * {@link RexFieldAccess} over a {@link RexCorrelVariable} (e.g. {@code
$cor0.id}),
+ * not as a {@link RexInputRef}. Each correlation variable carries a unique
+ * name {@code $corN} assigned by {@code SqlToRelConverter}; we keep an exact
+ * map from variable name to the enclosing {@link RelNode} row scope it refers
+ * to. The map entry is populated the FIRST time we see a given {@code $corN}
+ * during Rex traversal: we use {@code putIfAbsent(name, enclosingScope)} so
+ * that subsequent encounters (including cross-nested skip-level references
+ * from deeper subqueries to an outer {@code $cor0}) never overwrite the
+ * original, correct binding. This avoids the ambiguity of row-type-based
+ * heuristics: multiple enclosing scopes with identical schemas no longer
+ * cause false positives or missed detections, and no scope stack is needed —
+ * a single field tracks the innermost enclosing scope and is saved/restored
+ * around each subquery entry via try/finally.</p>
+ *
+ * <p>System schemas (INFORMATION_SCHEMA, sys) are bypassed inside the
authorizer
+ * implementation. When authorization is disabled, the visitor is a no-op
(fail-open).</p>
+ */
+class ColumnAccessChecker extends RelShuttleImpl {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ColumnAccessChecker.class);
+
+ private final UserSession session;
+ private final DrillConfig drillConfig;
+ private final RelMetadataQuery mq;
+
+ // Records each table's referenced column indices. Uses IdentityHashMap
because
+ // RelOptTable equals/hashCode may be expensive or not identity-based.
+ private final Map<RelOptTable, Set<Integer>> tableToReferencedCols = new
IdentityHashMap<>();
+
+ // The innermost enclosing row scope for correlated references. When we are
+ // about to descend into a RexSubQuery's Rel tree, we save the previous value
+ // and set this field to the outer-query inputNode that the subquery's
+ // $cor* variables are relative to; the saved value is restored in a finally
+ // block. Because variable → scope registration uses putIfAbsent in the Rex
+ // collector, a single field is sufficient even for deep nesting: skip-level
+ // references to an outer $corN reuse the binding that was established the
+ // first time that variable was seen (at a shallower enclosing scope).
+ private RelNode enclosingScope;
+
+ // Exact, name-keyed scope mapping: $cor0 → outer row scope, $cor1 → outer
+ // row scope, etc. Populated with putIfAbsent during Rex traversal so the
+ // first occurrence of each $corN establishes a permanent binding. Lookup is
+ // pure get(); a miss falls back (safely, in the over-approximate direction)
+ // to enclosingScope and emits a warning so unusual plans remain diagnosable.
+ private final Map<String, RelNode> correlationScopeByVar = new HashMap<>();
+
+ ColumnAccessChecker(UserSession session, DrillConfig drillConfig,
RelMetadataQuery mq) {
+ this.session = session;
+ this.drillConfig = drillConfig;
+ this.mq = mq;
+ }
+
+ /**
+ * Entry point: traverse the tree and enforce column-level access.
+ */
+ void check(RelNode root) {
+ // Also trace the root node's output columns (covers bare TableScan root or
+ // top-level Project output).
+ traceOutputColumns(root);
+
+ // Walk the tree to collect RexInputRef origins from all
expression-bearing nodes.
+ root.accept(this);
+ }
+
+ // ------------------------------------------------------------------
+ // RelShuttle overrides — collect RexInputRefs from expression-bearing nodes
+ // ------------------------------------------------------------------
+
+ @Override
+ public RelNode visit(LogicalProject project) {
+ collectRefs(project.getProjects(), project.getInput());
+ return super.visit(project);
+ }
+
+ @Override
+ public RelNode visit(LogicalFilter filter) {
+ if (filter.getCondition() != null) {
+ analyzeRex(filter.getCondition(), filter.getInput(), -1, null);
+ }
+ return super.visit(filter);
+ }
+
+ @Override
+ public RelNode visit(LogicalJoin join) {
+ if (join.getCondition() != null) {
+ int leftCount = join.getLeft().getRowType().getFieldCount();
+ analyzeRex(join.getCondition(), join.getRight(), leftCount,
join.getLeft());
+ }
+ return super.visit(join);
+ }
+
+ @Override
+ public RelNode visit(LogicalAggregate aggregate) {
+ for (int i : aggregate.getGroupSet()) {
+ traceColumnOrigin(aggregate.getInput(), i);
+ }
+ // AggregateCall arguments (the x in SUM(x)), FILTER (WHERE ...) columns,
+ // and WITHIN GROUP collation columns are not covered by getGroupSet().
+ // Trace them explicitly rather than relying on an implicit Project beneath
+ // the aggregate (plan shape can change across Calcite upgrades).
+ for (AggregateCall call : aggregate.getAggCallList()) {
+ for (int arg : call.getArgList()) {
+ traceColumnOrigin(aggregate.getInput(), arg);
+ }
+ if (call.filterArg >= 0) {
+ traceColumnOrigin(aggregate.getInput(), call.filterArg);
+ }
+ if (call.getCollation() != null) {
+ for (RelFieldCollation fc : call.getCollation().getFieldCollations()) {
+ traceColumnOrigin(aggregate.getInput(), fc.getFieldIndex());
+ }
+ }
+ }
+ return super.visit(aggregate);
+ }
+
+ @Override
+ public RelNode visit(LogicalSort sort) {
+ if (sort.getCollation() != null) {
+ sort.getCollation().getFieldCollations().forEach(fc ->
+ traceColumnOrigin(sort.getInput(), fc.getFieldIndex()));
+ }
+ // LIMIT/OFFSET are RexNode expressions that may carry RexSubQuery or
+ // RexInputRef (e.g. LIMIT (SELECT MAX(amount) FROM ...)). analyzeRex
+ // descends into them; without it, subqueries reachable only from Sort
+ // would never be column-checked.
+ if (sort.fetch != null) {
+ analyzeRex(sort.fetch, sort.getInput(), -1, null);
+ }
+ if (sort.offset != null) {
+ analyzeRex(sort.offset, sort.getInput(), -1, null);
+ }
+ return super.visit(sort);
+ }
+
+ @Override
+ public RelNode visit(TableScan scan) {
+ RelOptTable table = scan.getTable();
+
+ // Determine which columns to check: traced columns, or ALL if none were
traced
+ // (SELECT * FROM t case).
+ Set<Integer> referencedColIndices = tableToReferencedCols.get(table);
+ List<String> allColumnNames = scan.getRowType().getFieldNames();
+
+ Set<String> columnsToCheck;
+ if (referencedColIndices == null || referencedColIndices.isEmpty()) {
+ // SELECT * — check all columns
+ columnsToCheck = new HashSet<>(allColumnNames);
+ } else {
+ columnsToCheck = new HashSet<>();
+ for (int idx : referencedColIndices) {
+ if (idx >= 0 && idx < allColumnNames.size()) {
+ columnsToCheck.add(allColumnNames.get(idx));
+ }
+ }
+ }
+
+ if (columnsToCheck.isEmpty()) {
+ return scan;
+ }
+
+ enforceColumnAccess(table, columnsToCheck);
+ return scan;
+ }
+
+ /**
+ * Traces the output columns of a RelNode back to their table-scan origins.
+ */
+ private void traceOutputColumns(RelNode node) {
+ if (node == null) {
+ return;
+ }
+ int fieldCount = node.getRowType().getFieldCount();
+ for (int i = 0; i < fieldCount; i++) {
+ traceColumnOrigin(node, i);
+ }
+ }
+
+ /**
+ * Traces a single output column of {@code node} at index {@code
columnIndex} back to
+ * table-scan origins, recording them in {@link #tableToReferencedCols}.
+ */
+ private void traceColumnOrigin(RelNode node, int columnIndex) {
+ if (node == null || mq == null) {
+ return;
+ }
+ Set<RelColumnOrigin> origins;
+ try {
+ origins = mq.getColumnOrigins(node, columnIndex);
+ } catch (Exception e) {
+ logger.debug("getColumnOrigins failed for {} column {}", node,
columnIndex, e);
+ return;
+ }
+ if (origins == null) {
+ return;
+ }
+ for (RelColumnOrigin origin : origins) {
+ RelOptTable originTable = origin.getOriginTable();
+ if (originTable != null) {
+ // Record origins for ALL table types (DrillTable, JdbcTable, etc.).
+ // Previously this only recorded DrillTable origins, which caused
+ // JDBC storage plugin tables (JdbcTable) to be skipped entirely.
+ tableToReferencedCols
+ .computeIfAbsent(originTable, k -> new HashSet<>())
+ .add(origin.getOriginColumnOrdinal());
+ }
+ }
+ }
+
+ /**
+ * Collects RexInputRefs from a list of RexNodes and traces each to its
+ * table-scan origin via the input node's metadata. Also processes any
+ * {@link RexSubQuery} found in the expressions (scalar/IN/EXISTS subqueries)
+ * so that column references inside subqueries are authorized.
+ */
+ private void collectRefs(List<RexNode> rexNodes, RelNode inputNode) {
+ if (rexNodes == null || inputNode == null) {
+ return;
+ }
+ for (RexNode rex : rexNodes) {
+ analyzeRex(rex, inputNode, -1, null);
+ }
+ }
+
+ /**
+ * Analyzes a {@link RexNode} expression, collecting {@link RexInputRef}s and
+ * {@link RexSubQuery}s, tracing each input ref to its table-scan origin and
+ * recursively visiting each subquery's {@link RelNode} tree.
+ *
+ * @param rex the expression to analyze
+ * @param inputNode the input RelNode that RexInputRefs resolve against
+ * @param leftCount if {@code >= 0}, indicates a join condition: refs with
+ * index {@code < leftCount} resolve against {@code
leftInput},
+ * others resolve against {@code inputNode} (the right
input)
+ * with offset {@code leftCount}. If {@code < 0}, all refs
+ * resolve against {@code inputNode}.
+ * @param leftInput the left input of a join, or {@code null} when
+ * {@code leftCount < 0}.
+ */
+ private void analyzeRex(RexNode rex, RelNode inputNode, int leftCount,
RelNode leftInput) {
+ if (rex == null) {
+ return;
+ }
+ Set<Integer> refs = new HashSet<>();
+ List<RexSubQuery> subQueries = new ArrayList<>();
+ List<CorrelFieldAccess> correlAccesses = new ArrayList<>();
+ rex.accept(new RexRefCollector(refs, subQueries, correlAccesses));
+ for (int refIndex : refs) {
+ if (leftCount >= 0 && refIndex < leftCount) {
+ traceColumnOrigin(leftInput, refIndex);
+ } else if (leftCount >= 0) {
+ traceColumnOrigin(inputNode, refIndex - leftCount);
+ } else {
+ traceColumnOrigin(inputNode, refIndex);
+ }
+ }
+ // Resolve correlated outer-column references. A RexFieldAccess over a
+ // RexCorrelVariable (e.g. $cor0.id inside a correlated subquery) always
+ // refers to an ENCLOSING subquery's row scope, never to inputNode itself
+ // (a reference to inputNode would be a plain RexInputRef).
+ //
+ // Registration happens inside RexRefCollector.visitFieldAccess with
+ // putIfAbsent(varName, enclosingScope), so by the time we reach here the
+ // scope for every $corN we just collected has been bound if possible.
+ // Lookup is a direct map get(); the uncommon case of an unbound variable
+ // falls back (safely over-approximating) to enclosingScope before giving
+ // up — tracing an extra scope is harmless and keeps us on the secure side
+ // of the check.
+ for (CorrelFieldAccess cfa : correlAccesses) {
+ RelNode scope = correlationScopeByVar.get(cfa.varName);
+ if (scope != null) {
+ traceColumnOrigin(scope, cfa.fieldIndex);
+ continue;
+ }
+ if (enclosingScope != null) {
+ logger.debug("Correlated reference '{}' was not pre-bound; "
+ + "falling back to current enclosingScope for a safe
over-approximation.",
+ cfa.varName);
+ traceColumnOrigin(enclosingScope, cfa.fieldIndex);
+ } else {
+ logger.warn("Unresolved correlated column reference '{}' (field index
{}): "
Review Comment:
`LogicalCorrelate` (LATERAL/UNNEST) isn't handled, so these references land
here and are only logged.
```sql
SELECT u.c FROM (SELECT pub, secret_arr FROM t WHERE pub > 0) x,
UNNEST(x.secret_arr) u(c)
```
returns `secret_arr` contents unchecked.
Fix: add a `visit(LogicalCorrelate)` that binds its `CorrelationId` to the
left input, and fail closed (throw) here instead of warning.
> Integrate Apache Ranger authorization for Drill
> -----------------------------------------------
>
> Key: DRILL-8548
> URL: https://issues.apache.org/jira/browse/DRILL-8548
> Project: Apache Drill
> Issue Type: New Feature
> Components: Server
> Affects Versions: 1.23.0
> Reporter: shihuafeng
> Priority: Major
> Fix For: 1.23.0
>
>
> This issue introduces Apache Ranger as a pluggable authorization framework
> for Drill, enabling centralized table-level and column-level access control
> for Drill queries. It is a substantial feature spanning three layers: a new
> drill-ranger module , integration hooks in exec/java-exec, and distribution
> packaging. The design follows Drill's existing AccessAuthorizer SPI and
> Calcite's RelShuttle mechanism so that column-level checks happen in the
> toRel phase before physical planning
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
