cgivre commented on code in PR #3056:
URL: https://github.com/apache/drill/pull/3056#discussion_r4185239588
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/SqlConverter.java:
##########
@@ -245,6 +246,19 @@ public RelRoot toRel(final SqlNode validatedNode) {
RelNode project = LogicalProject.create(rel.rel,
Collections.emptyList(), expressions, rel.validatedRowType);
rel = RelRoot.of(project, rel.validatedRowType, rel.kind);
}
+
+ // Column-level SELECT authorization check. Done after SqlToRelConverter
has
+ // resolved all column references (so we can trace each to its TableScan)
+ // and before flattenTypes/optimization (so column references are intact).
+ //
+ // Skip the whole check when authorization is disabled: the tree walk plus
+ // a getColumnOrigins() metadata query per output column is measurable
+ // planning cost that deployments not using the feature must not pay on
+ // every query.
+ if (AccessAuthorizerManager.isEnabled(drillConfig)) {
+ new ColumnAccessChecker(session, drillConfig,
cluster.getMetadataQuery()).check(rel.rel);
Review Comment:
`DrillViewExpander` calls `toRel` on the view SQL, so this also runs during
view expansion and checks every column the view outputs. With `v = SELECT pub,
secret FROM t`, `SELECT pub FROM v` is denied, which breaks views as a
column-hiding mechanism.
Fix: skip the check when `toRel` is invoked for view expansion and run it
once on the top-level plan only.
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/ColumnAccessChecker.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.conversion;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelShuttleImpl;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexSubQuery;
+import org.apache.calcite.rex.RexVisitor;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.security.AccessAuthorizerManager;
+import org.apache.drill.exec.security.TableAccessResource;
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Visitor that traverses a RelNode tree and enforces column-level SELECT
authorization
+ * via the configured {@link AccessAuthorizer} (Ranger by default).
+ *
+ * <p><b>Design:</b> For every RelNode that carries {@link RexNode} expressions
+ * (Project, Filter, Join, Aggregate, Sort), the visitor collects all {@link
RexInputRef}s
+ * and uses Calcite's {@link RelMetadataQuery#getColumnOrigins(RelNode, int)}
to trace each
+ * referenced column back to its originating {@link TableScan} column index.
This correctly
+ * handles multi-hop projections, filters, joins, and aggregations.</p>
+ *
+ * <p>For a {@link TableScan} that has NO traced column references (e.g.
+ * {@code SELECT * FROM t} with no intervening Project), ALL columns of that
table
+ * are checked.</p>
+ *
+ * <p>Correlated subqueries are handled as well: a reference to an outer query
+ * column from inside a correlated subquery is represented by Calcite as a
+ * {@link RexFieldAccess} over a {@link RexCorrelVariable} (e.g. {@code
$cor0.id}),
+ * not as a {@link RexInputRef}. Each correlation variable carries a unique
+ * name {@code $corN} assigned by {@code SqlToRelConverter}; we keep an exact
+ * map from variable name to the enclosing {@link RelNode} row scope it refers
+ * to. The map entry is populated the FIRST time we see a given {@code $corN}
+ * during Rex traversal: we use {@code putIfAbsent(name, enclosingScope)} so
+ * that subsequent encounters (including cross-nested skip-level references
+ * from deeper subqueries to an outer {@code $cor0}) never overwrite the
+ * original, correct binding. This avoids the ambiguity of row-type-based
+ * heuristics: multiple enclosing scopes with identical schemas no longer
+ * cause false positives or missed detections, and no scope stack is needed —
+ * a single field tracks the innermost enclosing scope and is saved/restored
+ * around each subquery entry via try/finally.</p>
+ *
+ * <p>System schemas (INFORMATION_SCHEMA, sys) are bypassed inside the
authorizer
+ * implementation. When authorization is disabled, the visitor is a no-op
(fail-open).</p>
+ */
+class ColumnAccessChecker extends RelShuttleImpl {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ColumnAccessChecker.class);
+
+ private final UserSession session;
+ private final DrillConfig drillConfig;
+ private final RelMetadataQuery mq;
+
+ // Records each table's referenced column indices. Uses IdentityHashMap
because
+ // RelOptTable equals/hashCode may be expensive or not identity-based.
+ private final Map<RelOptTable, Set<Integer>> tableToReferencedCols = new
IdentityHashMap<>();
+
+ // The innermost enclosing row scope for correlated references. When we are
+ // about to descend into a RexSubQuery's Rel tree, we save the previous value
+ // and set this field to the outer-query inputNode that the subquery's
+ // $cor* variables are relative to; the saved value is restored in a finally
+ // block. Because variable → scope registration uses putIfAbsent in the Rex
+ // collector, a single field is sufficient even for deep nesting: skip-level
+ // references to an outer $corN reuse the binding that was established the
+ // first time that variable was seen (at a shallower enclosing scope).
+ private RelNode enclosingScope;
+
+ // Exact, name-keyed scope mapping: $cor0 → outer row scope, $cor1 → outer
+ // row scope, etc. Populated with putIfAbsent during Rex traversal so the
+ // first occurrence of each $corN establishes a permanent binding. Lookup is
+ // pure get(); a miss falls back (safely, in the over-approximate direction)
+ // to enclosingScope and emits a warning so unusual plans remain diagnosable.
+ private final Map<String, RelNode> correlationScopeByVar = new HashMap<>();
+
+ ColumnAccessChecker(UserSession session, DrillConfig drillConfig,
RelMetadataQuery mq) {
+ this.session = session;
+ this.drillConfig = drillConfig;
+ this.mq = mq;
+ }
+
+ /**
+ * Entry point: traverse the tree and enforce column-level access.
+ */
+ void check(RelNode root) {
+ // Also trace the root node's output columns (covers bare TableScan root or
+ // top-level Project output).
+ traceOutputColumns(root);
+
+ // Walk the tree to collect RexInputRef origins from all
expression-bearing nodes.
+ root.accept(this);
+ }
+
+ // ------------------------------------------------------------------
+ // RelShuttle overrides — collect RexInputRefs from expression-bearing nodes
+ // ------------------------------------------------------------------
+
+ @Override
+ public RelNode visit(LogicalProject project) {
+ collectRefs(project.getProjects(), project.getInput());
+ return super.visit(project);
+ }
+
+ @Override
+ public RelNode visit(LogicalFilter filter) {
+ if (filter.getCondition() != null) {
+ analyzeRex(filter.getCondition(), filter.getInput(), -1, null);
+ }
+ return super.visit(filter);
+ }
+
+ @Override
+ public RelNode visit(LogicalJoin join) {
+ if (join.getCondition() != null) {
+ int leftCount = join.getLeft().getRowType().getFieldCount();
+ analyzeRex(join.getCondition(), join.getRight(), leftCount,
join.getLeft());
Review Comment:
Subqueries in the join condition get only the right input as their
correlation scope, but their correlated field indexes are into the combined
left+right row. For `a JOIN b ON a.x IN (SELECT y FROM c WHERE c.z =
a.secret)`, `a.secret` is resolved against `b`.
Fix: pass the join itself as the correlation scope so indexes resolve across
both inputs.
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/ColumnAccessChecker.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.conversion;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelShuttleImpl;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexSubQuery;
+import org.apache.calcite.rex.RexVisitor;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.security.AccessAuthorizerManager;
+import org.apache.drill.exec.security.TableAccessResource;
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Visitor that traverses a RelNode tree and enforces column-level SELECT
authorization
+ * via the configured {@link AccessAuthorizer} (Ranger by default).
+ *
+ * <p><b>Design:</b> For every RelNode that carries {@link RexNode} expressions
+ * (Project, Filter, Join, Aggregate, Sort), the visitor collects all {@link
RexInputRef}s
+ * and uses Calcite's {@link RelMetadataQuery#getColumnOrigins(RelNode, int)}
to trace each
+ * referenced column back to its originating {@link TableScan} column index.
This correctly
+ * handles multi-hop projections, filters, joins, and aggregations.</p>
+ *
+ * <p>For a {@link TableScan} that has NO traced column references (e.g.
+ * {@code SELECT * FROM t} with no intervening Project), ALL columns of that
table
+ * are checked.</p>
+ *
+ * <p>Correlated subqueries are handled as well: a reference to an outer query
+ * column from inside a correlated subquery is represented by Calcite as a
+ * {@link RexFieldAccess} over a {@link RexCorrelVariable} (e.g. {@code
$cor0.id}),
+ * not as a {@link RexInputRef}. Each correlation variable carries a unique
+ * name {@code $corN} assigned by {@code SqlToRelConverter}; we keep an exact
+ * map from variable name to the enclosing {@link RelNode} row scope it refers
+ * to. The map entry is populated the FIRST time we see a given {@code $corN}
+ * during Rex traversal: we use {@code putIfAbsent(name, enclosingScope)} so
+ * that subsequent encounters (including cross-nested skip-level references
+ * from deeper subqueries to an outer {@code $cor0}) never overwrite the
+ * original, correct binding. This avoids the ambiguity of row-type-based
+ * heuristics: multiple enclosing scopes with identical schemas no longer
+ * cause false positives or missed detections, and no scope stack is needed —
+ * a single field tracks the innermost enclosing scope and is saved/restored
+ * around each subquery entry via try/finally.</p>
+ *
+ * <p>System schemas (INFORMATION_SCHEMA, sys) are bypassed inside the
authorizer
+ * implementation. When authorization is disabled, the visitor is a no-op
(fail-open).</p>
+ */
+class ColumnAccessChecker extends RelShuttleImpl {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ColumnAccessChecker.class);
+
+ private final UserSession session;
+ private final DrillConfig drillConfig;
+ private final RelMetadataQuery mq;
+
+ // Records each table's referenced column indices. Uses IdentityHashMap
because
+ // RelOptTable equals/hashCode may be expensive or not identity-based.
+ private final Map<RelOptTable, Set<Integer>> tableToReferencedCols = new
IdentityHashMap<>();
+
+ // The innermost enclosing row scope for correlated references. When we are
+ // about to descend into a RexSubQuery's Rel tree, we save the previous value
+ // and set this field to the outer-query inputNode that the subquery's
+ // $cor* variables are relative to; the saved value is restored in a finally
+ // block. Because variable → scope registration uses putIfAbsent in the Rex
+ // collector, a single field is sufficient even for deep nesting: skip-level
+ // references to an outer $corN reuse the binding that was established the
+ // first time that variable was seen (at a shallower enclosing scope).
+ private RelNode enclosingScope;
+
+ // Exact, name-keyed scope mapping: $cor0 → outer row scope, $cor1 → outer
+ // row scope, etc. Populated with putIfAbsent during Rex traversal so the
+ // first occurrence of each $corN establishes a permanent binding. Lookup is
+ // pure get(); a miss falls back (safely, in the over-approximate direction)
+ // to enclosingScope and emits a warning so unusual plans remain diagnosable.
+ private final Map<String, RelNode> correlationScopeByVar = new HashMap<>();
+
+ ColumnAccessChecker(UserSession session, DrillConfig drillConfig,
RelMetadataQuery mq) {
+ this.session = session;
+ this.drillConfig = drillConfig;
+ this.mq = mq;
+ }
+
+ /**
+ * Entry point: traverse the tree and enforce column-level access.
+ */
+ void check(RelNode root) {
+ // Also trace the root node's output columns (covers bare TableScan root or
+ // top-level Project output).
+ traceOutputColumns(root);
+
+ // Walk the tree to collect RexInputRef origins from all
expression-bearing nodes.
+ root.accept(this);
+ }
+
+ // ------------------------------------------------------------------
+ // RelShuttle overrides — collect RexInputRefs from expression-bearing nodes
+ // ------------------------------------------------------------------
+
+ @Override
+ public RelNode visit(LogicalProject project) {
+ collectRefs(project.getProjects(), project.getInput());
+ return super.visit(project);
+ }
+
+ @Override
+ public RelNode visit(LogicalFilter filter) {
+ if (filter.getCondition() != null) {
+ analyzeRex(filter.getCondition(), filter.getInput(), -1, null);
+ }
+ return super.visit(filter);
+ }
+
+ @Override
+ public RelNode visit(LogicalJoin join) {
+ if (join.getCondition() != null) {
+ int leftCount = join.getLeft().getRowType().getFieldCount();
+ analyzeRex(join.getCondition(), join.getRight(), leftCount,
join.getLeft());
+ }
+ return super.visit(join);
+ }
+
+ @Override
+ public RelNode visit(LogicalAggregate aggregate) {
+ for (int i : aggregate.getGroupSet()) {
+ traceColumnOrigin(aggregate.getInput(), i);
+ }
+ // AggregateCall arguments (the x in SUM(x)), FILTER (WHERE ...) columns,
+ // and WITHIN GROUP collation columns are not covered by getGroupSet().
+ // Trace them explicitly rather than relying on an implicit Project beneath
+ // the aggregate (plan shape can change across Calcite upgrades).
+ for (AggregateCall call : aggregate.getAggCallList()) {
+ for (int arg : call.getArgList()) {
+ traceColumnOrigin(aggregate.getInput(), arg);
+ }
+ if (call.filterArg >= 0) {
+ traceColumnOrigin(aggregate.getInput(), call.filterArg);
+ }
+ if (call.getCollation() != null) {
+ for (RelFieldCollation fc : call.getCollation().getFieldCollations()) {
+ traceColumnOrigin(aggregate.getInput(), fc.getFieldIndex());
+ }
+ }
+ }
+ return super.visit(aggregate);
+ }
+
+ @Override
+ public RelNode visit(LogicalSort sort) {
+ if (sort.getCollation() != null) {
+ sort.getCollation().getFieldCollations().forEach(fc ->
+ traceColumnOrigin(sort.getInput(), fc.getFieldIndex()));
+ }
+ // LIMIT/OFFSET are RexNode expressions that may carry RexSubQuery or
+ // RexInputRef (e.g. LIMIT (SELECT MAX(amount) FROM ...)). analyzeRex
+ // descends into them; without it, subqueries reachable only from Sort
+ // would never be column-checked.
+ if (sort.fetch != null) {
+ analyzeRex(sort.fetch, sort.getInput(), -1, null);
+ }
+ if (sort.offset != null) {
+ analyzeRex(sort.offset, sort.getInput(), -1, null);
+ }
+ return super.visit(sort);
+ }
+
+ @Override
+ public RelNode visit(TableScan scan) {
+ RelOptTable table = scan.getTable();
+
+ // Determine which columns to check: traced columns, or ALL if none were
traced
+ // (SELECT * FROM t case).
+ Set<Integer> referencedColIndices = tableToReferencedCols.get(table);
+ List<String> allColumnNames = scan.getRowType().getFieldNames();
+
+ Set<String> columnsToCheck;
+ if (referencedColIndices == null || referencedColIndices.isEmpty()) {
Review Comment:
Two issues with this branch:
1. Schemaless sources (JSON/CSV, Parquet w/o metadata): the row type is just
the dynamic star `**`, so that's the only name sent to Ranger. A policy
allowing `column=*` and denying `secret` matches `**`, and `SELECT *` returns
`secret`. Fix: if any checked column is a dynamic star, require access to all
columns (e.g. check `*` and fail if any column-level deny exists for the table).
2. "None referenced" is treated as `SELECT *`, so `COUNT(*)`, `SELECT 1`,
and `EXISTS (SELECT 1 ...)` are denied for anyone restricted on a single
column. Fix: distinguish "no columns read" from "star expanded"; for the
former, do a table-level check only.
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/ColumnAccessChecker.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.conversion;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelShuttleImpl;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexSubQuery;
+import org.apache.calcite.rex.RexVisitor;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.security.AccessAuthorizerManager;
+import org.apache.drill.exec.security.TableAccessResource;
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Visitor that traverses a RelNode tree and enforces column-level SELECT
authorization
+ * via the configured {@link AccessAuthorizer} (Ranger by default).
+ *
+ * <p><b>Design:</b> For every RelNode that carries {@link RexNode} expressions
+ * (Project, Filter, Join, Aggregate, Sort), the visitor collects all {@link
RexInputRef}s
+ * and uses Calcite's {@link RelMetadataQuery#getColumnOrigins(RelNode, int)}
to trace each
+ * referenced column back to its originating {@link TableScan} column index.
This correctly
+ * handles multi-hop projections, filters, joins, and aggregations.</p>
+ *
+ * <p>For a {@link TableScan} that has NO traced column references (e.g.
+ * {@code SELECT * FROM t} with no intervening Project), ALL columns of that
table
+ * are checked.</p>
+ *
+ * <p>Correlated subqueries are handled as well: a reference to an outer query
+ * column from inside a correlated subquery is represented by Calcite as a
+ * {@link RexFieldAccess} over a {@link RexCorrelVariable} (e.g. {@code
$cor0.id}),
+ * not as a {@link RexInputRef}. Each correlation variable carries a unique
+ * name {@code $corN} assigned by {@code SqlToRelConverter}; we keep an exact
+ * map from variable name to the enclosing {@link RelNode} row scope it refers
+ * to. The map entry is populated the FIRST time we see a given {@code $corN}
+ * during Rex traversal: we use {@code putIfAbsent(name, enclosingScope)} so
+ * that subsequent encounters (including cross-nested skip-level references
+ * from deeper subqueries to an outer {@code $cor0}) never overwrite the
+ * original, correct binding. This avoids the ambiguity of row-type-based
+ * heuristics: multiple enclosing scopes with identical schemas no longer
+ * cause false positives or missed detections, and no scope stack is needed —
+ * a single field tracks the innermost enclosing scope and is saved/restored
+ * around each subquery entry via try/finally.</p>
+ *
+ * <p>System schemas (INFORMATION_SCHEMA, sys) are bypassed inside the
authorizer
+ * implementation. When authorization is disabled, the visitor is a no-op
(fail-open).</p>
+ */
+class ColumnAccessChecker extends RelShuttleImpl {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ColumnAccessChecker.class);
+
+ private final UserSession session;
+ private final DrillConfig drillConfig;
+ private final RelMetadataQuery mq;
+
+ // Records each table's referenced column indices. Uses IdentityHashMap
because
+ // RelOptTable equals/hashCode may be expensive or not identity-based.
+ private final Map<RelOptTable, Set<Integer>> tableToReferencedCols = new
IdentityHashMap<>();
+
+ // The innermost enclosing row scope for correlated references. When we are
+ // about to descend into a RexSubQuery's Rel tree, we save the previous value
+ // and set this field to the outer-query inputNode that the subquery's
+ // $cor* variables are relative to; the saved value is restored in a finally
+ // block. Because variable → scope registration uses putIfAbsent in the Rex
+ // collector, a single field is sufficient even for deep nesting: skip-level
+ // references to an outer $corN reuse the binding that was established the
+ // first time that variable was seen (at a shallower enclosing scope).
+ private RelNode enclosingScope;
+
+ // Exact, name-keyed scope mapping: $cor0 → outer row scope, $cor1 → outer
+ // row scope, etc. Populated with putIfAbsent during Rex traversal so the
+ // first occurrence of each $corN establishes a permanent binding. Lookup is
+ // pure get(); a miss falls back (safely, in the over-approximate direction)
+ // to enclosingScope and emits a warning so unusual plans remain diagnosable.
+ private final Map<String, RelNode> correlationScopeByVar = new HashMap<>();
+
+ ColumnAccessChecker(UserSession session, DrillConfig drillConfig,
RelMetadataQuery mq) {
+ this.session = session;
+ this.drillConfig = drillConfig;
+ this.mq = mq;
+ }
+
+ /**
+ * Entry point: traverse the tree and enforce column-level access.
+ */
+ void check(RelNode root) {
+ // Also trace the root node's output columns (covers bare TableScan root or
+ // top-level Project output).
+ traceOutputColumns(root);
+
+ // Walk the tree to collect RexInputRef origins from all
expression-bearing nodes.
+ root.accept(this);
+ }
+
+ // ------------------------------------------------------------------
+ // RelShuttle overrides — collect RexInputRefs from expression-bearing nodes
+ // ------------------------------------------------------------------
+
+ @Override
+ public RelNode visit(LogicalProject project) {
+ collectRefs(project.getProjects(), project.getInput());
+ return super.visit(project);
+ }
+
+ @Override
+ public RelNode visit(LogicalFilter filter) {
+ if (filter.getCondition() != null) {
+ analyzeRex(filter.getCondition(), filter.getInput(), -1, null);
+ }
+ return super.visit(filter);
+ }
+
+ @Override
+ public RelNode visit(LogicalJoin join) {
+ if (join.getCondition() != null) {
+ int leftCount = join.getLeft().getRowType().getFieldCount();
+ analyzeRex(join.getCondition(), join.getRight(), leftCount,
join.getLeft());
+ }
+ return super.visit(join);
+ }
+
+ @Override
+ public RelNode visit(LogicalAggregate aggregate) {
+ for (int i : aggregate.getGroupSet()) {
+ traceColumnOrigin(aggregate.getInput(), i);
+ }
+ // AggregateCall arguments (the x in SUM(x)), FILTER (WHERE ...) columns,
+ // and WITHIN GROUP collation columns are not covered by getGroupSet().
+ // Trace them explicitly rather than relying on an implicit Project beneath
+ // the aggregate (plan shape can change across Calcite upgrades).
+ for (AggregateCall call : aggregate.getAggCallList()) {
+ for (int arg : call.getArgList()) {
+ traceColumnOrigin(aggregate.getInput(), arg);
+ }
+ if (call.filterArg >= 0) {
+ traceColumnOrigin(aggregate.getInput(), call.filterArg);
+ }
+ if (call.getCollation() != null) {
+ for (RelFieldCollation fc : call.getCollation().getFieldCollations()) {
+ traceColumnOrigin(aggregate.getInput(), fc.getFieldIndex());
+ }
+ }
+ }
+ return super.visit(aggregate);
+ }
+
+ @Override
+ public RelNode visit(LogicalSort sort) {
+ if (sort.getCollation() != null) {
+ sort.getCollation().getFieldCollations().forEach(fc ->
+ traceColumnOrigin(sort.getInput(), fc.getFieldIndex()));
+ }
+ // LIMIT/OFFSET are RexNode expressions that may carry RexSubQuery or
+ // RexInputRef (e.g. LIMIT (SELECT MAX(amount) FROM ...)). analyzeRex
+ // descends into them; without it, subqueries reachable only from Sort
+ // would never be column-checked.
+ if (sort.fetch != null) {
+ analyzeRex(sort.fetch, sort.getInput(), -1, null);
+ }
+ if (sort.offset != null) {
+ analyzeRex(sort.offset, sort.getInput(), -1, null);
+ }
+ return super.visit(sort);
+ }
+
+ @Override
+ public RelNode visit(TableScan scan) {
+ RelOptTable table = scan.getTable();
+
+ // Determine which columns to check: traced columns, or ALL if none were
traced
+ // (SELECT * FROM t case).
+ Set<Integer> referencedColIndices = tableToReferencedCols.get(table);
+ List<String> allColumnNames = scan.getRowType().getFieldNames();
+
+ Set<String> columnsToCheck;
+ if (referencedColIndices == null || referencedColIndices.isEmpty()) {
+ // SELECT * — check all columns
+ columnsToCheck = new HashSet<>(allColumnNames);
+ } else {
+ columnsToCheck = new HashSet<>();
+ for (int idx : referencedColIndices) {
+ if (idx >= 0 && idx < allColumnNames.size()) {
+ columnsToCheck.add(allColumnNames.get(idx));
+ }
+ }
+ }
+
+ if (columnsToCheck.isEmpty()) {
+ return scan;
+ }
+
+ enforceColumnAccess(table, columnsToCheck);
+ return scan;
+ }
+
+ /**
+ * Traces the output columns of a RelNode back to their table-scan origins.
+ */
+ private void traceOutputColumns(RelNode node) {
+ if (node == null) {
+ return;
+ }
+ int fieldCount = node.getRowType().getFieldCount();
+ for (int i = 0; i < fieldCount; i++) {
+ traceColumnOrigin(node, i);
+ }
+ }
+
+ /**
+ * Traces a single output column of {@code node} at index {@code
columnIndex} back to
+ * table-scan origins, recording them in {@link #tableToReferencedCols}.
+ */
+ private void traceColumnOrigin(RelNode node, int columnIndex) {
+ if (node == null || mq == null) {
+ return;
+ }
+ Set<RelColumnOrigin> origins;
+ try {
+ origins = mq.getColumnOrigins(node, columnIndex);
+ } catch (Exception e) {
+ logger.debug("getColumnOrigins failed for {} column {}", node,
columnIndex, e);
+ return;
+ }
+ if (origins == null) {
+ return;
+ }
+ for (RelColumnOrigin origin : origins) {
+ RelOptTable originTable = origin.getOriginTable();
+ if (originTable != null) {
+ // Record origins for ALL table types (DrillTable, JdbcTable, etc.).
+ // Previously this only recorded DrillTable origins, which caused
+ // JDBC storage plugin tables (JdbcTable) to be skipped entirely.
+ tableToReferencedCols
+ .computeIfAbsent(originTable, k -> new HashSet<>())
+ .add(origin.getOriginColumnOrdinal());
+ }
+ }
+ }
+
+ /**
+ * Collects RexInputRefs from a list of RexNodes and traces each to its
+ * table-scan origin via the input node's metadata. Also processes any
+ * {@link RexSubQuery} found in the expressions (scalar/IN/EXISTS subqueries)
+ * so that column references inside subqueries are authorized.
+ */
+ private void collectRefs(List<RexNode> rexNodes, RelNode inputNode) {
+ if (rexNodes == null || inputNode == null) {
+ return;
+ }
+ for (RexNode rex : rexNodes) {
+ analyzeRex(rex, inputNode, -1, null);
+ }
+ }
+
+ /**
+ * Analyzes a {@link RexNode} expression, collecting {@link RexInputRef}s and
+ * {@link RexSubQuery}s, tracing each input ref to its table-scan origin and
+ * recursively visiting each subquery's {@link RelNode} tree.
+ *
+ * @param rex the expression to analyze
+ * @param inputNode the input RelNode that RexInputRefs resolve against
+ * @param leftCount if {@code >= 0}, indicates a join condition: refs with
+ * index {@code < leftCount} resolve against {@code
leftInput},
+ * others resolve against {@code inputNode} (the right
input)
+ * with offset {@code leftCount}. If {@code < 0}, all refs
+ * resolve against {@code inputNode}.
+ * @param leftInput the left input of a join, or {@code null} when
+ * {@code leftCount < 0}.
+ */
+ private void analyzeRex(RexNode rex, RelNode inputNode, int leftCount,
RelNode leftInput) {
+ if (rex == null) {
+ return;
+ }
+ Set<Integer> refs = new HashSet<>();
+ List<RexSubQuery> subQueries = new ArrayList<>();
+ List<CorrelFieldAccess> correlAccesses = new ArrayList<>();
+ rex.accept(new RexRefCollector(refs, subQueries, correlAccesses));
+ for (int refIndex : refs) {
+ if (leftCount >= 0 && refIndex < leftCount) {
+ traceColumnOrigin(leftInput, refIndex);
+ } else if (leftCount >= 0) {
+ traceColumnOrigin(inputNode, refIndex - leftCount);
+ } else {
+ traceColumnOrigin(inputNode, refIndex);
+ }
+ }
+ // Resolve correlated outer-column references. A RexFieldAccess over a
+ // RexCorrelVariable (e.g. $cor0.id inside a correlated subquery) always
+ // refers to an ENCLOSING subquery's row scope, never to inputNode itself
+ // (a reference to inputNode would be a plain RexInputRef).
+ //
+ // Registration happens inside RexRefCollector.visitFieldAccess with
+ // putIfAbsent(varName, enclosingScope), so by the time we reach here the
+ // scope for every $corN we just collected has been bound if possible.
+ // Lookup is a direct map get(); the uncommon case of an unbound variable
+ // falls back (safely over-approximating) to enclosingScope before giving
+ // up — tracing an extra scope is harmless and keeps us on the secure side
+ // of the check.
+ for (CorrelFieldAccess cfa : correlAccesses) {
+ RelNode scope = correlationScopeByVar.get(cfa.varName);
+ if (scope != null) {
+ traceColumnOrigin(scope, cfa.fieldIndex);
+ continue;
+ }
+ if (enclosingScope != null) {
+ logger.debug("Correlated reference '{}' was not pre-bound; "
+ + "falling back to current enclosingScope for a safe
over-approximation.",
+ cfa.varName);
+ traceColumnOrigin(enclosingScope, cfa.fieldIndex);
+ } else {
+ logger.warn("Unresolved correlated column reference '{}' (field index
{}): "
+ + "no enclosing correlation scope could be determined.
Column-level "
+ + "authorization may be incomplete for this reference.",
+ cfa.varName, cfa.fieldIndex);
+ }
+ }
+ for (RexSubQuery sq : subQueries) {
+ // Trace the subquery's output columns to their table-scan origins.
+ // For scalar subqueries (e.g. SELECT sum(user_id) FROM t), this traces
+ // the aggregate output back to the underlying table column.
+ traceOutputColumns(sq.rel);
+ // The subquery's $cor* variables reference the rows produced by
+ // inputNode (the input of the node whose expression contained this
+ // RexSubQuery). Make inputNode the active enclosing scope while we
+ // descend into sq.rel; save/restore the previous value so nesting and
+ // skip-level references behave correctly. Registration of new $corN
+ // names in correlationScopeByVar uses putIfAbsent, so a variable whose
+ // binding was established at a shallower scope is never overwritten.
+ RelNode prevEnclosing = enclosingScope;
+ enclosingScope = inputNode;
+ try {
+ // Recursively visit the subquery's RelNode tree so that RexInputRefs
+ // and correlated references inside the subquery are also collected
+ // and traced.
+ sq.rel.accept(this);
+ } finally {
+ enclosingScope = prevEnclosing;
+ }
+ }
+ }
+
+ /**
+ * Enforces column-level access for the given table and column set.
+ */
+ private void enforceColumnAccess(RelOptTable table, Set<String> columns) {
+ AccessAuthorizer authorizer =
AccessAuthorizerManager.getAuthorizer(drillConfig);
+ // When authorization is disabled the manager returns the NoOp authorizer,
+ // which allows all access — no enabled/disabled branching needed here.
+
+ // Resolve datasource / schema / table from the qualified name via the
+ // shared resolver, so column-level checks address exactly the same
+ // resource as the table-level checks in DrillCalciteCatalogReader. This
+ // works for ALL table types (DrillTable, JdbcTable, etc.) — previously
+ // this method required a DrillTable and skipped JdbcTable, leaving JDBC
+ // storage plugin tables without column-level authorization.
+ TableAccessResource resource =
TableAccessResource.resolve(table.getQualifiedName());
+ String userName = session.getCredentials().getUserName();
+
+ if (!authorizer.checkColumnAccess(UserIdentity.of(userName),
resource.getDataSource(),
+ resource.getSchemaPath(), resource.getTable(), columns,
AccessType.SELECT)) {
+ throw UserException.permissionError()
+ .message("Access denied: user '%s' lacks SELECT privilege on one or
more columns " +
+ "(%s) of table %s", userName, columns, resource)
+ .build(logger);
+ }
+ }
+
+ /**
+ * Holder for a correlated outer-column reference captured during Rex
+ * traversal: the variable name of the {@link RexCorrelVariable}
+ * (e.g. {@code "$cor0"}, used to do an exact lookup of the enclosing row
+ * scope via {@link #correlationScopeByVar}) and the index of the referenced
+ * field within that row scope.
+ */
+ private static final class CorrelFieldAccess {
+ final String varName;
+ final int fieldIndex;
+
+ CorrelFieldAccess(String varName, int fieldIndex) {
+ this.varName = varName;
+ this.fieldIndex = fieldIndex;
+ }
+ }
+
+ /**
+ * RexVisitor that collects all {@link RexInputRef} indices,
+ * {@link RexSubQuery} instances, and correlated outer-column references
+ * encountered in a {@link RexNode} tree. A correlated reference to an outer
+ * query column appears as a {@link RexFieldAccess} over a
+ * {@link RexCorrelVariable} (e.g. {@code $cor0.id}); such references are not
+ * {@link RexInputRef}s and would be silently skipped by a plain
+ * {@code RexInputRef}-only visitor, bypassing column-level authorization for
+ * the referenced outer column.
+ *
+ * <p>This collector is non-static so it can eagerly register each {@code
+ * $corN} in {@link #correlationScopeByVar} at the exact moment we first
+ * encounter it during Rex traversal. Registration is
+ * {@code putIfAbsent(name, enclosingScope)}, which guarantees we never
+ * overwrite a variable that was already bound at a shallower nesting level
+ * (which is exactly how skip-level references to outer variables stay
+ * correctly bound).
+ *
+ * <p>{@link RexSubQuery#accept(RexVisitor)} dispatches to
+ * {@link #visitSubQuery(RexSubQuery)} (not {@code visitCall}), so a plain
+ * {@code RexInputRef}-only visitor silently skips over subqueries. This
+ * collector overrides {@code visitSubQuery} to capture the subquery and then
+ * continues traversing its operands so that nested {@link RexInputRef}s
+ * (e.g. the left side of {@code x IN (SELECT ...)}) and nested subqueries
+ * are also collected.</p>
+ */
+ private final class RexRefCollector extends RexVisitorImpl<Void> {
+ private final Set<Integer> refs;
+ private final List<RexSubQuery> subQueries;
+ private final List<CorrelFieldAccess> correlAccesses;
+
+ RexRefCollector(Set<Integer> refs, List<RexSubQuery> subQueries,
+ List<CorrelFieldAccess> correlAccesses) {
+ super(true);
+ this.refs = refs;
+ this.subQueries = subQueries;
+ this.correlAccesses = correlAccesses;
+ }
+
+ @Override
+ public Void visitInputRef(RexInputRef ref) {
+ refs.add(ref.getIndex());
+ return null;
+ }
+
+ @Override
+ public Void visitSubQuery(RexSubQuery subQuery) {
+ subQueries.add(subQuery);
+ // Continue traversing operands (e.g. the left expression of `x IN
(...)`)
+ // to collect any RexInputRefs and nested RexSubQueries within them.
+ for (RexNode operand : subQuery.getOperands()) {
+ operand.accept(this);
+ }
+ return null;
+ }
+
+ @Override
+ public Void visitFieldAccess(RexFieldAccess fieldAccess) {
+ // A RexFieldAccess over a RexCorrelVariable is how Calcite represents a
+ // correlated reference to an outer query's column (e.g. $cor0.id inside
+ // a subquery). Such references are not RexInputRefs and would otherwise
+ // be silently skipped by this collector, bypassing column-level
+ // authorization for the referenced outer column.
+ RexNode refExpr = fieldAccess.getReferenceExpr();
+ if (refExpr instanceof RexCorrelVariable) {
+ RexCorrelVariable corVar = (RexCorrelVariable) refExpr;
+ String varName = corVar.getName();
+ // Eager, idempotent registration: putIfAbsent so the first
+ // enclosing scope we saw this variable under wins, and any later
+ // occurrences (possibly from a deeper nested scope after
+ // enclosingScope has been overwritten) do not clobber the binding.
+ if (enclosingScope != null) {
+ correlationScopeByVar.putIfAbsent(varName, enclosingScope);
Review Comment:
`putIfAbsent` binds the correlation variable to whichever scope is active
when it's first seen, which is only correct if each intermediate level also
references it.
```sql
SELECT a.pub FROM a WHERE EXISTS (
SELECT 1 FROM b WHERE EXISTS (
SELECT 1 FROM c WHERE c.k = a.secret AND c.j = b.x))
```
`$cor0` gets bound to `b`, so `a.secret`'s index is resolved against `b` and
never checked on `a`.
Fix: bind by `CorrelationId` at the node that defines it (the
`LogicalFilter`/`LogicalProject`/`LogicalJoin` whose `getVariablesSet()`
contains it, or the `LogicalCorrelate`), not on first use. Worth adding this
query as a test.
##########
auth/drill-ranger-plugin/src/main/java/org/apache/ranger/authorization/drill/authorizer/DrillAccessControl.java:
##########
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file distributed
with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.ranger.authorization.drill.authorizer;
+
+
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.ranger.authorization.drill.resource.DrillAccessType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Drill-facing authorization facade: implementation of the Drill
+ * {@link AccessAuthorizer} SPI backed by Ranger.
+ *
+ * <p>The class is instantiated once by the shim (drill-ranger-plugin-shim)
+ * via {@code DrillAccessControl(String serviceName)} — the constructor
+ * completes all initialization (or throws, fail-closed), mirroring Presto's
+ * {@code RangerPrestoAccessControl}. All access checks are instance methods
+ * invoked directly through the SPI interface.</p>
+ *
+ * <p>Group resolution: if the engine-supplied {@link UserIdentity} carries
+ * groups (resolved at authentication time), they are used as-is; otherwise
+ * the groups are resolved via Hadoop UGI ({@link #getUserGroups}).</p>
+ *
+ * <p>On Ranger evaluation error, checks return {@code false} (fail-closed).
+ * The service name identifies the Ranger service instance whose policies
+ * are evaluated.</p>
+ */
+public class DrillAccessControl implements AccessAuthorizer {
+
+ private static final Logger logger =
LoggerFactory.getLogger(DrillAccessControl.class);
+
+ private final DrillAuthorizer authorizer;
+
+ // Set of system schemas that bypass authorization (information_schema, sys,
etc.)
+ // Stored in uppercase; isSystemSchema() uppercases input before lookup so
the
+ // bypass is case-insensitive (e.g. "information_schema",
"INFORMATION_SCHEMA",
+ // "Sys", "SYS" all match).
+ private static final Set<String> SYSTEM_SCHEMAS = new
HashSet<>(Arrays.asList(
+ "INFORMATION_SCHEMA", "SYS"
+ ));
+
+ /**
+ * Creates and initializes the Ranger Drill plugin. Completes all
+ * initialization or throws (fail-closed) — the caller (the shim) must not
+ * receive a half-initialized authorizer.
+ *
+ * @param serviceName the Ranger service instance name (must match a service
created in Ranger Admin)
+ */
+ public DrillAccessControl(String serviceName) {
+ logger.info("Initializing Ranger Drill authorization plugin for service:
{}", serviceName);
+ try {
+ this.authorizer = new DrillAuthorizer(serviceName);
+ logger.info("Ranger Drill authorization plugin initialized
successfully");
+ } catch (Exception e) {
+ logger.error("Failed to initialize Ranger Drill plugin for service {}",
serviceName, e);
+ throw new RuntimeException(
+ "Failed to initialize Ranger Drill plugin — authorization disabled "
+ serviceName
+ + " with exception: " + e);
+ }
+ }
+
+ /**
+ * Package-private constructor for unit tests: injects a (mock) authorizer
+ * directly, bypassing Ranger Admin connectivity.
+ */
+ DrillAccessControl(DrillAuthorizer authorizer) {
+ this.authorizer = authorizer;
+ }
+
+ /**
+ * Resolves the OS-level groups for a given user via Hadoop UGI. Used only
+ * as a fallback when the engine-supplied {@link UserIdentity} carries no
+ * groups.
+ *
+ * @param user the username
+ * @return a set of group names (never null, empty on failure)
+ */
+ public static Set<String> getUserGroups(String user) {
+ if (user == null || user.trim().isEmpty()) {
+ return Collections.emptySet();
+ }
+ try {
+ UserGroupInformation ugi = UserGroupInformation.createRemoteUser(user);
+ String[] groups = ugi.getGroupNames();
+ return groups == null ? Collections.emptySet() : new
HashSet<>(Arrays.asList(groups));
+ } catch (Exception e) {
+ logger.warn("Failed to determine groups for user={}", user, e);
+ return Collections.emptySet();
+ }
+ }
+
+ /**
+ * Checks table-level access. The SPI {@link AccessType} is mapped to
+ * {@link DrillAccessType} by name; if the mapping fails (e.g. the SPI
+ * added a type this plugin does not know yet) access is denied
+ * (fail-closed).
+ *
+ * <p>System schemas ({@code INFORMATION_SCHEMA}, {@code sys}) bypass
+ * authorization. A null/empty schema is invalid input and fails closed
+ * rather than silently bypassing Ranger.</p>
+ *
+ * @param user the querying user identity
+ * @param dataSource the Drill storage plugin name (e.g. "dfs", "hbase")
+ * @param schema the schema path (e.g. "dfs.tmp")
+ * @param table the table name
+ * @param accessType the access type (e.g. SELECT, CREATE)
+ * @return {@code true} if access is allowed
+ */
+ @Override
+ public boolean checkTableAccess(UserIdentity user, String dataSource, String
schema,
+ String table, AccessType accessType) {
+ DrillAccessType operator = parseAccessType(accessType, user, schema,
table);
+ if (operator == null) {
+ return false;
+ }
+ Boolean bypass = systemSchemaBypass(user, schema, table);
+ if (bypass != null) {
+ return bypass;
+ }
+ try {
+ return authorizer.checkTableAccess(resolveIdentity(user), dataSource,
schema, table, operator);
+ } catch (Exception e) {
+ logger.error("Checking table access for user={}, schema={}, table={}.
with exception:{}",
+ user.getUser(), schema, table, e.toString());
+ return false; // fail-closed on error
+ }
+ }
+
+ /**
+ * Checks column-level access for a set of columns. Returns {@code true}
+ * only if the user has the specified access type on ALL given columns.
+ *
+ * <p>System schemas ({@code INFORMATION_SCHEMA}, {@code sys}) bypass
+ * authorization. A null/empty schema is invalid input and fails closed
+ * rather than silently bypassing Ranger.</p>
+ *
+ * @param user the querying user identity
+ * @param dataSource the Drill storage plugin name
+ * @param schema the schema path
+ * @param table the table name
+ * @param columns the set of column names to check
+ * @param accessType the access type (e.g. SELECT)
+ * @return {@code true} if access is allowed for every column
+ */
+ @Override
+ public boolean checkColumnAccess(UserIdentity user, String dataSource,
String schema,
+ String table, Set<String> columns, AccessType accessType) {
+ DrillAccessType operator = parseAccessType(accessType, user, schema,
table);
+ if (operator == null) {
+ return false;
+ }
+ Boolean bypass = systemSchemaBypass(user, schema, table);
+ if (bypass != null) {
+ return bypass;
+ }
+ try {
+ return authorizer.checkColumnAccess(resolveIdentity(user), dataSource,
schema, table,
+ columns, operator);
+ } catch (Exception e) {
+ logger.error("Error checking column access for user={}, schema={},
table={}",
+ user.getUser(), schema, table, e);
+ return false; // fail-closed on error
+ }
+ }
+
+ /**
+ * Maps the SPI {@link AccessType} to a {@link DrillAccessType} by name.
+ * Returns {@code null} (and logs) when the SPI enum carries a type this
+ * plugin does not know yet — callers deny access in that case
+ * (fail-closed). This guards against drift when a newer SPI adds access
+ * types before the Ranger service-def does.
+ */
+ private DrillAccessType parseAccessType(AccessType accessType, UserIdentity
user,
+ String schema, String table) {
+ try {
+ return DrillAccessType.valueOf(accessType.name());
+ } catch (Exception e) {
+ logger.error("Unsupported access type '{}', denied access for user={},
schema={}, table={}",
+ accessType, user.getUser(), schema, table);
+ return null;
+ }
+ }
+
+ /**
+ * Resolves the effective identity for an access check: engine-supplied
+ * groups are used as-is; when the identity carries none (e.g. mount points
+ * that only have the authenticated user name), the groups are resolved via
+ * Hadoop UGI ({@link #getUserGroups}).
+ */
+ private UserIdentity resolveIdentity(UserIdentity user) {
+ Set<String> groups = user.getGroups();
+ if (groups != null && !groups.isEmpty()) {
+ return user;
+ }
+ return UserIdentity.builder()
+ .setUser(user.getUser())
+ .setGroups(getUserGroups(user.getUser()))
+ .build();
+ }
+
+ /**
+ * Returns whether the given schema is a system schema that should bypass
+ * authorization.
+ *
+ * <p>Comparison is case-insensitive so that SQL like
+ * {@code SELECT * FROM information_schema.tables} (lowercase) or
+ * {@code SELECT * FROM SYS.DRILLBITS} (uppercase) both bypass authorization,
+ * matching Drill's own case-insensitive schema resolution.
+ *
+ * <p>For compound schema paths like {@code dfs.tmp}, only the top-level
segment
+ * (the storage plugin name) is checked — that is intentional, because system
+ * schemas ({@code INFORMATION_SCHEMA}, {@code sys}) are always top-level.
+ */
+ private static boolean isSystemSchema(String schema) {
+ if (schema == null || schema.trim().isEmpty()) {
+ throw new IllegalArgumentException(
+ "Schema must not be null or empty for authorization check; refusing
to treat as system schema");
+ }
+ // Use only the top-level segment of a compound schema path
+ // (e.g. "dfs.tmp" -> "dfs", "INFORMATION_SCHEMA" -> "INFORMATION_SCHEMA")
+ String topLevel = schema;
+ int dot = schema.indexOf('.');
+ if (dot > 0) {
+ topLevel = schema.substring(0, dot);
+ }
+ return SYSTEM_SCHEMAS.contains(topLevel.toUpperCase());
Review Comment:
The schema passed here has the datasource stripped (`TableAccessResource`),
so `mysql.sys.*`, `pg.information_schema.*`, or a dfs workspace named `sys` all
bypass Ranger. MySQL ships a `sys` database by default.
Fix: pass the datasource into the bypass and match on it
(`SYSTEM_SCHEMAS.contains(dataSource.toUpperCase())`), not the schema.
##########
exec/java-exec/src/main/java/org/apache/drill/exec/planner/sql/conversion/ColumnAccessChecker.java:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.planner.sql.conversion;
+
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.RelShuttleImpl;
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.logical.LogicalAggregate;
+import org.apache.calcite.rel.logical.LogicalFilter;
+import org.apache.calcite.rel.logical.LogicalJoin;
+import org.apache.calcite.rel.logical.LogicalProject;
+import org.apache.calcite.rel.logical.LogicalSort;
+import org.apache.calcite.rel.metadata.RelColumnOrigin;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rex.RexCorrelVariable;
+import org.apache.calcite.rex.RexFieldAccess;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexSubQuery;
+import org.apache.calcite.rex.RexVisitor;
+import org.apache.calcite.rex.RexVisitorImpl;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.security.AccessAuthorizerManager;
+import org.apache.drill.exec.security.TableAccessResource;
+import org.apache.drill.exec.security.spi.AccessAuthorizer;
+import org.apache.drill.exec.security.spi.AccessType;
+import org.apache.drill.exec.security.spi.UserIdentity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Visitor that traverses a RelNode tree and enforces column-level SELECT
authorization
+ * via the configured {@link AccessAuthorizer} (Ranger by default).
+ *
+ * <p><b>Design:</b> For every RelNode that carries {@link RexNode} expressions
+ * (Project, Filter, Join, Aggregate, Sort), the visitor collects all {@link
RexInputRef}s
+ * and uses Calcite's {@link RelMetadataQuery#getColumnOrigins(RelNode, int)}
to trace each
+ * referenced column back to its originating {@link TableScan} column index.
This correctly
+ * handles multi-hop projections, filters, joins, and aggregations.</p>
+ *
+ * <p>For a {@link TableScan} that has NO traced column references (e.g.
+ * {@code SELECT * FROM t} with no intervening Project), ALL columns of that
table
+ * are checked.</p>
+ *
+ * <p>Correlated subqueries are handled as well: a reference to an outer query
+ * column from inside a correlated subquery is represented by Calcite as a
+ * {@link RexFieldAccess} over a {@link RexCorrelVariable} (e.g. {@code
$cor0.id}),
+ * not as a {@link RexInputRef}. Each correlation variable carries a unique
+ * name {@code $corN} assigned by {@code SqlToRelConverter}; we keep an exact
+ * map from variable name to the enclosing {@link RelNode} row scope it refers
+ * to. The map entry is populated the FIRST time we see a given {@code $corN}
+ * during Rex traversal: we use {@code putIfAbsent(name, enclosingScope)} so
+ * that subsequent encounters (including cross-nested skip-level references
+ * from deeper subqueries to an outer {@code $cor0}) never overwrite the
+ * original, correct binding. This avoids the ambiguity of row-type-based
+ * heuristics: multiple enclosing scopes with identical schemas no longer
+ * cause false positives or missed detections, and no scope stack is needed —
+ * a single field tracks the innermost enclosing scope and is saved/restored
+ * around each subquery entry via try/finally.</p>
+ *
+ * <p>System schemas (INFORMATION_SCHEMA, sys) are bypassed inside the
authorizer
+ * implementation. When authorization is disabled, the visitor is a no-op
(fail-open).</p>
+ */
+class ColumnAccessChecker extends RelShuttleImpl {
+
+ private static final Logger logger =
LoggerFactory.getLogger(ColumnAccessChecker.class);
+
+ private final UserSession session;
+ private final DrillConfig drillConfig;
+ private final RelMetadataQuery mq;
+
+ // Records each table's referenced column indices. Uses IdentityHashMap
because
+ // RelOptTable equals/hashCode may be expensive or not identity-based.
+ private final Map<RelOptTable, Set<Integer>> tableToReferencedCols = new
IdentityHashMap<>();
+
+ // The innermost enclosing row scope for correlated references. When we are
+ // about to descend into a RexSubQuery's Rel tree, we save the previous value
+ // and set this field to the outer-query inputNode that the subquery's
+ // $cor* variables are relative to; the saved value is restored in a finally
+ // block. Because variable → scope registration uses putIfAbsent in the Rex
+ // collector, a single field is sufficient even for deep nesting: skip-level
+ // references to an outer $corN reuse the binding that was established the
+ // first time that variable was seen (at a shallower enclosing scope).
+ private RelNode enclosingScope;
+
+ // Exact, name-keyed scope mapping: $cor0 → outer row scope, $cor1 → outer
+ // row scope, etc. Populated with putIfAbsent during Rex traversal so the
+ // first occurrence of each $corN establishes a permanent binding. Lookup is
+ // pure get(); a miss falls back (safely, in the over-approximate direction)
+ // to enclosingScope and emits a warning so unusual plans remain diagnosable.
+ private final Map<String, RelNode> correlationScopeByVar = new HashMap<>();
+
+ ColumnAccessChecker(UserSession session, DrillConfig drillConfig,
RelMetadataQuery mq) {
+ this.session = session;
+ this.drillConfig = drillConfig;
+ this.mq = mq;
+ }
+
+ /**
+ * Entry point: traverse the tree and enforce column-level access.
+ */
+ void check(RelNode root) {
+ // Also trace the root node's output columns (covers bare TableScan root or
+ // top-level Project output).
+ traceOutputColumns(root);
+
+ // Walk the tree to collect RexInputRef origins from all
expression-bearing nodes.
+ root.accept(this);
+ }
+
+ // ------------------------------------------------------------------
+ // RelShuttle overrides — collect RexInputRefs from expression-bearing nodes
+ // ------------------------------------------------------------------
+
+ @Override
+ public RelNode visit(LogicalProject project) {
+ collectRefs(project.getProjects(), project.getInput());
+ return super.visit(project);
+ }
+
+ @Override
+ public RelNode visit(LogicalFilter filter) {
+ if (filter.getCondition() != null) {
+ analyzeRex(filter.getCondition(), filter.getInput(), -1, null);
+ }
+ return super.visit(filter);
+ }
+
+ @Override
+ public RelNode visit(LogicalJoin join) {
+ if (join.getCondition() != null) {
+ int leftCount = join.getLeft().getRowType().getFieldCount();
+ analyzeRex(join.getCondition(), join.getRight(), leftCount,
join.getLeft());
+ }
+ return super.visit(join);
+ }
+
+ @Override
+ public RelNode visit(LogicalAggregate aggregate) {
+ for (int i : aggregate.getGroupSet()) {
+ traceColumnOrigin(aggregate.getInput(), i);
+ }
+ // AggregateCall arguments (the x in SUM(x)), FILTER (WHERE ...) columns,
+ // and WITHIN GROUP collation columns are not covered by getGroupSet().
+ // Trace them explicitly rather than relying on an implicit Project beneath
+ // the aggregate (plan shape can change across Calcite upgrades).
+ for (AggregateCall call : aggregate.getAggCallList()) {
+ for (int arg : call.getArgList()) {
+ traceColumnOrigin(aggregate.getInput(), arg);
+ }
+ if (call.filterArg >= 0) {
+ traceColumnOrigin(aggregate.getInput(), call.filterArg);
+ }
+ if (call.getCollation() != null) {
+ for (RelFieldCollation fc : call.getCollation().getFieldCollations()) {
+ traceColumnOrigin(aggregate.getInput(), fc.getFieldIndex());
+ }
+ }
+ }
+ return super.visit(aggregate);
+ }
+
+ @Override
+ public RelNode visit(LogicalSort sort) {
+ if (sort.getCollation() != null) {
+ sort.getCollation().getFieldCollations().forEach(fc ->
+ traceColumnOrigin(sort.getInput(), fc.getFieldIndex()));
+ }
+ // LIMIT/OFFSET are RexNode expressions that may carry RexSubQuery or
+ // RexInputRef (e.g. LIMIT (SELECT MAX(amount) FROM ...)). analyzeRex
+ // descends into them; without it, subqueries reachable only from Sort
+ // would never be column-checked.
+ if (sort.fetch != null) {
+ analyzeRex(sort.fetch, sort.getInput(), -1, null);
+ }
+ if (sort.offset != null) {
+ analyzeRex(sort.offset, sort.getInput(), -1, null);
+ }
+ return super.visit(sort);
+ }
+
+ @Override
+ public RelNode visit(TableScan scan) {
+ RelOptTable table = scan.getTable();
+
+ // Determine which columns to check: traced columns, or ALL if none were
traced
+ // (SELECT * FROM t case).
+ Set<Integer> referencedColIndices = tableToReferencedCols.get(table);
+ List<String> allColumnNames = scan.getRowType().getFieldNames();
+
+ Set<String> columnsToCheck;
+ if (referencedColIndices == null || referencedColIndices.isEmpty()) {
+ // SELECT * — check all columns
+ columnsToCheck = new HashSet<>(allColumnNames);
+ } else {
+ columnsToCheck = new HashSet<>();
+ for (int idx : referencedColIndices) {
+ if (idx >= 0 && idx < allColumnNames.size()) {
+ columnsToCheck.add(allColumnNames.get(idx));
+ }
+ }
+ }
+
+ if (columnsToCheck.isEmpty()) {
+ return scan;
+ }
+
+ enforceColumnAccess(table, columnsToCheck);
+ return scan;
+ }
+
+ /**
+ * Traces the output columns of a RelNode back to their table-scan origins.
+ */
+ private void traceOutputColumns(RelNode node) {
+ if (node == null) {
+ return;
+ }
+ int fieldCount = node.getRowType().getFieldCount();
+ for (int i = 0; i < fieldCount; i++) {
+ traceColumnOrigin(node, i);
+ }
+ }
+
+ /**
+ * Traces a single output column of {@code node} at index {@code
columnIndex} back to
+ * table-scan origins, recording them in {@link #tableToReferencedCols}.
+ */
+ private void traceColumnOrigin(RelNode node, int columnIndex) {
+ if (node == null || mq == null) {
+ return;
+ }
+ Set<RelColumnOrigin> origins;
+ try {
+ origins = mq.getColumnOrigins(node, columnIndex);
+ } catch (Exception e) {
+ logger.debug("getColumnOrigins failed for {} column {}", node,
columnIndex, e);
+ return;
+ }
+ if (origins == null) {
+ return;
+ }
+ for (RelColumnOrigin origin : origins) {
+ RelOptTable originTable = origin.getOriginTable();
+ if (originTable != null) {
+ // Record origins for ALL table types (DrillTable, JdbcTable, etc.).
+ // Previously this only recorded DrillTable origins, which caused
+ // JDBC storage plugin tables (JdbcTable) to be skipped entirely.
+ tableToReferencedCols
+ .computeIfAbsent(originTable, k -> new HashSet<>())
+ .add(origin.getOriginColumnOrdinal());
+ }
+ }
+ }
+
+ /**
+ * Collects RexInputRefs from a list of RexNodes and traces each to its
+ * table-scan origin via the input node's metadata. Also processes any
+ * {@link RexSubQuery} found in the expressions (scalar/IN/EXISTS subqueries)
+ * so that column references inside subqueries are authorized.
+ */
+ private void collectRefs(List<RexNode> rexNodes, RelNode inputNode) {
+ if (rexNodes == null || inputNode == null) {
+ return;
+ }
+ for (RexNode rex : rexNodes) {
+ analyzeRex(rex, inputNode, -1, null);
+ }
+ }
+
+ /**
+ * Analyzes a {@link RexNode} expression, collecting {@link RexInputRef}s and
+ * {@link RexSubQuery}s, tracing each input ref to its table-scan origin and
+ * recursively visiting each subquery's {@link RelNode} tree.
+ *
+ * @param rex the expression to analyze
+ * @param inputNode the input RelNode that RexInputRefs resolve against
+ * @param leftCount if {@code >= 0}, indicates a join condition: refs with
+ * index {@code < leftCount} resolve against {@code
leftInput},
+ * others resolve against {@code inputNode} (the right
input)
+ * with offset {@code leftCount}. If {@code < 0}, all refs
+ * resolve against {@code inputNode}.
+ * @param leftInput the left input of a join, or {@code null} when
+ * {@code leftCount < 0}.
+ */
+ private void analyzeRex(RexNode rex, RelNode inputNode, int leftCount,
RelNode leftInput) {
+ if (rex == null) {
+ return;
+ }
+ Set<Integer> refs = new HashSet<>();
+ List<RexSubQuery> subQueries = new ArrayList<>();
+ List<CorrelFieldAccess> correlAccesses = new ArrayList<>();
+ rex.accept(new RexRefCollector(refs, subQueries, correlAccesses));
+ for (int refIndex : refs) {
+ if (leftCount >= 0 && refIndex < leftCount) {
+ traceColumnOrigin(leftInput, refIndex);
+ } else if (leftCount >= 0) {
+ traceColumnOrigin(inputNode, refIndex - leftCount);
+ } else {
+ traceColumnOrigin(inputNode, refIndex);
+ }
+ }
+ // Resolve correlated outer-column references. A RexFieldAccess over a
+ // RexCorrelVariable (e.g. $cor0.id inside a correlated subquery) always
+ // refers to an ENCLOSING subquery's row scope, never to inputNode itself
+ // (a reference to inputNode would be a plain RexInputRef).
+ //
+ // Registration happens inside RexRefCollector.visitFieldAccess with
+ // putIfAbsent(varName, enclosingScope), so by the time we reach here the
+ // scope for every $corN we just collected has been bound if possible.
+ // Lookup is a direct map get(); the uncommon case of an unbound variable
+ // falls back (safely over-approximating) to enclosingScope before giving
+ // up — tracing an extra scope is harmless and keeps us on the secure side
+ // of the check.
+ for (CorrelFieldAccess cfa : correlAccesses) {
+ RelNode scope = correlationScopeByVar.get(cfa.varName);
+ if (scope != null) {
+ traceColumnOrigin(scope, cfa.fieldIndex);
+ continue;
+ }
+ if (enclosingScope != null) {
+ logger.debug("Correlated reference '{}' was not pre-bound; "
+ + "falling back to current enclosingScope for a safe
over-approximation.",
+ cfa.varName);
+ traceColumnOrigin(enclosingScope, cfa.fieldIndex);
+ } else {
+ logger.warn("Unresolved correlated column reference '{}' (field index
{}): "
Review Comment:
`LogicalCorrelate` (LATERAL/UNNEST) isn't handled, so these references land
here and are only logged.
```sql
SELECT u.c FROM (SELECT pub, secret_arr FROM t WHERE pub > 0) x,
UNNEST(x.secret_arr) u(c)
```
returns `secret_arr` contents unchecked.
Fix: add a `visit(LogicalCorrelate)` that binds its `CorrelationId` to the
left input, and fail closed (throw) here instead of warning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
