github-actions[bot] commented on code in PR #68220:
URL: https://github.com/apache/doris/pull/68220#discussion_r4068643753
##########
fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-hive/src/main/java/org/apache/doris/catalog/authorizer/ranger/hive/RangerHiveAccessController.java:
##########
@@ -343,6 +343,9 @@ private RangerAccessRequestImpl
createRequest(AuthorizedSubject subject, Set<Str
AccessContext context) {
RangerAccessRequestImpl request = new RangerAccessRequestImpl();
request.setUser(subject.getUser());
+ // Ranger's groups for the user, out of the user store this service's
plugin downloads; a Hive
+ // service's policies are usually kept by group, and Doris has no
groups of its own to send.
+ request.setUserGroups(groupsOf(subject));
Review Comment:
[P1] Snapshot groups once per logical authorization, as this method already
does for roles. `createRequest()` rereads and copies the asynchronously
refreshed user store for every column. If a refresh moves the user from group A
to B between two iterations, with A granting only c1 and B granting only c2,
this one batch carries c1/A plus c2/B and grants a column set no single
membership snapshot permits. This is distinct from the existing publication
thread: both snapshots may be correctly published. Please capture one immutable
group set before the loop, pass it to every request (and the analogous
Doris/data-policy loops), and cover a refresh barrier during a multi-column
check.
##########
fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-common/src/main/java/org/apache/doris/catalog/authorizer/ranger/RangerUserStoreGroups.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog.authorizer.ranger;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.ranger.authorization.hadoop.config.RangerPluginConfig;
+import org.apache.ranger.plugin.contextenricher.RangerAdminUserStoreRetriever;
+import org.apache.ranger.plugin.contextenricher.RangerUserStoreEnricher;
+import org.apache.ranger.plugin.policyengine.RangerPluginContext;
+import org.apache.ranger.plugin.service.RangerAuthContext;
+import org.apache.ranger.plugin.service.RangerBasePlugin;
+import org.apache.ranger.plugin.util.RangerUserStoreUtil;
+import org.apache.ranger.plugin.util.ServiceDefUtil;
+import org.apache.ranger.plugin.util.ServicePolicies;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Where a request built by a Doris source gets the groups a Ranger policy
item may be written against.
+ *
+ * <p>A Ranger policy item names users, groups and roles, and Ranger's own
plugins fill in the groups from
+ * the service they run in - the Hive plugin asks Hadoop's group mapping.
Doris has nothing to ask: an
+ * account holds roles, and no part of the engine knows which groups a user is
in. So a request that carries
+ * no groups matches no group item at all, and a service whose policies are
kept by group - the normal way
+ * to run Ranger, so that nobody edits a policy each time somebody joins a
team - grants nothing to a Doris
+ * user; and, worse, denies nothing either. An item that denies a group is
exactly as silent about it.
+ *
+ * <p>What Ranger provides for a plugin in that position is its <b>user
store</b>: the users and groups Ranger
+ * Admin itself holds, kept current by usersync from LDAP or the OS, which a
plugin downloads next to its
+ * policies through a {@link RangerUserStoreEnricher}. Ranger 2.5 and later
reads the groups of the request's
+ * user out of it when {@code ranger.plugin.<type>.use.rangerGroups} is set;
the two sources Doris embeds do
+ * the same thing here, in the request builder, so that it neither depends on
the Ranger version they were
+ * built against - branches still on 2.4 have no such setting - nor on an
operator finding a Ranger property
+ * no Doris document names.
+ *
+ * <p>It is on by default, because a group item that is silently ignored is a
bug in the deployment's eyes,
+ * not a behaviour anybody opted into; the property Ranger reads for this is
the one that switches it off,
+ * so that a deployment which has already decided the question in Ranger's
terms has decided it here too.
+ * Switched off, requests carry no groups and this source matches what it
matched before.
+ */
+public final class RangerUserStoreGroups {
+ private static final Logger LOG =
LogManager.getLogger(RangerUserStoreGroups.class);
+
+ /**
+ * Suffix of the property switching this off, {@code
ranger.plugin.<type>.use.rangerGroups} in full: the
+ * name Ranger itself reads for the same thing, so that one setting
decides both.
+ */
+ public static final String USE_RANGER_GROUPS = ".use.rangerGroups";
+
+ /** How often the plugin asks Ranger Admin for a newer user store, when
nothing configures it. */
+ private static final String DEFAULT_REFRESH_INTERVAL_MS =
Integer.toString(60 * 1000);
+
+ private RangerUserStoreGroups() {
+ }
+
+ /** Whether the plugin configured by {@code config} attaches user store
groups; on unless switched off. */
+ public static boolean enabledFor(RangerPluginConfig config) {
+ // No configuration at all - a plugin stubbed out in a test - has not
switched anything off.
+ return config == null || config.getBoolean(config.getPropertyPrefix()
+ USE_RANGER_GROUPS, true);
Review Comment:
[P1] Reject malformed values for this authorization opt-out. Hadoop's
`Configuration.getBoolean()` returns its default for any value other than
`true` or `false`, so `ranger.plugin.<type>.use.rangerGroups=flase` silently
evaluates to `true` here and activates group allows/denies/masks/filters after
an operator attempted the compatibility opt-out. Ranger itself cached the same
malformed setting with default `false`, so the two consumers also disagree.
Please parse the raw value strictly (missing means true; only case-insensitive
true/false are accepted), fail construction otherwise, and add an invalid-value
test.
##########
fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-common/src/main/java/org/apache/doris/catalog/authorizer/ranger/RangerUserStoreGroups.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog.authorizer.ranger;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.ranger.authorization.hadoop.config.RangerPluginConfig;
+import org.apache.ranger.plugin.contextenricher.RangerAdminUserStoreRetriever;
+import org.apache.ranger.plugin.contextenricher.RangerUserStoreEnricher;
+import org.apache.ranger.plugin.policyengine.RangerPluginContext;
+import org.apache.ranger.plugin.service.RangerAuthContext;
+import org.apache.ranger.plugin.service.RangerBasePlugin;
+import org.apache.ranger.plugin.util.RangerUserStoreUtil;
+import org.apache.ranger.plugin.util.ServiceDefUtil;
+import org.apache.ranger.plugin.util.ServicePolicies;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Where a request built by a Doris source gets the groups a Ranger policy
item may be written against.
+ *
+ * <p>A Ranger policy item names users, groups and roles, and Ranger's own
plugins fill in the groups from
+ * the service they run in - the Hive plugin asks Hadoop's group mapping.
Doris has nothing to ask: an
+ * account holds roles, and no part of the engine knows which groups a user is
in. So a request that carries
+ * no groups matches no group item at all, and a service whose policies are
kept by group - the normal way
+ * to run Ranger, so that nobody edits a policy each time somebody joins a
team - grants nothing to a Doris
+ * user; and, worse, denies nothing either. An item that denies a group is
exactly as silent about it.
+ *
+ * <p>What Ranger provides for a plugin in that position is its <b>user
store</b>: the users and groups Ranger
+ * Admin itself holds, kept current by usersync from LDAP or the OS, which a
plugin downloads next to its
+ * policies through a {@link RangerUserStoreEnricher}. Ranger 2.5 and later
reads the groups of the request's
+ * user out of it when {@code ranger.plugin.<type>.use.rangerGroups} is set;
the two sources Doris embeds do
+ * the same thing here, in the request builder, so that it neither depends on
the Ranger version they were
+ * built against - branches still on 2.4 have no such setting - nor on an
operator finding a Ranger property
+ * no Doris document names.
+ *
+ * <p>It is on by default, because a group item that is silently ignored is a
bug in the deployment's eyes,
+ * not a behaviour anybody opted into; the property Ranger reads for this is
the one that switches it off,
+ * so that a deployment which has already decided the question in Ranger's
terms has decided it here too.
+ * Switched off, requests carry no groups and this source matches what it
matched before.
+ */
+public final class RangerUserStoreGroups {
+ private static final Logger LOG =
LogManager.getLogger(RangerUserStoreGroups.class);
+
+ /**
+ * Suffix of the property switching this off, {@code
ranger.plugin.<type>.use.rangerGroups} in full: the
+ * name Ranger itself reads for the same thing, so that one setting
decides both.
+ */
+ public static final String USE_RANGER_GROUPS = ".use.rangerGroups";
+
+ /** How often the plugin asks Ranger Admin for a newer user store, when
nothing configures it. */
+ private static final String DEFAULT_REFRESH_INTERVAL_MS =
Integer.toString(60 * 1000);
+
+ private RangerUserStoreGroups() {
+ }
+
+ /** Whether the plugin configured by {@code config} attaches user store
groups; on unless switched off. */
+ public static boolean enabledFor(RangerPluginConfig config) {
+ // No configuration at all - a plugin stubbed out in a test - has not
switched anything off.
+ return config == null || config.getBoolean(config.getPropertyPrefix()
+ USE_RANGER_GROUPS, true);
+ }
+
+ /**
+ * Puts a user store enricher on the service definition Ranger just
downloaded, unless one is there or
+ * this is switched off, so that the plugin fetches and refreshes the user
store {@link #groupsOf} reads.
+ *
+ * <p>Called by {@link LoadedRangerPlugin#setPolicies} before handing the
policies on, on every
+ * call and not only the first: a download of policy deltas comes with its
own copy of the service
+ * definition, which is why {@code RangerBasePlugin} re-adds the enricher
on deltas too. The retriever
+ * class and the refresh interval are read under the option names Ranger
itself reads them under, so that
+ * an operator who has tuned them for Ranger's own {@code
use.rangerGroups} has tuned them here.
+ */
+ public static void addUserStoreEnricher(RangerPluginConfig config,
ServicePolicies policies) {
+ if (policies == null || config == null || !enabledFor(config)) {
+ return;
+ }
+ String retriever =
config.get(RangerUserStoreEnricher.USERSTORE_RETRIEVER_CLASSNAME_OPTION,
+ RangerAdminUserStoreRetriever.class.getCanonicalName());
+ String refreshIntervalMs =
config.get(RangerUserStoreEnricher.USERSTORE_REFRESHER_POLLINGINTERVAL_OPTION,
Review Comment:
[P2] Validate this interval as strictly positive before adding the enricher.
Ranger 2.8 accepts `0`/negative through `Long.parseLong`, performs the initial
download, starts `RangerUserStoreRefresher`, creates a `Timer`, and only then
calls `Timer.schedule()`, which throws `IllegalArgumentException` for a
non-positive period. `setPolicies()` catches that before publishing the engine,
so `LoadedRangerPlugin.cleanup()` cannot reach the partially initialized
enricher; its refresher remains blocked forever, and each failed ranger-hive
bind retry leaks another daemon thread. Please reject non-positive values
before Ranger init, ensure partial enricher construction is cleaned up, and add
a `0` interval test that verifies the refresher terminates.
##########
fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-common/src/main/java/org/apache/doris/catalog/authorizer/ranger/RangerUserStoreGroups.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog.authorizer.ranger;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.ranger.authorization.hadoop.config.RangerPluginConfig;
+import org.apache.ranger.plugin.contextenricher.RangerAdminUserStoreRetriever;
+import org.apache.ranger.plugin.contextenricher.RangerUserStoreEnricher;
+import org.apache.ranger.plugin.policyengine.RangerPluginContext;
+import org.apache.ranger.plugin.service.RangerAuthContext;
+import org.apache.ranger.plugin.service.RangerBasePlugin;
+import org.apache.ranger.plugin.util.RangerUserStoreUtil;
+import org.apache.ranger.plugin.util.ServiceDefUtil;
+import org.apache.ranger.plugin.util.ServicePolicies;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Where a request built by a Doris source gets the groups a Ranger policy
item may be written against.
+ *
+ * <p>A Ranger policy item names users, groups and roles, and Ranger's own
plugins fill in the groups from
+ * the service they run in - the Hive plugin asks Hadoop's group mapping.
Doris has nothing to ask: an
+ * account holds roles, and no part of the engine knows which groups a user is
in. So a request that carries
+ * no groups matches no group item at all, and a service whose policies are
kept by group - the normal way
+ * to run Ranger, so that nobody edits a policy each time somebody joins a
team - grants nothing to a Doris
+ * user; and, worse, denies nothing either. An item that denies a group is
exactly as silent about it.
+ *
+ * <p>What Ranger provides for a plugin in that position is its <b>user
store</b>: the users and groups Ranger
+ * Admin itself holds, kept current by usersync from LDAP or the OS, which a
plugin downloads next to its
+ * policies through a {@link RangerUserStoreEnricher}. Ranger 2.5 and later
reads the groups of the request's
+ * user out of it when {@code ranger.plugin.<type>.use.rangerGroups} is set;
the two sources Doris embeds do
+ * the same thing here, in the request builder, so that it neither depends on
the Ranger version they were
+ * built against - branches still on 2.4 have no such setting - nor on an
operator finding a Ranger property
+ * no Doris document names.
+ *
+ * <p>It is on by default, because a group item that is silently ignored is a
bug in the deployment's eyes,
Review Comment:
[P2] Add the promised operator documentation before making this default-on.
This changes effective allows, denies, row filters, and masks on upgrade, adds
opt-out/refresh settings for both plugins, and introduces a new
FE-start/catalog-bind failure contract, but this PR contains no docs change.
The linked #68203 is closed and was also code-only. Please update or link the
actual `ranger.md` change, including both exact opt-out keys, refresh tuning,
the no-policy/cache prerequisite, and mixed-version rolling-upgrade behavior.
##########
fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-hive/src/main/java/org/apache/doris/catalog/authorizer/ranger/hive/RangerHiveAccessController.java:
##########
@@ -343,6 +343,9 @@ private RangerAccessRequestImpl
createRequest(AuthorizedSubject subject, Set<Str
AccessContext context) {
RangerAccessRequestImpl request = new RangerAccessRequestImpl();
request.setUser(subject.getUser());
+ // Ranger's groups for the user, out of the user store this service's
plugin downloads; a Hive
Review Comment:
[P1] Merge Ranger-derived roles instead of letting unrelated Doris roles
suppress them. This request adds the Ranger groups here and then sets Doris
roles on the next line, but Ranger 2.8 derives roles from `user/groups` only
when the incoming role set is empty. A user holding Doris role `reporter` and
Ranger group `analysts` therefore never receives Ranger role `data_reader`
granted to that group; role-based denies can be omitted just like allows. The
new Hive test masks this by forcing `rolesOf()` to return empty. After the same
normalization required above, merge Ranger's `getRolesFromUserAndGroups()`
result with the Doris roles and add a
nonempty-Doris-role/group-granted-Ranger-role test.
##########
fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-common/src/main/java/org/apache/doris/catalog/authorizer/ranger/RangerUserStoreGroups.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog.authorizer.ranger;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.ranger.authorization.hadoop.config.RangerPluginConfig;
+import org.apache.ranger.plugin.contextenricher.RangerAdminUserStoreRetriever;
+import org.apache.ranger.plugin.contextenricher.RangerUserStoreEnricher;
+import org.apache.ranger.plugin.policyengine.RangerPluginContext;
+import org.apache.ranger.plugin.service.RangerAuthContext;
+import org.apache.ranger.plugin.service.RangerBasePlugin;
+import org.apache.ranger.plugin.util.RangerUserStoreUtil;
+import org.apache.ranger.plugin.util.ServiceDefUtil;
+import org.apache.ranger.plugin.util.ServicePolicies;
+
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Where a request built by a Doris source gets the groups a Ranger policy
item may be written against.
+ *
+ * <p>A Ranger policy item names users, groups and roles, and Ranger's own
plugins fill in the groups from
+ * the service they run in - the Hive plugin asks Hadoop's group mapping.
Doris has nothing to ask: an
+ * account holds roles, and no part of the engine knows which groups a user is
in. So a request that carries
+ * no groups matches no group item at all, and a service whose policies are
kept by group - the normal way
+ * to run Ranger, so that nobody edits a policy each time somebody joins a
team - grants nothing to a Doris
+ * user; and, worse, denies nothing either. An item that denies a group is
exactly as silent about it.
+ *
+ * <p>What Ranger provides for a plugin in that position is its <b>user
store</b>: the users and groups Ranger
+ * Admin itself holds, kept current by usersync from LDAP or the OS, which a
plugin downloads next to its
+ * policies through a {@link RangerUserStoreEnricher}. Ranger 2.5 and later
reads the groups of the request's
+ * user out of it when {@code ranger.plugin.<type>.use.rangerGroups} is set;
the two sources Doris embeds do
+ * the same thing here, in the request builder, so that it neither depends on
the Ranger version they were
+ * built against - branches still on 2.4 have no such setting - nor on an
operator finding a Ranger property
+ * no Doris document names.
+ *
+ * <p>It is on by default, because a group item that is silently ignored is a
bug in the deployment's eyes,
+ * not a behaviour anybody opted into; the property Ranger reads for this is
the one that switches it off,
+ * so that a deployment which has already decided the question in Ranger's
terms has decided it here too.
+ * Switched off, requests carry no groups and this source matches what it
matched before.
+ */
+public final class RangerUserStoreGroups {
+ private static final Logger LOG =
LogManager.getLogger(RangerUserStoreGroups.class);
+
+ /**
+ * Suffix of the property switching this off, {@code
ranger.plugin.<type>.use.rangerGroups} in full: the
+ * name Ranger itself reads for the same thing, so that one setting
decides both.
+ */
+ public static final String USE_RANGER_GROUPS = ".use.rangerGroups";
+
+ /** How often the plugin asks Ranger Admin for a newer user store, when
nothing configures it. */
+ private static final String DEFAULT_REFRESH_INTERVAL_MS =
Integer.toString(60 * 1000);
+
+ private RangerUserStoreGroups() {
+ }
+
+ /** Whether the plugin configured by {@code config} attaches user store
groups; on unless switched off. */
+ public static boolean enabledFor(RangerPluginConfig config) {
+ // No configuration at all - a plugin stubbed out in a test - has not
switched anything off.
+ return config == null || config.getBoolean(config.getPropertyPrefix()
+ USE_RANGER_GROUPS, true);
+ }
+
+ /**
+ * Puts a user store enricher on the service definition Ranger just
downloaded, unless one is there or
+ * this is switched off, so that the plugin fetches and refreshes the user
store {@link #groupsOf} reads.
+ *
+ * <p>Called by {@link LoadedRangerPlugin#setPolicies} before handing the
policies on, on every
+ * call and not only the first: a download of policy deltas comes with its
own copy of the service
+ * definition, which is why {@code RangerBasePlugin} re-adds the enricher
on deltas too. The retriever
+ * class and the refresh interval are read under the option names Ranger
itself reads them under, so that
+ * an operator who has tuned them for Ranger's own {@code
use.rangerGroups} has tuned them here.
+ */
+ public static void addUserStoreEnricher(RangerPluginConfig config,
ServicePolicies policies) {
+ if (policies == null || config == null || !enabledFor(config)) {
+ return;
+ }
+ String retriever =
config.get(RangerUserStoreEnricher.USERSTORE_RETRIEVER_CLASSNAME_OPTION,
+ RangerAdminUserStoreRetriever.class.getCanonicalName());
+ String refreshIntervalMs =
config.get(RangerUserStoreEnricher.USERSTORE_REFRESHER_POLLINGINTERVAL_OPTION,
+ DEFAULT_REFRESH_INTERVAL_MS);
+ // Ranger logs the addition itself, once per download that needed it;
the operator-facing line about
+ // why the store is downloaded at all is the plugin's, written once
when it starts (see describe).
+ if (ServiceDefUtil.addUserStoreEnricher(policies, retriever,
refreshIntervalMs) && LOG.isDebugEnabled()) {
+ LOG.debug("Ranger service {} will download its user store every {}
ms", policies.getServiceName(),
+ refreshIntervalMs);
+ }
+ }
+
+ /** One line for the plugin's start-up log, saying what this does for it
and how to switch it off. */
+ public static String describe(RangerPluginConfig config) {
+ String property = config.getPropertyPrefix() + USE_RANGER_GROUPS;
+ return enabledFor(config)
+ ? "Ranger service " + config.getServiceName() + ": requests
carry the groups Ranger's user store"
+ + " puts the user in, so that policy items written
against a group apply; set "
+ + property + "=false to switch that off"
+ : "Ranger service " + config.getServiceName() + ": " +
property + "=false, so requests carry no"
+ + " groups and policy items written against a group
never apply";
+ }
+
+ /**
+ * The groups the user store {@code plugin} has downloaded puts {@code
user} in.
+ *
+ * <p>Empty when this is switched off, when no store has arrived - Ranger
Admin could not be reached for
+ * it and nothing was cached, which {@link LoadedRangerPlugin} says why it
does not refuse - and when the
+ * store does not know the user, which is the case for every account that
exists in Doris only. Empty
+ * and not null on purpose: a request with an empty group set matches
items written against users and
+ * roles exactly as it did before.
+ */
+ public static Set<String> groupsOf(RangerBasePlugin plugin, String user) {
+ if (user == null || !enabledFor(plugin.getConfig())) {
+ return Collections.emptySet();
+ }
+ // The auth context the policy engine publishes is where Ranger's own
request processing reads the
+ // store from; it is replaced together with the engine, and the store
is carried over when it is.
+ RangerPluginContext pluginContext = plugin.getPluginContext();
+ RangerAuthContext authContext = pluginContext == null ? null :
pluginContext.getAuthContext();
+ RangerUserStoreUtil userStore = authContext == null ? null :
authContext.getUserStoreUtil();
+ Set<String> groups = userStore == null ? null :
userStore.getUserGroups(user);
Review Comment:
[P1] Preserve Ranger's user/group preprocessing order here. Ranger 2.8
transforms the request username (including `convert.emailToUser`) before
looking up store groups and does not transform those canonical groups
afterward. This helper instead queries with the raw Doris username and supplies
the result as caller groups. Thus `[email protected]` can miss a store keyed by
`alice`; Ranger later converts the user, but with `use.rangerGroups` omitted
its cached flag is false and the missing group deny is never repaired. Worse,
Ranger 2.8's case-conversion-only path leaves `groupNameTransformer` null but
calls it when caller groups are nonempty, so this newly nonempty set can make
every request throw. This is distinct from the existing
unavailable-store/publication threads because the store is healthy. Please
populate groups at Ranger's supported post-transformation stage and cover email
plus case-only name transformation.
##########
fe/fe-authorization/fe-authorization-plugins/fe-authorization-plugin-ranger-common/src/main/java/org/apache/doris/catalog/authorizer/ranger/LoadedRangerPlugin.java:
##########
@@ -0,0 +1,130 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.catalog.authorizer.ranger;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.ranger.plugin.service.RangerBasePlugin;
+import org.apache.ranger.plugin.util.ServicePolicies;
+
+import java.util.concurrent.TimeUnit;
+
+/**
+ * What the two Ranger plugins Doris embeds have in common: they are built
with their service's policies or
+ * not at all, and the requests built over them carry the groups Ranger keeps
for a user.
+ *
+ * <p><b>Built with the policies, or not at all.</b> {@code
RangerBasePlugin.init()} downloads the service's
+ * roles and policies - and, with the enricher below on them, its user store -
before it returns, and it
+ * survives a Ranger admin it cannot reach: it answers out of the local policy
cache
+ * ({@code ranger.plugin.<type>.policy.cache.dir}) when there is one, and out
of no policies at all when there
+ * is not, refusing every check until a later poll succeeds. That is the right
thing for a plugin embedded in
+ * a service that has to keep running, and the wrong thing at the moment Doris
binds one: an FE whose instance
+ * scope {@code access_controller_type=ranger-doris} governs, and which has no
policies, is an FE nobody can
+ * use - no account bypasses the source - and a catalog bound to a {@code
ranger-hive} source in that state
+ * refuses every statement against it, in both cases with nothing but a line
in fe.log to say why. So
+ * {@link #init()} refuses that state instead: a load that ended with no
policies, from the admin or the
+ * cache, stops what it started and throws, with the cause. That fails the FE
start, the {@code CREATE
+ * CATALOG} (whose dry run builds the plugin), or the statement binding a
catalog to its source again, which
+ * is where an operator sees it. Once built, an outage of the admin is
Ranger's to survive, as it always
+ * was: the refresher keeps the last policies it downloaded and keeps polling.
+ *
+ * <p>The user store is not part of that. A load that found the policies but
no user store is logged and
+ * accepted - policy items written against a group do not apply until the
store arrives, which the enricher
+ * keeps asking for - because "no store has arrived" cannot be told apart from
"this admin serves none", and
+ * an admin from before the user store download existed, or one that fails
that download while serving the
+ * policies, ran every deployment before groups were attached at all; see
{@link RangerUserStoreGroups}.
+ *
+ * <p><b>Requests carry Ranger's own groups.</b> Doris has none to offer, and
a policy item written against a
+ * group matches nothing without them; the store they are read from is asked
for with the policies, see
+ * {@link #setPolicies} and {@link RangerUserStoreGroups}.
+ */
+public abstract class LoadedRangerPlugin extends RangerBasePlugin {
+ private static final Logger LOG =
LogManager.getLogger(LoadedRangerPlugin.class);
+
+ protected LoadedRangerPlugin(String serviceType, String serviceName,
String appId) {
+ super(serviceType, serviceName, appId);
+ }
+
+ /**
+ * Loads the plugin - {@code RangerBasePlugin.init()}: roles, policies and
user store, on this thread -
+ * and refuses to leave it without policies; see the class comment.
+ *
+ * @throws IllegalStateException when the load ended with no policies from
either the admin or the cache
+ */
+ @Override
+ public void init() {
+ LOG.info(RangerUserStoreGroups.describe(getConfig()));
+ long startedAtNanos = System.nanoTime();
+ try {
+ super.init();
+ if (getPoliciesVersion() < 0) {
Review Comment:
[P2] Do not diagnose every `-1` engine version as an Admin/cache miss.
Ranger 2.8 catches policy-engine construction exceptions inside `setPolicies()`
and returns with the engine null. A concrete new path is
`userStoreRefresherPollingInterval=often`: this PR adds the user-store
enricher, its `init()` throws while parsing the interval, Ranger logs and
swallows that exception, and this branch then reports that Admin was
unreachable/the service was missing and no cache existed. `CREATE CATALOG`
therefore loses the actionable cause promised by the PR and directs the
operator to the wrong repair. Track whether a non-null policy payload failed to
produce an engine (and validate the new interval before init), preserve an
accurate cause/message, and add this failure-path test.
##########
regression-test/suites/ranger_p2/test_ranger_group_policy.groovy:
##########
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.ranger.RangerClient
+import org.apache.ranger.plugin.model.RangerPolicy
+
+// Policies written against a Ranger GROUP and never against the user: how a
deployment runs Ranger so that
+// nobody edits a policy each time somebody joins a team. Doris has no groups
of its own, so these apply
+// only because the ranger-doris source reads the user's groups out of
Ranger's user store - the store the
+// plugin downloads next to its policies. Every other suite here writes its
policy items against a user.
+suite("test_ranger_group_policy", "p2,ranger,external") {
+ String enabled = context.config.otherConfigs.get("enableRangerTest")
+ String rangerEndpoint =
context.config.otherConfigs.get("rangerEndpoint")
+ String rangerUser = context.config.otherConfigs.get("rangerUser")
+ String rangerPassword =
context.config.otherConfigs.get("rangerPassword")
+ String rangerServiceName =
context.config.otherConfigs.get("rangerServiceName")
+
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ return
+ }
+
+ String db = 'ranger_group_db_1'
+ String table = 'ranger_group_tbl_1'
+ // The same database, no policy of any kind: what the group's SELECT
must not open.
+ String otherTable = 'ranger_group_tbl_2'
+ String user = 'ranger_group_user_1'
+ String pwd = 'C123_567p'
+ String group = 'ranger_group_readers_1'
+ String accessPolicy = 'ranger_test_group_access_policy'
+ String rowFilterPolicy = 'ranger_test_group_row_filter_policy'
+ String maskPolicy = 'ranger_test_group_mask_policy'
+ String denyPolicy = 'ranger_test_group_deny_policy'
+ // A table the user is allowed on by name while the group is denied on
it; the deny has to win.
+ String deniedTable = 'ranger_group_tbl_3'
+
+ sql """CREATE DATABASE IF NOT EXISTS ${db}"""
+ [table, otherTable, deniedTable].each {
+ sql """DROP TABLE IF EXISTS ${db}.${it}"""
+ sql """CREATE TABLE ${db}.${it} (
+ id BIGINT,
+ c1 VARCHAR(20),
+ c2 VARCHAR(20)
+ )
+ DISTRIBUTED BY HASH(id) BUCKETS 2
+ PROPERTIES (
+ "replication_num" = "1"
+ )"""
+ sql """INSERT INTO ${db}.${it} VALUES
+ (1, 'DataOne01', 'SampleA1'),
+ (2, 'DataTwo02', 'SampleB2'),
+ (3, 'DataThr03', 'SampleC3'),
+ (4, 'DataFou04', 'SampleD4'),
+ (5, 'DataFiv05', 'SampleE5'),
+ (6, 'DataSix06', 'SampleF6'),
+ (7, 'DataSev07', 'SampleG7'),
+ (8, 'DataEig08', 'SampleH8'),
+ (9, 'DataNin09', 'SampleI9'),
+ (10, 'DataTen10', 'SampleJ0')"""
+ }
+ sql """DROP USER IF EXISTS ${user}"""
+ sql """CREATE USER '${user}' IDENTIFIED BY '${pwd}'"""
+
+ // Ranger's side of the same names: the group, the user, and the
membership. Nothing syncs users between
+ // Doris and Ranger, so the names are simply kept equal, as the other
suites do for users.
+ createRangerGroup(group)
+ createRangerUser(user, pwd, ["ROLE_USER"] as String[])
+ setRangerUserGroups(user, [group])
+
+ RangerClient rangerClient = new
RangerClient("http://${rangerEndpoint}";, "simple", rangerUser, rangerPassword,
null)
+ def dropPolicy = { String name ->
+ try {
+ rangerClient.deletePolicy(rangerServiceName, name)
+ } catch (Exception e) {
+ log.info("Policy not found: ${e.getMessage()}")
+ }
+ }
+ [accessPolicy, rowFilterPolicy, maskPolicy, denyPolicy].each {
dropPolicy(it) }
+
+ def resourcesOf = { String tbl, String column ->
+ Map<String, RangerPolicy.RangerPolicyResource> resources = new
HashMap<>()
+ resources.put("catalog", new
RangerPolicy.RangerPolicyResource("internal"))
+ resources.put("database", new
RangerPolicy.RangerPolicyResource(db))
+ resources.put("table", new
RangerPolicy.RangerPolicyResource(tbl))
+ if (column != null) {
+ resources.put("column", new
RangerPolicy.RangerPolicyResource(column))
+ }
+ return resources
+ }
+ def select = [new RangerPolicy.RangerPolicyItemAccess("SELECT")]
+
+ // 1. Access: SELECT on the table for the group.
+ RangerPolicy policy = new RangerPolicy()
+ policy.setService(rangerServiceName)
+ policy.setName(accessPolicy)
+ policy.setResources(resourcesOf(table, null))
+ RangerPolicy.RangerPolicyItem accessItem = new
RangerPolicy.RangerPolicyItem()
+ accessItem.setGroups([group])
+ accessItem.setAccesses(select)
+ policy.setPolicyItems([accessItem])
+ rangerClient.createPolicy(policy)
+
+ // 2. A row filter for the group.
+ policy = new RangerPolicy()
+ policy.setService(rangerServiceName)
+ policy.setName(rowFilterPolicy)
+ policy.setPolicyType(RangerPolicy.POLICY_TYPE_ROWFILTER)
+ policy.setResources(resourcesOf(table, null))
+ RangerPolicy.RangerRowFilterPolicyItem rowFilterItem = new
RangerPolicy.RangerRowFilterPolicyItem()
+ rowFilterItem.setGroups([group])
+ rowFilterItem.setAccesses(select)
+ rowFilterItem.setRowFilterInfo(new
RangerPolicy.RangerPolicyItemRowFilterInfo("id >= 5"))
+ policy.setRowFilterPolicyItems([rowFilterItem])
+ rangerClient.createPolicy(policy)
+
+ // 3. A column mask for the group.
+ policy = new RangerPolicy()
+ policy.setService(rangerServiceName)
+ policy.setName(maskPolicy)
+ policy.setPolicyType(RangerPolicy.POLICY_TYPE_DATAMASK)
+ policy.setResources(resourcesOf(table, "c1"))
+ RangerPolicy.RangerDataMaskPolicyItem maskItem = new
RangerPolicy.RangerDataMaskPolicyItem()
+ maskItem.setGroups([group])
+ maskItem.setAccesses(select)
+ maskItem.setDataMaskInfo(new
RangerPolicy.RangerPolicyItemDataMaskInfo("MASK_SHOW_LAST_4", null, null))
+ policy.setDataMaskPolicyItems([maskItem])
+ rangerClient.createPolicy(policy)
+
+ // 4. A deny written against the group, on a table the user is allowed
on by name. Before groups were
+ // attached this deny was silently ignored and the user read the table
- the dangerous half of the bug.
+ policy = new RangerPolicy()
+ policy.setService(rangerServiceName)
+ policy.setName(denyPolicy)
+ policy.setResources(resourcesOf(deniedTable, null))
+ RangerPolicy.RangerPolicyItem allowUserItem = new
RangerPolicy.RangerPolicyItem()
+ allowUserItem.setUsers([user])
+ allowUserItem.setAccesses(select)
+ policy.setPolicyItems([allowUserItem])
+ RangerPolicy.RangerPolicyItem denyGroupItem = new
RangerPolicy.RangerPolicyItem()
+ denyGroupItem.setGroups([group])
+ denyGroupItem.setAccesses(select)
+ policy.setDenyPolicyItems([denyGroupItem])
+ rangerClient.createPolicy(policy)
+
+ def tokens = context.config.jdbcUrl.split('/')
+ def defaultJdbcUrl = tokens[0] + "//" + tokens[2] + "/?"
+ def readable = { String tbl ->
+ return connect("${user}", "${pwd}", "${defaultJdbcUrl}") {
+ try {
+ sql """SELECT * FROM internal.${db}.${tbl}"""
+ return true
+ } catch (Exception e) {
+ log.info("not readable yet: ${e.getMessage()}")
+ return false
+ }
+ }
+ }
+
+ // The policies reach the FE within its policy poll interval; the
membership reaches it with the next
+ // user store download, which the plugin makes every 60 seconds unless
userStoreRefresherPollingInterval
+ // in ranger-doris-security.xml says otherwise. Hence waiting on the
effect rather than a fixed sleep.
+ logger.info("waiting for the group's SELECT to reach ${user}")
+ awaitUntil(180, 3) { readable(table) }
Review Comment:
[P2] Wait for the complete policy generation, not only the first access
policy. These four policies are created by separate Ranger Admin calls, so the
FE refresher can poll after the access policy but before the filter, mask, or
deny exists. `readable(table)` then passes, while the one-shot filtered/masked
query and group-deny assertion below fail until the next poll. Please make this
readiness loop prove the final filtered/masked result and denied-table refusal
(or wait on the FE consuming the final policy version) before running the
assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
