maytasm commented on code in PR #20687:
URL: https://github.com/apache/druid/pull/20687#discussion_r4352941185
##########
server/src/main/java/org/apache/druid/server/http/LookupCoordinatorResource.java:
##########
@@ -70,13 +71,15 @@
* Contains information about lookups exposed through the coordinator
*/
@Path("/druid/coordinator/v1/lookups")
-@ResourceFilters(ConfigResourceFilter.class)
+@ResourceFilters(LookupResourceFilter.class)
public class LookupCoordinatorResource
{
private static final Logger LOG = new
Logger(LookupCoordinatorResource.class);
private final LookupCoordinatorManager lookupCoordinatorManager;
private final ObjectMapper smileMapper;
private final ObjectMapper jsonMapper;
+ @Context
+ private HttpServletRequest request;
Review Comment:
nit: Many existing methods already took `@Context` HttpServletRequest req as
a parameter. Injecting it as a parameter in the GET methods too would match the
rest of the file and most Druid resources.
##########
server/src/main/java/org/apache/druid/server/http/security/LookupResourceFilter.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.http.security;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import com.google.inject.Injector;
+import com.sun.jersey.spi.container.ContainerRequest;
+import org.apache.druid.discovery.LookupNodeService;
+import org.apache.druid.query.lookup.LookupsState;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthorizationResult;
+import org.apache.druid.server.security.AuthorizationUtils;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.server.security.ForbiddenException;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MultivaluedMap;
+import javax.ws.rs.core.UriInfo;
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+/**
+ * Use this resource filter for API endpoints that contain {@code lookups} in
their request path.
+ *
+ * The resource name is {@code tier/lookup}, or {@code tier} when the tier has
no lookups.
+ * Coordinator-wide requests with no tier or lookup use {@code LOOKUP:LOOKUP}.
+ * When lookup authorization is disabled, all requests use {@code
CONFIG:CONFIG}.
+ *
+ * Here are some example paths where this filter is used -
+ * - druid/coordinator/v1/lookups/config/{tier}/{lookup}
+ * - druid/coordinator/v1/lookups/status/{tier}/{lookup}
+ * - druid/coordinator/v1/lookups/config/{tier}
+ * - druid/coordinator/v1/lookups/config/all
+ */
+public class LookupResourceFilter extends AbstractResourceFilter
+{
+ private static final String BULK_FILTER =
LookupResourceFilter.class.getName() + ".bulkFilter";
Review Comment:
nit: Could we follow the SupervisorResource pattern instead? The filter
would handle only the path-param routes. Bulk endpoints would inject
AuthorizerMapper and call AuthorizationUtils.filterAuthorizedResources /
authorizeAllResourceActions directly with explicit (tier, lookup) resources.
That removes the request-attribute handoff, the instanceof heuristics, the
static injector lookup and the ForbiddenException rethrows, and the enabled
path becomes testable.
##########
docs/operations/security-user-auth.md:
##########
@@ -85,6 +86,16 @@ Resource names for this type are datasource names.
Specifying a datasource permi
Any authenticated user can read the contents of any
[lookup](../querying/lookups.md). Avoid storing sensitive data in lookups.
:::
+### `LOOKUP`
+
+When `druid.auth.enableLookupAuthorization` is true, lookup endpoints
authorize against `LOOKUP` resources named `tier/lookup`. The tier comes from
the URL, bulk request body, coordinator lookup state or the serving node.
`READ` covers GET requests, including lookup configuration, status, node status
and introspection. `WRITE` covers POST, PUT and DELETE requests.
+
+This setting applies to lookup HTTP endpoints. It does not currently change
lookup use in SQL or native queries, i.e., query execution does not check
`LOOKUP` permissions.
Review Comment:
Just thinking ahead, when we have enforcement for LOOKUP use in SQL or
native query, will we be adding another new flag?
##########
docs/operations/security-user-auth.md:
##########
@@ -85,6 +86,16 @@ Resource names for this type are datasource names.
Specifying a datasource permi
Any authenticated user can read the contents of any
[lookup](../querying/lookups.md). Avoid storing sensitive data in lookups.
:::
+### `LOOKUP`
Review Comment:
can you also mention that for clusters using druid basic security, they will
need an upgrade step that existing admin/druid_system needs LOOKUP READ/WRITE
on all lookups (e.g. .* in basic-security) before enabling the flag?
(The druid_system role is seeded with makeSuperUserPermissions() only when
it's first created, so existing clusters won't have LOOKUP grants even though
LOOKUP is now in knownTypes())
##########
server/src/main/java/org/apache/druid/server/http/LookupCoordinatorResource.java:
##########
@@ -618,7 +635,7 @@ public Response getSpecificNodeStatus(
.entity(ServletResourceUtils.jsonize("Node [%s] status
is unknown.", hostAndPort))
.build();
} else {
- return Response.ok(lookupsState).build();
+ return Response.ok(LookupResourceFilter.filterBulk(request, tier,
lookupsState)).build();
Review Comment:
getSpecificNodeStatus authorizes the host's lookups against the {tier} in
the URL but never checks that hostAndPort belongs to that tier. A user with
READ on tierA can request nodeStatus/tierA/<host-in-tierB> and get tierB's
lookup specs back, because every name is built as tierA/<lookup> and passes.
Can we return 404 when the host's announced tier doesn't match {tier}?
##########
server/src/main/java/org/apache/druid/server/http/security/LookupResourceFilter.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.http.security;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import com.google.inject.Injector;
+import com.sun.jersey.spi.container.ContainerRequest;
+import org.apache.druid.discovery.LookupNodeService;
+import org.apache.druid.query.lookup.LookupsState;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthorizationResult;
+import org.apache.druid.server.security.AuthorizationUtils;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.server.security.ForbiddenException;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MultivaluedMap;
+import javax.ws.rs.core.UriInfo;
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+/**
+ * Use this resource filter for API endpoints that contain {@code lookups} in
their request path.
+ *
+ * The resource name is {@code tier/lookup}, or {@code tier} when the tier has
no lookups.
+ * Coordinator-wide requests with no tier or lookup use {@code LOOKUP:LOOKUP}.
+ * When lookup authorization is disabled, all requests use {@code
CONFIG:CONFIG}.
+ *
+ * Here are some example paths where this filter is used -
+ * - druid/coordinator/v1/lookups/config/{tier}/{lookup}
+ * - druid/coordinator/v1/lookups/status/{tier}/{lookup}
+ * - druid/coordinator/v1/lookups/config/{tier}
+ * - druid/coordinator/v1/lookups/config/all
+ */
+public class LookupResourceFilter extends AbstractResourceFilter
+{
+ private static final String BULK_FILTER =
LookupResourceFilter.class.getName() + ".bulkFilter";
+
+ private final boolean enabled;
+ private final Injector injector;
+
+ @Context
+ UriInfo uriInfo;
+
+ @Inject
+ public LookupResourceFilter(AuthorizerMapper authorizerMapper, AuthConfig
authConfig, Injector injector)
+ {
+ super(authorizerMapper);
+ this.enabled = authConfig.isEnableLookupAuthorization();
+ this.injector = injector;
+ }
+
+ @Override
+ public ContainerRequest filter(ContainerRequest request)
+ {
+ if (!enabled) {
+ authorize(ImmutableList.of(new ResourceAction(new Resource("CONFIG",
ResourceType.CONFIG), getAction(request))));
+ return request;
+ }
+ final MultivaluedMap<String, String> pathParams =
uriInfo.getPathParameters();
+ final String lookup = lookupPathParam(pathParams);
+ if (lookup == null) {
+ // Bulk routes provide their lookup names after reading the request body
or response data.
+ getReq().setAttribute(BULK_FILTER, this);
+ } else {
+ authorize(pathParams.getFirst("tier"), lookup, getAction(request));
+ }
+ return request;
+ }
+
+ private static String lookupPathParam(MultivaluedMap<String, String>
pathParams)
+ {
+ for (final String param : ImmutableList.of("lookup", "lookupId", "id")) {
+ if (pathParams.containsKey(param)) {
+ return pathParams.getFirst(param);
+ }
+ }
+ return null;
+ }
+
+ public static <T> T filterBulk(HttpServletRequest request, T value)
+ {
+ return filterBulk(request, null, value);
+ }
+
+ /**
+ * Loads a tier's current lookups only when bulk authorization is active for
this request.
+ */
+ public static void filterBulk(HttpServletRequest request, String tier,
Supplier<?> value)
+ {
+ if (request != null && request.getAttribute(BULK_FILTER) != null) {
+ filterBulk(request, tier, value.get());
+ }
+ }
+
+ /**
+ * Checks every lookup in a bulk write or returns only entries allowed by
{@link Action#READ} for a GET.
+ * An empty value uses the tier name, or {@link ResourceType#LOOKUP} when no
tier is available.
+ */
+ public static <T> T filterBulk(HttpServletRequest request, String tier, T
value)
+ {
+ final LookupResourceFilter filter = request == null ? null :
(LookupResourceFilter) request.getAttribute(BULK_FILTER);
+ if (filter == null) {
+ return value;
+ }
+ final String method = request.getMethod();
+ final Action action = "GET".equals(method) || "HEAD".equals(method) ?
Action.READ : Action.WRITE;
+ final String targetTier = tier == null && value instanceof LookupsState
+ ?
filter.injector.getInstance(LookupNodeService.class).getLookupTier()
+ : tier;
+ final Set<String> names = new LinkedHashSet<>();
+ addNames(names, targetTier, value);
+ if (names.isEmpty()) {
+ names.add(name(targetTier, null));
+ }
+ if ("GET".equals(method)) {
Review Comment:
Are we missing HEAD? Should this check action == Action.READ instead of the
method string?
##########
server/src/main/java/org/apache/druid/query/lookup/LookupListeningResource.java:
##########
@@ -89,13 +93,21 @@ public Response handleUpdates(InputStream inputStream,
ObjectMapper mapper)
}
try {
+ final HttpServletRequest request = requestProvider.get();
+ LookupResourceFilter.filterBulk(
+ request,
+ new LookupsState<>(null, state.getToLoad(),
state.getToDrop())
+ );
state.getToLoad().forEach(manager::add);
state.getToDrop().forEach(lookName -> {
manager.remove(lookName,
state.getToLoad().getOrDefault(lookName, null));
});
return
Response.status(Response.Status.ACCEPTED).entity(manager.getAllLookupsState()).build();
Review Comment:
Is this a bug? The response should be READ-filtered the same way getAll is.
##########
server/src/main/java/org/apache/druid/server/http/security/LookupResourceFilter.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.server.http.security;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.inject.Inject;
+import com.google.inject.Injector;
+import com.sun.jersey.spi.container.ContainerRequest;
+import org.apache.druid.discovery.LookupNodeService;
+import org.apache.druid.query.lookup.LookupsState;
+import org.apache.druid.server.security.Action;
+import org.apache.druid.server.security.AuthConfig;
+import org.apache.druid.server.security.AuthorizationResult;
+import org.apache.druid.server.security.AuthorizationUtils;
+import org.apache.druid.server.security.AuthorizerMapper;
+import org.apache.druid.server.security.ForbiddenException;
+import org.apache.druid.server.security.Resource;
+import org.apache.druid.server.security.ResourceAction;
+import org.apache.druid.server.security.ResourceType;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.core.Context;
+import javax.ws.rs.core.MultivaluedMap;
+import javax.ws.rs.core.UriInfo;
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
+
+/**
+ * Use this resource filter for API endpoints that contain {@code lookups} in
their request path.
+ *
+ * The resource name is {@code tier/lookup}, or {@code tier} when the tier has
no lookups.
+ * Coordinator-wide requests with no tier or lookup use {@code LOOKUP:LOOKUP}.
+ * When lookup authorization is disabled, all requests use {@code
CONFIG:CONFIG}.
+ *
+ * Here are some example paths where this filter is used -
+ * - druid/coordinator/v1/lookups/config/{tier}/{lookup}
+ * - druid/coordinator/v1/lookups/status/{tier}/{lookup}
+ * - druid/coordinator/v1/lookups/config/{tier}
+ * - druid/coordinator/v1/lookups/config/all
+ */
+public class LookupResourceFilter extends AbstractResourceFilter
+{
+ private static final String BULK_FILTER =
LookupResourceFilter.class.getName() + ".bulkFilter";
+
+ private final boolean enabled;
+ private final Injector injector;
+
+ @Context
+ UriInfo uriInfo;
+
+ @Inject
+ public LookupResourceFilter(AuthorizerMapper authorizerMapper, AuthConfig
authConfig, Injector injector)
+ {
+ super(authorizerMapper);
+ this.enabled = authConfig.isEnableLookupAuthorization();
+ this.injector = injector;
+ }
+
+ @Override
+ public ContainerRequest filter(ContainerRequest request)
+ {
+ if (!enabled) {
+ authorize(ImmutableList.of(new ResourceAction(new Resource("CONFIG",
ResourceType.CONFIG), getAction(request))));
+ return request;
+ }
+ final MultivaluedMap<String, String> pathParams =
uriInfo.getPathParameters();
+ final String lookup = lookupPathParam(pathParams);
+ if (lookup == null) {
+ // Bulk routes provide their lookup names after reading the request body
or response data.
+ getReq().setAttribute(BULK_FILTER, this);
+ } else {
+ authorize(pathParams.getFirst("tier"), lookup, getAction(request));
+ }
+ return request;
+ }
+
+ private static String lookupPathParam(MultivaluedMap<String, String>
pathParams)
+ {
+ for (final String param : ImmutableList.of("lookup", "lookupId", "id")) {
+ if (pathParams.containsKey(param)) {
+ return pathParams.getFirst(param);
+ }
+ }
+ return null;
+ }
+
+ public static <T> T filterBulk(HttpServletRequest request, T value)
+ {
+ return filterBulk(request, null, value);
+ }
+
+ /**
+ * Loads a tier's current lookups only when bulk authorization is active for
this request.
+ */
+ public static void filterBulk(HttpServletRequest request, String tier,
Supplier<?> value)
+ {
+ if (request != null && request.getAttribute(BULK_FILTER) != null) {
+ filterBulk(request, tier, value.get());
+ }
+ }
+
+ /**
+ * Checks every lookup in a bulk write or returns only entries allowed by
{@link Action#READ} for a GET.
+ * An empty value uses the tier name, or {@link ResourceType#LOOKUP} when no
tier is available.
+ */
+ public static <T> T filterBulk(HttpServletRequest request, String tier, T
value)
+ {
+ final LookupResourceFilter filter = request == null ? null :
(LookupResourceFilter) request.getAttribute(BULK_FILTER);
+ if (filter == null) {
+ return value;
+ }
+ final String method = request.getMethod();
+ final Action action = "GET".equals(method) || "HEAD".equals(method) ?
Action.READ : Action.WRITE;
+ final String targetTier = tier == null && value instanceof LookupsState
+ ?
filter.injector.getInstance(LookupNodeService.class).getLookupTier()
+ : tier;
+ final Set<String> names = new LinkedHashSet<>();
+ addNames(names, targetTier, value);
+ if (names.isEmpty()) {
+ names.add(name(targetTier, null));
+ }
+ if ("GET".equals(method)) {
+ return readable(filter.readableNames(request, names), targetTier, value);
+ }
+ filter.authorize(Iterables.transform(names, target -> lookupAction(target,
action)));
+ return value;
+ }
+
+ /**
+ * Filters a tier listing using known lookup names and tier grants where no
lookups are known.
+ */
+ public static Set<String> filterBulk(
+ HttpServletRequest request,
+ Set<String> tiers,
+ Map<String, ? extends Map<String, ?>> knownLookups
+ )
+ {
+ final LookupResourceFilter filter = request == null ? null :
(LookupResourceFilter) request.getAttribute(BULK_FILTER);
+ if (filter == null) {
+ return tiers;
+ }
+ final Set<String> names = new LinkedHashSet<>();
+ addNames(names, null, knownLookups);
+ tiers.forEach(tier -> {
+ if (knownLookups == null || !knownLookups.containsKey(tier)) {
+ names.add(tier);
+ }
+ });
+ final Set<String> readable = filter.readableNames(request, names);
+ final Set<String> filtered = new LinkedHashSet<>();
+ tiers.forEach(tier -> {
+ if (canReadTier(readable, tier)) {
+ filtered.add(tier);
+ }
+ });
+ return filtered;
+ }
+
+ private Set<String> readableNames(HttpServletRequest request, Set<String>
names)
+ {
+ if (names.isEmpty()) {
+ names.add(name(null, null));
+ }
+ return ImmutableSet.copyOf(AuthorizationUtils.filterAuthorizedResources(
+ request,
+ names,
+ target -> ImmutableList.of(lookupAction(target, Action.READ)),
+ getAuthorizerMapper()
+ ));
+ }
+
+ private void authorize(Iterable<ResourceAction> actions)
+ {
+ final AuthorizationResult authResult =
AuthorizationUtils.authorizeAllResourceActions(
+ getReq(), actions, getAuthorizerMapper()
+ );
+ if (!authResult.allowAccessWithNoRestriction()) {
+ throw new ForbiddenException(authResult.getErrorMessage());
+ }
+ }
+
+ /**
+ * Uses the serving node's tier for lookup routes without a tier path
parameter.
+ */
+ private void authorize(String tier, String lookup, Action action)
+ {
+ final String targetTier = tier == null ?
injector.getInstance(LookupNodeService.class).getLookupTier() : tier;
+ authorize(ImmutableList.of(lookupAction(name(targetTier, lookup),
action)));
+ }
+
+ private static ResourceAction lookupAction(String name, Action action)
+ {
+ return new ResourceAction(new Resource(name, ResourceType.LOOKUP), action);
+ }
+
+ private static String name(String tier, String lookup)
+ {
+ return tier == null ? ResourceType.LOOKUP : lookup == null ? tier : tier +
"/" + lookup;
Review Comment:
Resource names built as tier + "/" + lookup are ambiguous. Lookup x/y in
tier a and lookup y in tier a/x both map to LOOKUP:a/x/y, and canReadTier's
startsWith(tier + "/") makes tier a readable to anyone with a grant in tier
a/b. The lookup API accepts / in both names today, so could we escape each part
(e.g. URL-encode) before joining?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
