[
https://issues.apache.org/jira/browse/SENTRY-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981937#comment-15981937
]
Na Li edited comment on SENTRY-1703 at 4/25/17 5:55 AM:
--------------------------------------------------------
I Had conference with [~wldata], the summary is below
1. The authorization granularity of Solr is at collection, not individual
document.
2. All documents in a Solr request belong to the same collection. So the
authorization response for one document is the same as the other document in
the same Solr request.
If the above assumptions are true, we have two approaches to improve the
performance
a) When Solr server processes the documents of the same Solr request, it can
get authorization response of the first request, and then reuse for the other
documents in the same request. This approach has security risk. If the Solr
client puts documents from different collections in the same request, user may
be able to access documents that should not be accessed by that user. Using
persisted connections at sentry client is a better approach
or
b) Enable pooled connections in sentry client or even cache the authorization
response using [subject, action, collection name] as key.
was (Author: linaataustin):
I Had conference with [~wldata], the summary is below
1. The authorization granularity of Solr is at collection, not individual
document.
2. All documents in a Solr request belong to the same collection. So the
authorization response for one document is the same as the other document in
the same Solr request.
If the above assumptions are true, we have two approaches to improve the
performance
a) When Solr server processes the documents of the same Solr request, it can
get authorization response of the first request, and then reuse for the other
documents in the same request.
or
b) Enable pooled connections in sentry client or even cache the authorization
response using [subject, action, collection name] as key.
> Solr-Sentry in kerberos mode makes too many KDC requests and returns
> unauthorized on KDC timeout
> ------------------------------------------------------------------------------------------------
>
> Key: SENTRY-1703
> URL: https://issues.apache.org/jira/browse/SENTRY-1703
> Project: Sentry
> Issue Type: Bug
> Components: Solr Plugin
> Affects Versions: 1.5.1
> Reporter: Tushar I
> Priority: Blocker
> Attachments: kdc.log.txt, solr-sentry-test-master.zip,
> stacktrace1.log.txt, stacktrace2.log.txt, stacktrace3.log.txt
>
>
> Sentry Version: 1.5.1-cdh5.8.0
> We are seeing intermittent authorization failures with Sentry Solr plugin in
> a Kerberos environment.
> 1. We are writing to Solr using the SolrJ client from within Spark jobs in a
> multi-node Spark/Hadoop cluster and frequently get authorization errors from
> Solr in individual spark tasks saying "User XX does not have privileges for
> YYcollection" which are generated by the Solr-Sentry plugin. (The user very
> well has access to the collection and it works fine rest of the times).
> 2. The root cause seems to be that on every Solr call from the client, Sentry
> reaches out to KDC on behalf of solr/hostname, thereby drowning the KDC in
> tons of requests per second, and at some point fails on a KDC timeout,
> throwing the exception:
> {{org.apache.sentry.binding.solr.authz.SentrySolrAuthorizationException: User
> XX does not have privileges for YYcollection}} to the calling client.
> I didn't get enough time to investigate why Sentry is making so many KDC
> calls, maybe it's doing it for each document in a batched Solr operation, or
> it logs in using keytab each time and doesn't cache the ticket, etc.
> Caching the result of {{authProvider.hasAccess()}} in SolrAuthzBinding.java
> for a reasonably short time might not be a bad idea.
> My question in the meantime is: Are there any tuning knobs to somehow reduce
> the load on KDC, or increase the KDC request timeout value, or anything along
> these lines?
> Relevant stacktraces captured from Solr Admin are attached:
> 1. stacktrace1.log : The timeout from KDC for sentry call
> 2. stacktrace2.log: When Sentry cannot authenticate with KDC due to # 1 above
> 3. stacktrace3.log: SolrException when {{authProvider.hasAccess()}} returns
> false due to # 2 above.
> Also attached is a _snippet_ from the KDC log - the full log bloats to 17 MB
> within a minute, full of messages like:
> {code}
> Apr 10 17:06:37 a0 krb5kdc[20427](info): TGS_REQ (1 etypes {23}) 10.0.0.1:
> ISSUE: authtime 1491818430, etypes {rep=23 tkt=23 ses=23},
> solr/[email protected] for sentry/[email protected]
> {code}
> This is reproducible in two separate clusters with different environments:
> CDH 5.10.1 and
> CDH 5.8.0
> Please let me know if I've left out any key information.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
