[jira] [Commented] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
[ https://issues.apache.org/jira/browse/SOLR-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410761#comment-15410761 ] Gregory Chanan commented on SOLR-9324: -- I'm not going to have a chance to backport this to 6x in the short term...[~hgadre] do you want to take a look? > Support Secure Impersonation / Proxy User for solr authentication > - > > Key: SOLR-9324 > URL: https://issues.apache.org/jira/browse/SOLR-9324 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: SOLR-9324-tests.patch, SOLR-9324.patch, SOLR-9324.patch, > SOLR-9324.patch, SOLR-9324_branch_6x.patch, build-6025.log > > > Solr should support Proxy User / Secure Impersonation for authentication, as > supported by hadoop > (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) > and supported by the hadoop AuthenticationFilter (which we use for the > KerberosPlugin). > There are a number of use cases, but a common one is this: > There is a front end for searches (say, Hue http://gethue.com/) that supports > its own login mechanisms. If the cluster uses kerberos for authentication, > hue must have kerberos credentials for each user, which is a pain to manage. > Instead, hue can be allowed to impersonate known users from known machines so > it only needs its own kerberos credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
[
https://issues.apache.org/jira/browse/SOLR-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9324:
-
Attachment: SOLR-9324-tests.patch
Here's a patch that attempts to fix the test failures (since I can't reproduce
I can't be sure).
For the case Varun points out, if a group can't be found, anything is accepted.
For the case Steve points out, it allows any
InetAddress.getLoopbackAddress().{getCanonicalHostName/getHostName/getHostAddress}
> Support Secure Impersonation / Proxy User for solr authentication
> -
>
> Key: SOLR-9324
> URL: https://issues.apache.org/jira/browse/SOLR-9324
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9324-tests.patch, SOLR-9324.patch, SOLR-9324.patch,
> SOLR-9324.patch, SOLR-9324_branch_6x.patch, build-6025.log
>
>
> Solr should support Proxy User / Secure Impersonation for authentication, as
> supported by hadoop
> (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html)
> and supported by the hadoop AuthenticationFilter (which we use for the
> KerberosPlugin).
> There are a number of use cases, but a common one is this:
> There is a front end for searches (say, Hue http://gethue.com/) that supports
> its own login mechanisms. If the cluster uses kerberos for authentication,
> hue must have kerberos credentials for each user, which is a pain to manage.
> Instead, hue can be allowed to impersonate known users from known machines so
> it only needs its own kerberos credentials.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
[ https://issues.apache.org/jira/browse/SOLR-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410451#comment-15410451 ] Gregory Chanan commented on SOLR-9324: -- I believe these are related to the assumptions the test makes about the local box. In the case Varun points to, the assumption is that the user running the process belongs to at least one group. In the cases Steve points to, I believe I assumption is that the loopback device is 127.0.0.1. > Support Secure Impersonation / Proxy User for solr authentication > - > > Key: SOLR-9324 > URL: https://issues.apache.org/jira/browse/SOLR-9324 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: SOLR-9324.patch, SOLR-9324.patch, SOLR-9324.patch, > SOLR-9324_branch_6x.patch, build-6025.log > > > Solr should support Proxy User / Secure Impersonation for authentication, as > supported by hadoop > (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) > and supported by the hadoop AuthenticationFilter (which we use for the > KerberosPlugin). > There are a number of use cases, but a common one is this: > There is a front end for searches (say, Hue http://gethue.com/) that supports > its own login mechanisms. If the cluster uses kerberos for authentication, > hue must have kerberos credentials for each user, which is a pain to manage. > Instead, hue can be allowed to impersonate known users from known machines so > it only needs its own kerberos credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
[ https://issues.apache.org/jira/browse/SOLR-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410429#comment-15410429 ] Gregory Chanan commented on SOLR-9324: -- Interesting, I wasn't able to reproduce any of those failures on my Mac. > Support Secure Impersonation / Proxy User for solr authentication > - > > Key: SOLR-9324 > URL: https://issues.apache.org/jira/browse/SOLR-9324 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: SOLR-9324.patch, SOLR-9324.patch, SOLR-9324.patch, > SOLR-9324_branch_6x.patch, build-6025.log > > > Solr should support Proxy User / Secure Impersonation for authentication, as > supported by hadoop > (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) > and supported by the hadoop AuthenticationFilter (which we use for the > KerberosPlugin). > There are a number of use cases, but a common one is this: > There is a front end for searches (say, Hue http://gethue.com/) that supports > its own login mechanisms. If the cluster uses kerberos for authentication, > hue must have kerberos credentials for each user, which is a pain to manage. > Instead, hue can be allowed to impersonate known users from known machines so > it only needs its own kerberos credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
[ https://issues.apache.org/jira/browse/SOLR-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-9324: - Attachment: SOLR-9324_branch_6x.patch Here's a branch 6 patch. > Support Secure Impersonation / Proxy User for solr authentication > - > > Key: SOLR-9324 > URL: https://issues.apache.org/jira/browse/SOLR-9324 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: SOLR-9324.patch, SOLR-9324.patch, SOLR-9324.patch, > SOLR-9324_branch_6x.patch > > > Solr should support Proxy User / Secure Impersonation for authentication, as > supported by hadoop > (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) > and supported by the hadoop AuthenticationFilter (which we use for the > KerberosPlugin). > There are a number of use cases, but a common one is this: > There is a front end for searches (say, Hue http://gethue.com/) that supports > its own login mechanisms. If the cluster uses kerberos for authentication, > hue must have kerberos credentials for each user, which is a pain to manage. > Instead, hue can be allowed to impersonate known users from known machines so > it only needs its own kerberos credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
[ https://issues.apache.org/jira/browse/SOLR-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-9324: - Attachment: SOLR-9324.patch Latest changes caused a failure in delegation token tests -- this should address that. > Support Secure Impersonation / Proxy User for solr authentication > - > > Key: SOLR-9324 > URL: https://issues.apache.org/jira/browse/SOLR-9324 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: SOLR-9324.patch, SOLR-9324.patch, SOLR-9324.patch > > > Solr should support Proxy User / Secure Impersonation for authentication, as > supported by hadoop > (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) > and supported by the hadoop AuthenticationFilter (which we use for the > KerberosPlugin). > There are a number of use cases, but a common one is this: > There is a front end for searches (say, Hue http://gethue.com/) that supports > its own login mechanisms. If the cluster uses kerberos for authentication, > hue must have kerberos credentials for each user, which is a pain to manage. > Instead, hue can be allowed to impersonate known users from known machines so > it only needs its own kerberos credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
[ https://issues.apache.org/jira/browse/SOLR-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-9324: - Attachment: SOLR-9324.patch > Support Secure Impersonation / Proxy User for solr authentication > - > > Key: SOLR-9324 > URL: https://issues.apache.org/jira/browse/SOLR-9324 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: SOLR-9324.patch, SOLR-9324.patch > > > Solr should support Proxy User / Secure Impersonation for authentication, as > supported by hadoop > (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) > and supported by the hadoop AuthenticationFilter (which we use for the > KerberosPlugin). > There are a number of use cases, but a common one is this: > There is a front end for searches (say, Hue http://gethue.com/) that supports > its own login mechanisms. If the cluster uses kerberos for authentication, > hue must have kerberos credentials for each user, which is a pain to manage. > Instead, hue can be allowed to impersonate known users from known machines so > it only needs its own kerberos credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan resolved SOLR-9200.
--
Resolution: Fixed
Fix Version/s: master (7.0)
6.2
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 6.2, master (7.0)
>
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch, SOLR-9200_branch_6x.patch,
> SOLR-9200_branch_6x.patch, SOLR-9200_branch_6x.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200_branch_6x.patch
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch, SOLR-9200_branch_6x.patch,
> SOLR-9200_branch_6x.patch, SOLR-9200_branch_6x.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200_branch_6x.patch
Interestingly, had to add SuppressSSL to DelegationToken test for 6x, because
of some PKIAuthentication errors. The PKI tests have that annotation as well,
so the interesting part is why that isn't required on master.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch, SOLR-9200_branch_6x.patch,
> SOLR-9200_branch_6x.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200_branch_6x.patch
Here's a branch 6 patch, will commit soon assuming tests pass.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch, SOLR-9200_branch_6x.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399882#comment-15399882
]
Gregory Chanan commented on SOLR-9200:
--
Just letting it bake for a bit. Will merge to 6x today or early next week.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399567#comment-15399567
]
Gregory Chanan commented on SOLR-9200:
--
I think your change is correct, looks like some auto import issue. Not sure
why it didn't cause problems locally.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398043#comment-15398043
]
Gregory Chanan edited comment on SOLR-9200 at 7/28/16 7:10 PM:
---
Thanks for taking a look, [~ichattopadhyaya]
bq. I'm still wondering if we need the internode calls within the Solr cluster
to use tokens. It seems to me that they do not, as of this patch, even if the
user calls use delegation tokens.
I don't think they do, this is just for client authentication
{quote}Also, it is my belief that end to end requests work fine, even when
internode requests are involved. However there are no tests for this, afaict;
neither for kerberos plugin with delegation tokens, nor with delegation tokens.
The former situation couldn't be tested at the time of writing the kerberos
plugin due to lack of ticket cache of minikdc, but maybe that limitation
doesn't stop us from writing tests for the latter. So, even if we don't write
such tests here in this issue, I think we should write some as a follow up
issue, so that we are alerted when such support breaks.{quote}
Sure, SOLR-9324 contains such a test. Although it doesn't really solve the
minikdc problem, that just uses a different authentication mechanism.
{quote}
Overall, I am okay with us committing the current patch. However, I think I'd
be more comfortable if internode calls used tokens as well (or even PKI,
instead of tokens). I would ideally do that here (unless there are some reasons
for not doing it that I'm missing completely), however we can as well do that
as a follow up issue, if you think that is a better approach.
{quote}
Maybe I'm misunderstanding something, but don't the internode calls already use
PKI -- that seems to always be used for internode calls with SolrCloud. I
don't see what's different with this patch than before it.
was (Author: gchanan):
Thanks for taking a look, [~ichattopadhyaya]
bq. I'm still wondering if we need the internode calls within the Solr cluster
to use tokens. It seems to me that they do not, as of this patch, even if the
user calls use delegation tokens.
I don't think they do, this is just for client authentication
{quote}Also, it is my belief that end to end requests work fine, even when
internode requests are involved. However there are no tests for this, afaict;
neither for kerberos plugin with delegation tokens, nor with delegation tokens.
The former situation couldn't be tested at the time of writing the kerberos
plugin due to lack of ticket cache of minikdc, but maybe that limitation
doesn't stop us from writing tests for the latter. So, even if we don't write
such tests here in this issue, I think we should write some as a follow up
issue, so that we are alerted when such support breaks.{quote}
Sure, SOLR-9324 contains such a test. Although it doesn't really solve the
minikdc problem, that just uses a different authentication mechanism.
{code}
Overall, I am okay with us committing the current patch. However, I think I'd
be more comfortable if internode calls used tokens as well (or even PKI,
instead of tokens). I would ideally do that here (unless there are some reasons
for not doing it that I'm missing completely), however we can as well do that
as a follow up issue, if you think that is a better approach.
{code}
Maybe I'm misunderstanding something, but don't the internode calls already use
PKI -- that seems to always be used for internode calls with SolrCloud. I
don't see what's different with this patch than before it.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
>
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398043#comment-15398043
]
Gregory Chanan commented on SOLR-9200:
--
Thanks for taking a look, [~ichattopadhyaya]
bq. I'm still wondering if we need the internode calls within the Solr cluster
to use tokens. It seems to me that they do not, as of this patch, even if the
user calls use delegation tokens.
I don't think they do, this is just for client authentication
{quote}Also, it is my belief that end to end requests work fine, even when
internode requests are involved. However there are no tests for this, afaict;
neither for kerberos plugin with delegation tokens, nor with delegation tokens.
The former situation couldn't be tested at the time of writing the kerberos
plugin due to lack of ticket cache of minikdc, but maybe that limitation
doesn't stop us from writing tests for the latter. So, even if we don't write
such tests here in this issue, I think we should write some as a follow up
issue, so that we are alerted when such support breaks.{quote}
Sure, SOLR-9324 contains such a test. Although it doesn't really solve the
minikdc problem, that just uses a different authentication mechanism.
{code}
Overall, I am okay with us committing the current patch. However, I think I'd
be more comfortable if internode calls used tokens as well (or even PKI,
instead of tokens). I would ideally do that here (unless there are some reasons
for not doing it that I'm missing completely), however we can as well do that
as a follow up issue, if you think that is a better approach.
{code}
Maybe I'm misunderstanding something, but don't the internode calls already use
PKI -- that seems to always be used for internode calls with SolrCloud. I
don't see what's different with this patch than before it.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397603#comment-15397603
]
Gregory Chanan commented on SOLR-9200:
--
I am planning to commit this soon if there are no objections.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
[ https://issues.apache.org/jira/browse/SOLR-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-9324: - Attachment: SOLR-9324.patch Here's a patch that implements this. Note this assumes SOLR-9200 is applied, which hasn't been committed yet. > Support Secure Impersonation / Proxy User for solr authentication > - > > Key: SOLR-9324 > URL: https://issues.apache.org/jira/browse/SOLR-9324 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: SOLR-9324.patch > > > Solr should support Proxy User / Secure Impersonation for authentication, as > supported by hadoop > (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) > and supported by the hadoop AuthenticationFilter (which we use for the > KerberosPlugin). > There are a number of use cases, but a common one is this: > There is a front end for searches (say, Hue http://gethue.com/) that supports > its own login mechanisms. If the cluster uses kerberos for authentication, > hue must have kerberos credentials for each user, which is a pain to manage. > Instead, hue can be allowed to impersonate known users from known machines so > it only needs its own kerberos credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9344) BasicAuthIntegrationTest test failures on update
Gregory Chanan created SOLR-9344:
Summary: BasicAuthIntegrationTest test failures on update
Key: SOLR-9344
URL: https://issues.apache.org/jira/browse/SOLR-9344
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: security, Tests
Affects Versions: trunk
Reporter: Gregory Chanan
I've seen this a number of times while developing SOLR-9200 and SOLR-9324;
there's also a public failure here:
http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17372/
{code}
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: IOException
occured when talking to server at:
http://127.0.0.1:45882/solr/testSolrCloudCollection_shard1_replica2
at
__randomizedtesting.SeedInfo.seed([99BB0D0378978FA8:A463A32F4079D1D8]:0)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:760)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1172)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1061)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:997)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at
org.apache.solr.security.BasicAuthIntegrationTest.doExtraTests(BasicAuthIntegrationTest.java:193)
at
org.apache.solr.cloud.TestMiniSolrCloudClusterBase.testCollectionCreateSearchDelete(TestMiniSolrCloudClusterBase.java:196)
at
org.apache.solr.cloud.TestMiniSolrCloudClusterBase.testBasics(TestMiniSolrCloudClusterBase.java:79)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native
Method)
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:533)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
[jira] [Assigned] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
[ https://issues.apache.org/jira/browse/SOLR-9324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan reassigned SOLR-9324: Assignee: Gregory Chanan > Support Secure Impersonation / Proxy User for solr authentication > - > > Key: SOLR-9324 > URL: https://issues.apache.org/jira/browse/SOLR-9324 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Gregory Chanan >Assignee: Gregory Chanan > > Solr should support Proxy User / Secure Impersonation for authentication, as > supported by hadoop > (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) > and supported by the hadoop AuthenticationFilter (which we use for the > KerberosPlugin). > There are a number of use cases, but a common one is this: > There is a front end for searches (say, Hue http://gethue.com/) that supports > its own login mechanisms. If the cluster uses kerberos for authentication, > hue must have kerberos credentials for each user, which is a pain to manage. > Instead, hue can be allowed to impersonate known users from known machines so > it only needs its own kerberos credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389591#comment-15389591
]
Gregory Chanan commented on SOLR-9200:
--
removed some unused imports.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200.patch
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200.patch
sorry attached the wrong patch, should be correct now.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200.patch
Here's a new version of the patch, taking into account [~ichattopadhyaya]'s
comments. In particular:
1. All security related znodes have been moved under /security
2. The existing ZkACLProviders now derived from SecurityAwareZkACLProvider,
which by default does not allow read only access to znodes under /security for
non-solr users.
3. Moved all kerberos-related test setup from KerberosTestUtil to
KerberosTestServices. This object does "all in one" setup, i.e. dealing with
non-supported locales, kdc setup, Configuration management. It also handles
resetting the Configuration at the end of the test, so other tests in the same
jvm can run successfully.
4. Got rid of extra imports.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch,
> SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9324) Support Secure Impersonation / Proxy User for solr authentication
Gregory Chanan created SOLR-9324: Summary: Support Secure Impersonation / Proxy User for solr authentication Key: SOLR-9324 URL: https://issues.apache.org/jira/browse/SOLR-9324 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud Reporter: Gregory Chanan Solr should support Proxy User / Secure Impersonation for authentication, as supported by hadoop (http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html) and supported by the hadoop AuthenticationFilter (which we use for the KerberosPlugin). There are a number of use cases, but a common one is this: There is a front end for searches (say, Hue http://gethue.com/) that supports its own login mechanisms. If the cluster uses kerberos for authentication, hue must have kerberos credentials for each user, which is a pain to manage. Instead, hue can be allowed to impersonate known users from known machines so it only needs its own kerberos credentials. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378619#comment-15378619
]
Gregory Chanan commented on SOLR-9200:
--
{quote}Looks like there is some adverse interaction between
TestSolrCloudWithDelegationTokens and TestSolrCloudWithKerberosAlt when they
run in the same jvm; in the same jvm TestSolrCloudWithKerberosAlt fails when
run second. I beasted the test individually and it didn't fail in 100
runs.{quote}
Figured this out -- in Krb5HttpClientBuilder the JaasConfiguration is set up
statically with baseConfig whatever is the current config, but the tests like
TestSolrCloudWithKerberosAlt require that the baseConfig is what they set it to
at the beginning of test;
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378355#comment-15378355
]
Gregory Chanan commented on SOLR-9200:
--
Looks like there is some adverse interaction between
TestSolrCloudWithDelegationTokens and TestSolrCloudWithKerberosAlt when they
run in the same jvm; in the same jvm TestSolrCloudWithKerberosAlt fails when
run second. I beasted the test individually and it didn't fail in 100 runs.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373674#comment-15373674
]
Gregory Chanan commented on SOLR-9200:
--
[~ichattopadhyaya] your argument sounds reasonable to me.
[~anshumg] and Ishan, thanks for taking a look.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200.patch
New patch that passes the forbidden api checks. I had to add one suppression
as follows:
{code}
HttpServletResponse rspCloseShield = new HttpServletResponseWrapper(frsp) {
@SuppressForbidden(reason = "Hadoop DelegationTokenAuthenticationFilter
uses response writer, this" +
"is providing a CloseShield on top of that")
@Override
public PrintWriter getWriter() throws IOException {
final PrintWriter pw = new PrintWriterWrapper(frsp.getWriter()) {
@Override
public void close() {};
};
return pw;
}
};
{code}
I'm not 100% sure if the getWriter problem affects the hadoop usage, which is
here:
https://github.com/apache/hadoop/blob/9d46a49c746b9e1ef552dbb10d1e22f87db68c76/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java#L280-L282
I can certainly submit a patch to hadoop (or add it to HADOOP-13346) to use
OutputStream instead -- [~thetaphi]?
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366917#comment-15366917
]
Gregory Chanan commented on SOLR-9200:
--
CC [~anshumg], who worked on SOLR-7274.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366913#comment-15366913
]
Gregory Chanan commented on SOLR-9200:
--
Functionality:
1) This patch allows configuration of delegation tokens for the KerberosPlugin.
Existing
configurations will not change and will not support delegation tokens.
The configuration parameters are as follows:
||Key||Type||Default||Description||
|solr.kerberos.delegation.token.enabled|boolean|false|Set to true to enable
delegation tokens
|solr.kerberos.delegation.token.kind|String|solr-dt|Type name of delegation
tokens, most likely doesn't need to change
|solr.kerberos.delegation.token.validity|integer|36000|Lifetime in seconds for
which delegation tokens are valid|
|solr.kerberos.delegation.token.signer.secret.provider|String|zookeeper|Where
delegation token information is stored internally; if not "zookeeper"
delegation tokens won't work across solr servers
|solr.kerberos.delegation.token.signer.secret.provider.zookeper.path|String|(chrooted
path) + /token|Zookeeper location where secret provider information is stored
|solr.kerberos.delegation.token.secret.manager.znode.working.path|String|(chrooted
path) + /zkdtsm|Zookeeper location where token information is stored
2) Includes solrj support for delegation tokens as follows:
a) DelegationTokenRequest/DelegationTokenResponse can be used to
get/cancel/renew delegation tokens
b) HttpSolrClient.Builder now includes a "withDelegationToken" function for
creating an HttpSolrClient
that uses a delegation token to authenticate
Note that hadoop's delegation token responses are in json map format, so
there is a new ResponseParser
for that in DelegationTokenResponse; I couldn't find an existing response
parser that worked
Issues / Workarounds:
3) AuthenticationPlugin has an incompatible change that means this should only
go into the next major version.
Basically:
{code}void doAuthenticate{code}
changed to:
{code}boolean doAuthenticate{code}
This is to support cases where authentication succeeds, but solr itself
shouldn't process the request; e.g.
in the delegation token management operations (get, cancel, renew). The
boolean, if false, indicates a short
circuit out of the rest of the solr dispatch logic.
4) DelegationTokenKerberosFilter includes a workaround for null query strings.
The versions of
hadoop / httpclient that we use don't work with null HttpServletRequest query
strings, which the jetty
version we use can provide. This can be solved by using HTTPCLIENT-1746 (not
released yet) or HADOOP-12767
(not released in a stable version).
5) hadoop's delegation token code writes to a closed PrintWriter; this doesn't
seem to be a problem for the
version of jetty that hadoop uses, but it causes an issue with our version. I
filed HADOOP-13346 to fix that,
until then, I wrote a PrintWriterWrapper that ignores closes.
6) The hadoop zookeeper delegation token code uses Curator rather than
SolrZkClient; the conversion
from SolrZkClient is messy in a few places:
a) We use the ZkController.ZkClient's ACL Provider for the delegation tokens
in ZK, but it's not obvious this
is what you'd actually want to use. For example, it may be reasonable to
have most solr znodes be readable
(because clients read e.g. clusterstate.json), but you probably don't want
the delegation token information
to be readable by anyone outside "solr". I haven't checked recently, but I
don't think we provide any
built in ACLProviders that would do something reasonable here in the
general case. Basically, it's easy to
get this wrong and leak security information.
b) Getting credentials information to curator also isn't great. Again, we
use ZkController.ZkClient's credentials
(at AuthenticationPlugin.init) time, but given the
"setZkCredentialsToAddAutomatically" function, these could
in theory change. I haven't looked into changing that into a builder so
it's guaranteed not to change.
c) Retrying logic is handled completely differently. In theory, you'd like
the curator logic to follow ZkController.ZkClient's
ZkClientConnectionStrategy logic, but it's not clear how to implement
this. Instead, we just use curator's ExponentialBackoffRetry.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to
[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200.patch
[~ichattopadhyaya] thanks so much! I attached my current patch; I would very
much appreciate a review. My first couple of runs through the tests failed,
although I haven't determined if those are just existing flaky tests or not.
I'll write up some notes shortly.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9153) Update beanutils version to 1.9.2
[
https://issues.apache.org/jira/browse/SOLR-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359651#comment-15359651
]
Gregory Chanan commented on SOLR-9153:
--
My reading of BEANUTILS-463 and the 1.9.2 release notes is that 1.9.2 only
contains a fix, it doesn't actually apply the fix by default. E.g. from the
release notes:
{code}
* [BEANUTILS-463]
Added new SuppressPropertiesBeanIntrospector class to deal with a potential
class loader vulnerability.
{code}
> Update beanutils version to 1.9.2
> -
>
> Key: SOLR-9153
> URL: https://issues.apache.org/jira/browse/SOLR-9153
> Project: Solr
> Issue Type: Bug
> Components: contrib - Velocity
>Affects Versions: 6.0
>Reporter: Mike Drob
>Priority: Minor
> Attachments: SOLR-9153.patch
>
>
> See CVE-2014-0114 --
> https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-0114
> {quote}
> Apache Commons BeanUtils, as distributed in lib/commons-beanutils-1.8.0.jar
> in Apache Struts 1.x through 1.3.10 and in other products requiring
> commons-beanutils through 1.9.2, does not suppress the class property, which
> allows remote attackers to "manipulate" the ClassLoader and execute arbitrary
> code via the class parameter, as demonstrated by the passing of this
> parameter to the getClass method of the ActionForm object in Struts 1.
> {quote}
> We transitively depend on BeanUtils through Velocity, but it doesn't look
> like there is much movement on the project there. See BEANUTILS-463 and
> VELTOOLS-170
> Also, this might have impact on SOLR-3736 but that issue also looks largely
> abandoned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9076) Update to Hadoop 2.7.2
[
https://issues.apache.org/jira/browse/SOLR-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353532#comment-15353532
]
Gregory Chanan commented on SOLR-9076:
--
bq. Which is odd, because this drove moving to Netty 4 as well, so why does it
want Netty 3 classes - does it have conflicting Netty reqs?
Good question. Looks like the dependency is coming from the bkjournal contrib
-- I wonder if there's some configuration we can use to disable that in the
tests.
Here's mvn dependency:tree output:
{code}
[INFO] org.apache.hadoop.contrib:hadoop-hdfs-bkjournal:jar:2.7.2
...
[INFO] +- org.apache.bookkeeper:bookkeeper-server:jar:4.2.3:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.10:compile
[INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.10:compile
[INFO] | +- org.jboss.netty:netty:jar:3.2.4.Final:compile
...
{code}
> Update to Hadoop 2.7.2
> --
>
> Key: SOLR-9076
> URL: https://issues.apache.org/jira/browse/SOLR-9076
> Project: Solr
> Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 6.2, master (7.0)
>
> Attachments: SOLR-9076-Hack.patch, SOLR-9076-fixnetty.patch,
> SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch,
> SOLR-9076.patch, SOLR-9076.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9076) Update to Hadoop 2.7.2
[
https://issues.apache.org/jira/browse/SOLR-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352114#comment-15352114
]
Gregory Chanan commented on SOLR-9076:
--
Saw this on another machine:
{code}
[junit4]> Throwable #1: java.io.IOException: Failed on local exception:
java.io.IOException: Broken pipe; Host Details : local host is:
"ubuntu14-ec2-beefy-slave-03a7.vpc.cloudera.com/172.26.18.223"; destination
host is: "ubuntu14-ec2-beefy-slave-03a7.vpc.cloudera.com":53094;
[junit4]>at
__randomizedtesting.SeedInfo.seed([F57BFC0B596FCED0:FB29480558F9FCDF]:0)
[junit4]>at
org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
[junit4]>at org.apache.hadoop.ipc.Client.call(Client.java:1479)
[junit4]>at org.apache.hadoop.ipc.Client.call(Client.java:1412)
[junit4]>at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
[junit4]>at com.sun.proxy.$Proxy112.getClusterMetrics(Unknown
Source)
[junit4]>at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:206)
[junit4]>at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
[junit4]>at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
[junit4]>at com.sun.proxy.$Proxy113.getClusterMetrics(Unknown
Source)
[junit4]>at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:487)
[junit4]>at
org.apache.hadoop.mapred.ResourceMgrDelegate.getClusterMetrics(ResourceMgrDelegate.java:151)
[junit4]>at
org.apache.hadoop.mapred.YARNRunner.getClusterMetrics(YARNRunner.java:179)
[junit4]>at
org.apache.hadoop.mapreduce.Cluster.getClusterStatus(Cluster.java:247)
[junit4]>at
org.apache.hadoop.mapred.JobClient$3.run(JobClient.java:748)
[junit4]>at
org.apache.hadoop.mapred.JobClient$3.run(JobClient.java:746)
[junit4]>at java.security.AccessController.doPrivileged(Native
Method)
[junit4]>at javax.security.auth.Subject.doAs(Subject.java:422)
[junit4]>at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
[junit4]>at
org.apache.hadoop.mapred.JobClient.getClusterStatus(JobClient.java:746)
[junit4]>at
org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:642)
[junit4]>at
org.apache.solr.hadoop.MapReduceIndexerTool.run(MapReduceIndexerTool.java:605)
[junit4]>at
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
[junit4]>at
org.apache.solr.hadoop.MorphlineBasicMiniMRTest.mrRun(MorphlineBasicMiniMRTest.java:364)
[junit4]>at java.lang.Thread.run(Thread.java:745)
[junit4]> Caused by: java.io.IOException: Broken pipe
[junit4]>at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
[junit4]>at
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
[junit4]>at
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
[junit4]>at sun.nio.ch.IOUtil.write(IOUtil.java:65)
[junit4]>at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
[junit4]>at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
[junit4]>at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
[junit4]>at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
[junit4]>at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
[junit4]>at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
[junit4]>at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
[junit4]>at
java.io.DataOutputStream.flush(DataOutputStream.java:123)
[junit4]>at
org.apache.hadoop.ipc.Client$Connection$3.run(Client.java:1043)
[junit4]>at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4]>at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4]>at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4]>at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4]>... 1 more
{code}
> Update to Hadoop 2.7.2
> --
>
> Key: SOLR-9076
> URL: https://issues.apache.org/jira/browse/SOLR-9076
> Project: Solr
>
[jira] [Updated] (SOLR-9076) Update to Hadoop 2.7.2
[ https://issues.apache.org/jira/browse/SOLR-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-9076: - Attachment: SOLR-9076-fixnetty.patch Here's a patch that adds the netty dependency. I'm still seeing test failures locally, not sure if they are a product of my environment or not yet. > Update to Hadoop 2.7.2 > -- > > Key: SOLR-9076 > URL: https://issues.apache.org/jira/browse/SOLR-9076 > Project: Solr > Issue Type: Improvement >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9076-fixnetty.patch, SOLR-9076.patch, > SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch, > SOLR-9076.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9076) Update to Hadoop 2.7.2
[
https://issues.apache.org/jira/browse/SOLR-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351964#comment-15351964
]
Gregory Chanan commented on SOLR-9076:
--
I added org.jboss.netty.netty version 3.2.4.Final and now I get this:
{code}
2> java.lang.reflect.InvocationTargetException
[junit4] 2>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit4] 2>at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] 2>at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2>at java.lang.reflect.Method.invoke(Method.java:498)
[junit4] 2>at
org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
[junit4] 2>at
org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
[junit4] 2>at
org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:401)
[junit4] 2>at
org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
[junit4] 2>at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:194)
[junit4] 2>at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
[junit4] 2>at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
[junit4] 2>at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
[junit4] 2>at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
[junit4] 2>at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
[junit4] 2>at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
[junit4] 2>at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
[junit4] 2>at
com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
[junit4] 2>at
org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
[junit4] 2>at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:226)
[junit4] 2>at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:211)
[junit4] 2>at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:463)
[junit4] 2>at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:213)
[junit4] 2>at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:594)
[junit4] 2>at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
[junit4] 2>at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
[junit4] 2>at
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:171)
[junit4] 2>at
org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:872)
[junit4] 2>at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1726)
[junit4] 2>at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1705)
[junit4] 2>at
org.apache.solr.hadoop.MorphlineBasicMiniMRTest.teardownClass(MorphlineBasicMiniMRTest.java:196)
[junit4] 2>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit4] 2>at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] 2>at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] 2>at java.lang.reflect.Method.invoke(Method.java:498)
[junit4] 2>at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
[junit4] 2>at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
[junit4] 2>at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2>at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
[junit4] 2>at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4] 2>at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2>
[jira] [Commented] (SOLR-9076) Update to Hadoop 2.7.2
[ https://issues.apache.org/jira/browse/SOLR-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351895#comment-15351895 ] Gregory Chanan commented on SOLR-9076: -- bq. Strange, I wonder why that didn't show up when I ran the tests? Maybe I need a different profile. It's a nightly test. I was able to reproduce it. > Update to Hadoop 2.7.2 > -- > > Key: SOLR-9076 > URL: https://issues.apache.org/jira/browse/SOLR-9076 > Project: Solr > Issue Type: Improvement >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch, > SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9076) Update to Hadoop 2.7.2
[ https://issues.apache.org/jira/browse/SOLR-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15351795#comment-15351795 ] Gregory Chanan commented on SOLR-9076: -- Strange, I wonder why that didn't show up when I ran the tests? Maybe I need a different profile. I'll take a look. > Update to Hadoop 2.7.2 > -- > > Key: SOLR-9076 > URL: https://issues.apache.org/jira/browse/SOLR-9076 > Project: Solr > Issue Type: Improvement >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch, > SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-9076) Update to Hadoop 2.7.2
[ https://issues.apache.org/jira/browse/SOLR-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan resolved SOLR-9076. -- Resolution: Fixed Fix Version/s: master (7.0) 6.2 Committed to 7.0 and 6.2. Thanks Mark! > Update to Hadoop 2.7.2 > -- > > Key: SOLR-9076 > URL: https://issues.apache.org/jira/browse/SOLR-9076 > Project: Solr > Issue Type: Improvement >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 6.2, master (7.0) > > Attachments: SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch, > SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9076) Update to Hadoop 2.7.2
[ https://issues.apache.org/jira/browse/SOLR-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-9076: - Attachment: SOLR-9076.patch Here's a patch that passed the test and precommits. Only change from previous patch is it removes the org.htrace versions (which error'ed out in the precommit because they aren't used anymore) and removes the org.htrace license/notice/sha1. I will commit this shortly. > Update to Hadoop 2.7.2 > -- > > Key: SOLR-9076 > URL: https://issues.apache.org/jira/browse/SOLR-9076 > Project: Solr > Issue Type: Improvement >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch, > SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9076) Update to Hadoop 2.7.2
[ https://issues.apache.org/jira/browse/SOLR-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-9076: - Attachment: SOLR-9076.patch rebased patch -- looks like the TestRecoveryHdfs.java change already went in. > Update to Hadoop 2.7.2 > -- > > Key: SOLR-9076 > URL: https://issues.apache.org/jira/browse/SOLR-9076 > Project: Solr > Issue Type: Improvement >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-9076.patch, SOLR-9076.patch, SOLR-9076.patch, > SOLR-9076.patch, SOLR-9076.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345380#comment-15345380
]
Gregory Chanan commented on SOLR-9200:
--
Did some more work on this. Currently blocked because we need HADOOP-11492,
which is only available in hadoop 2.7.0+. Upgrading to hadoop 2.7.2 is
currently being tracked in SOLR-9076.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15336808#comment-15336808
]
Gregory Chanan commented on SOLR-9200:
--
actively working on it -- are you interested in the feature?
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323649#comment-15323649
]
Gregory Chanan edited comment on SOLR-9200 at 6/10/16 12:25 AM:
I started working on this. One issue I immediately hit was HADOOP-12767 -- it
appears upgrading the httpclient version causes a null check to need to be
inserted on the path of delegation token checking.
Also note that HADOOP-12767 was fixed in hadoop 2.8 but the latest stable
release is 2.7.2.
was (Author: gchanan):
I started working on this. One issue I immediately hit was HADOOP-12767 -- it
appears upgrading the httpclient version causes a null check to need to be
inserted on the path of delegation token checking.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr
[
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323649#comment-15323649
]
Gregory Chanan commented on SOLR-9200:
--
I started working on this. One issue I immediately hit was HADOOP-12767 -- it
appears upgrading the httpclient version causes a null check to need to be
inserted on the path of delegation token checking.
> Add Delegation Token Support to Solr
>
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
> Issue Type: New Feature
> Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>
> SOLR-7468 added support for kerberos authentication via the hadoop
> authentication filter. Hadoop also has support for an authentication filter
> that supports delegation tokens, which allow authenticated users the ability
> to grab/renew/delete a token that can be used to bypass the normal
> authentication path for a time. This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access
> to the user's kerberos credentials. Instead, the job runner can grab a
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more
> privileged user can request a delegation token that can be passed to the less
> privileged user.
> Note to self:
> In
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
> I made the following comment which I need to investigate further, since I
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
> moving forward (I understand this is more a generic auth question than
> kerberos specific). For example, in the latest version of the filter we are
> using at Cloudera, we play around with the ServletContext in order to pass
> information around
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
> Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-9199) ZkController#publishAndWaitForDownStates logic is inefficient
[ https://issues.apache.org/jira/browse/SOLR-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan resolved SOLR-9199. -- Resolution: Fixed Fix Version/s: 6.2 master (7.0) > ZkController#publishAndWaitForDownStates logic is inefficient > - > > Key: SOLR-9199 > URL: https://issues.apache.org/jira/browse/SOLR-9199 > Project: Solr > Issue Type: Bug >Reporter: Hrishikesh Gadre >Assignee: Hrishikesh Gadre > Fix For: master (7.0), 6.2 > > Attachments: SOLR-9199.patch > > > The following logic introduced as part of SOLR-8720 is inefficient. > https://github.com/apache/lucene-solr/blob/6c0331b8309603eaaf14b6677afba5ffe99f16a3/solr/core/src/java/org/apache/solr/cloud/ZkController.java#L687-L712 > Specifically, > * foundStates flag is set to TRUE before the for loop. > * In the for loop we check if any replica on this node is not in the DOWN > state. If yes, then foundStates = FALSE > * If foundStates == TRUE then we break out of the loop and return. > The problem here is that once foundStates is set to FALSE, it is never reset > to TRUE. Hence we end up spending the whole 60 secs iterating over the > collections even though all the replicas are marked as DOWN in later > iterations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9199) ZkController#publishAndWaitForDownStates logic is inefficient
[ https://issues.apache.org/jira/browse/SOLR-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322083#comment-15322083 ] Gregory Chanan commented on SOLR-9199: -- Precommit succeeded for me, will commit in the morning. > ZkController#publishAndWaitForDownStates logic is inefficient > - > > Key: SOLR-9199 > URL: https://issues.apache.org/jira/browse/SOLR-9199 > Project: Solr > Issue Type: Bug >Reporter: Hrishikesh Gadre >Assignee: Hrishikesh Gadre > Attachments: SOLR-9199.patch > > > The following logic introduced as part of SOLR-8720 is inefficient. > https://github.com/apache/lucene-solr/blob/6c0331b8309603eaaf14b6677afba5ffe99f16a3/solr/core/src/java/org/apache/solr/cloud/ZkController.java#L687-L712 > Specifically, > * foundStates flag is set to TRUE before the for loop. > * In the for loop we check if any replica on this node is not in the DOWN > state. If yes, then foundStates = FALSE > * If foundStates == TRUE then we break out of the loop and return. > The problem here is that once foundStates is set to FALSE, it is never reset > to TRUE. Hence we end up spending the whole 60 secs iterating over the > collections even though all the replicas are marked as DOWN in later > iterations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9200) Add Delegation Token Support to Solr
Gregory Chanan created SOLR-9200:
Summary: Add Delegation Token Support to Solr
Key: SOLR-9200
URL: https://issues.apache.org/jira/browse/SOLR-9200
Project: Solr
Issue Type: New Feature
Components: security
Reporter: Gregory Chanan
Assignee: Gregory Chanan
SOLR-7468 added support for kerberos authentication via the hadoop
authentication filter. Hadoop also has support for an authentication filter
that supports delegation tokens, which allow authenticated users the ability to
grab/renew/delete a token that can be used to bypass the normal authentication
path for a time. This is useful in a variety of use cases:
1) distributed clients (e.g. MapReduce) where each client may not have access
to the user's kerberos credentials. Instead, the job runner can grab a
delegation token and use that during task execution.
2) If the load on the kerberos server is too high, delegation tokens can avoid
hitting the kerberos server after the first request
3) If requests/permissions need to be delegated to another user: the more
privileged user can request a delegation token that can be passed to the less
privileged user.
Note to self:
In
https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
I made the following comment which I need to investigate further, since I
don't know if anything changed in this area:
{quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin
moving forward (I understand this is more a generic auth question than kerberos
specific). For example, in the latest version of the filter we are using at
Cloudera, we play around with the ServletContext in order to pass information
around
(https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
Is there any way we can get the actual ServletContext in a plugin?{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9199) ZkController#publishAndWaitForDownStates logic is inefficient
[ https://issues.apache.org/jira/browse/SOLR-9199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321714#comment-15321714 ] Gregory Chanan commented on SOLR-9199: -- lgtm. Could maybe quibble that the old code wouldn't print "timing out" if it never entered the loop (e.g. if wait time was 0), but that doesn't seem right or wrong. I'll commit to trunk and 6.x assuming the precommit and tests pass. > ZkController#publishAndWaitForDownStates logic is inefficient > - > > Key: SOLR-9199 > URL: https://issues.apache.org/jira/browse/SOLR-9199 > Project: Solr > Issue Type: Bug >Reporter: Hrishikesh Gadre >Assignee: Hrishikesh Gadre > Attachments: SOLR-9199.patch > > > The following logic introduced as part of SOLR-8720 is inefficient. > https://github.com/apache/lucene-solr/blob/6c0331b8309603eaaf14b6677afba5ffe99f16a3/solr/core/src/java/org/apache/solr/cloud/ZkController.java#L687-L712 > Specifically, > * foundStates flag is set to TRUE before the for loop. > * In the for loop we check if any replica on this node is not in the DOWN > state. If yes, then foundStates = FALSE > * If foundStates == TRUE then we break out of the loop and return. > The problem here is that once foundStates is set to FALSE, it is never reset > to TRUE. Hence we end up spending the whole 60 secs iterating over the > collections even though all the replicas are marked as DOWN in later > iterations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9055) Make collection backup/restore extensible
[ https://issues.apache.org/jira/browse/SOLR-9055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269647#comment-15269647 ] Gregory Chanan commented on SOLR-9055: -- bq. Any thoughts on this one Mark Miller or Gregory Chanan perhaps? I agree with Hrishikesh's take here. In addition to the third party library issue, HDFS does not in general support all the local file system APIs you'd expect. I don't know if there are issues with nio specifically, but two examples in the past have been truncate and directory sync. > Make collection backup/restore extensible > - > > Key: SOLR-9055 > URL: https://issues.apache.org/jira/browse/SOLR-9055 > Project: Solr > Issue Type: Task >Reporter: Hrishikesh Gadre >Assignee: Mark Miller > Attachments: SOLR-9055.patch, SOLR-9055.patch > > > SOLR-5750 implemented backup/restore API for Solr. This JIRA is to track the > code cleanup/refactoring. Specifically following improvements should be made, > - Add Solr/Lucene version to check the compatibility between the backup > version and the version of Solr on which it is being restored. > - Add a backup implementation version to check the compatibility between the > "restore" implementation and backup format. > - Introduce a Strategy interface to define how the Solr index data is backed > up (e.g. using file copy approach). > - Introduce a Repository interface to define the file-system used to store > the backup data. (currently works only with local file system but can be > extended). This should be enhanced to introduce support for "registering" > repositories (e.g. HDFS, S3 etc.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-9047) zkcli should allow alternative locations for log4j configuration
[
https://issues.apache.org/jira/browse/SOLR-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan resolved SOLR-9047.
--
Resolution: Fixed
Fix Version/s: trunk
6.1
Thanks for taking a look Mark and Christine. Committed to trunk and 6.1.
> zkcli should allow alternative locations for log4j configuration
>
>
> Key: SOLR-9047
> URL: https://issues.apache.org/jira/browse/SOLR-9047
> Project: Solr
> Issue Type: Improvement
> Components: scripts and tools, SolrCloud
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Fix For: 6.1, trunk
>
> Attachments: SOLR-9047.patch, SOLR-9047.patch
>
>
> zkcli uses the log4j configuration in the local directory:
> {code}
> sdir="`dirname \"$0\"`"
> PATH=$JAVA_HOME/bin:$PATH $JVM
> -Dlog4j.configuration=file:$sdir/log4j.properties -classpath
> "$sdir/../../solr-webapp/webapp/WEB-INF/lib/*:$sdir/../../lib/ext/*"
> org.apache.solr.cloud.ZkCLI ${1+"$@"}
> {code}
> which is a reasonable default, but often people want to use a "global" log4j
> configuration. For example, one may define a log4j configuration that writes
> to an external log directory and want to point to this rather than copying it
> to each source checkout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9047) zkcli should allow alternative locations for log4j configuration
[
https://issues.apache.org/jira/browse/SOLR-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9047:
-
Attachment: SOLR-9047.patch
Here's a patch that matches the caps style (I had been copying the style from
solr.cmd)
> zkcli should allow alternative locations for log4j configuration
>
>
> Key: SOLR-9047
> URL: https://issues.apache.org/jira/browse/SOLR-9047
> Project: Solr
> Issue Type: Improvement
> Components: scripts and tools, SolrCloud
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: SOLR-9047.patch, SOLR-9047.patch
>
>
> zkcli uses the log4j configuration in the local directory:
> {code}
> sdir="`dirname \"$0\"`"
> PATH=$JAVA_HOME/bin:$PATH $JVM
> -Dlog4j.configuration=file:$sdir/log4j.properties -classpath
> "$sdir/../../solr-webapp/webapp/WEB-INF/lib/*:$sdir/../../lib/ext/*"
> org.apache.solr.cloud.ZkCLI ${1+"$@"}
> {code}
> which is a reasonable default, but often people want to use a "global" log4j
> configuration. For example, one may define a log4j configuration that writes
> to an external log directory and want to point to this rather than copying it
> to each source checkout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9047) zkcli should allow alternative locations for log4j configuration
[
https://issues.apache.org/jira/browse/SOLR-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-9047:
-
Attachment: SOLR-9047.patch
Here's a patch that lets you specify the log4j configuration file via the
LOG4J_PROPS environment variable.
I'd appreciate someone looking at the windows code since I can't test.
> zkcli should allow alternative locations for log4j configuration
>
>
> Key: SOLR-9047
> URL: https://issues.apache.org/jira/browse/SOLR-9047
> Project: Solr
> Issue Type: Improvement
> Components: scripts and tools, SolrCloud
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
> Attachments: SOLR-9047.patch
>
>
> zkcli uses the log4j configuration in the local directory:
> {code}
> sdir="`dirname \"$0\"`"
> PATH=$JAVA_HOME/bin:$PATH $JVM
> -Dlog4j.configuration=file:$sdir/log4j.properties -classpath
> "$sdir/../../solr-webapp/webapp/WEB-INF/lib/*:$sdir/../../lib/ext/*"
> org.apache.solr.cloud.ZkCLI ${1+"$@"}
> {code}
> which is a reasonable default, but often people want to use a "global" log4j
> configuration. For example, one may define a log4j configuration that writes
> to an external log directory and want to point to this rather than copying it
> to each source checkout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9047) zkcli should allow alternative locations for log4j configuration
[
https://issues.apache.org/jira/browse/SOLR-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15261416#comment-15261416
]
Gregory Chanan commented on SOLR-9047:
--
bq. is it time to just kill zkcli and move it's functionality into bin/solr ?
probably :). In any case that would be a 7.x change, right? This would be
useful in 6.x imo.
> zkcli should allow alternative locations for log4j configuration
>
>
> Key: SOLR-9047
> URL: https://issues.apache.org/jira/browse/SOLR-9047
> Project: Solr
> Issue Type: Improvement
> Components: scripts and tools, SolrCloud
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
>Priority: Minor
>
> zkcli uses the log4j configuration in the local directory:
> {code}
> sdir="`dirname \"$0\"`"
> PATH=$JAVA_HOME/bin:$PATH $JVM
> -Dlog4j.configuration=file:$sdir/log4j.properties -classpath
> "$sdir/../../solr-webapp/webapp/WEB-INF/lib/*:$sdir/../../lib/ext/*"
> org.apache.solr.cloud.ZkCLI ${1+"$@"}
> {code}
> which is a reasonable default, but often people want to use a "global" log4j
> configuration. For example, one may define a log4j configuration that writes
> to an external log directory and want to point to this rather than copying it
> to each source checkout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9047) zkcli should allow alternative locations for log4j configuration
Gregory Chanan created SOLR-9047:
Summary: zkcli should allow alternative locations for log4j
configuration
Key: SOLR-9047
URL: https://issues.apache.org/jira/browse/SOLR-9047
Project: Solr
Issue Type: Improvement
Components: scripts and tools, SolrCloud
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
zkcli uses the log4j configuration in the local directory:
{code}
sdir="`dirname \"$0\"`"
PATH=$JAVA_HOME/bin:$PATH $JVM
-Dlog4j.configuration=file:$sdir/log4j.properties -classpath
"$sdir/../../solr-webapp/webapp/WEB-INF/lib/*:$sdir/../../lib/ext/*"
org.apache.solr.cloud.ZkCLI ${1+"$@"}
{code}
which is a reasonable default, but often people want to use a "global" log4j
configuration. For example, one may define a log4j configuration that writes
to an external log directory and want to point to this rather than copying it
to each source checkout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7438) Look into using new HDFS truncate feature in HdfsTransactionLog.
[
https://issues.apache.org/jira/browse/SOLR-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15237739#comment-15237739
]
Gregory Chanan commented on SOLR-7438:
--
bq. It's in 2.7 but apparently 2.7 is not production ready.
Looks like there are production ready 2.7 releases now, e.g. 2.7.2
> Look into using new HDFS truncate feature in HdfsTransactionLog.
>
>
> Key: SOLR-7438
> URL: https://issues.apache.org/jira/browse/SOLR-7438
> Project: Solr
> Issue Type: Improvement
>Reporter: Mark Miller
>Assignee: Mark Miller
>
> Looks like truncate is added in 2.7.
> See HdfsTransactionLog:
> {code}
> // HACK
> // while waiting for HDFS-3107, instead of quickly
> // dropping, we slowly apply
> // This is somewhat brittle, but current usage
> // allows for it
> @Override
> public boolean dropBufferedUpdates() {
> Future future = applyBufferedUpdates();
> if (future != null) {
> try {
> future.get();
> } catch (InterruptedException | ExecutionException e) {
> throw new RuntimeException(e);
> }
> }
> return true;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-8892) Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls
[ https://issues.apache.org/jira/browse/SOLR-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan resolved SOLR-8892. -- Resolution: Fixed Thanks for the review, Mark. Committed to 6.1 and trunk. > Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls > > > Key: SOLR-8892 > URL: https://issues.apache.org/jira/browse/SOLR-8892 > Project: Solr > Issue Type: Bug > Components: JMX, web gui >Affects Versions: 6.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 6.1, trunk > > Attachments: SOLR-8892.patch, SOLR-8892.patch > > > Discussed this with [~markmiller]. SOLR-8869 allows us to not return field > cache entries from the SolrFieldCacheMBean. It would be nice to be a little > more flexible about this; the reason SOLR-8869 was useful was that the field > cache printing was expensive due to periodic monitoring calls to /jmx. But > turning off the field cache entry printing via SOLR-8869 also turns it off > from the web ui. It can be useful for users and admins to be able to view > the cache entries there, non-periodically, without paying a performance > penalty due to /jmx calls. > The proposal here is to allow SolrInfoMBeans to define statistics to be > returned specificially for jmx purposes and to implement a specific instance > of this for the SolrFieldCacheMBean. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8892) Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls
[ https://issues.apache.org/jira/browse/SOLR-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-8892: - Attachment: SOLR-8892.patch Here's a patch that adds the ability to turn on/off the jmx changes. > Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls > > > Key: SOLR-8892 > URL: https://issues.apache.org/jira/browse/SOLR-8892 > Project: Solr > Issue Type: Bug > Components: JMX, web gui >Affects Versions: 6.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 6.1, trunk > > Attachments: SOLR-8892.patch, SOLR-8892.patch > > > Discussed this with [~markmiller]. SOLR-8869 allows us to not return field > cache entries from the SolrFieldCacheMBean. It would be nice to be a little > more flexible about this; the reason SOLR-8869 was useful was that the field > cache printing was expensive due to periodic monitoring calls to /jmx. But > turning off the field cache entry printing via SOLR-8869 also turns it off > from the web ui. It can be useful for users and admins to be able to view > the cache entries there, non-periodically, without paying a performance > penalty due to /jmx calls. > The proposal here is to allow SolrInfoMBeans to define statistics to be > returned specificially for jmx purposes and to implement a specific instance > of this for the SolrFieldCacheMBean. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8892) Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls
[ https://issues.apache.org/jira/browse/SOLR-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222177#comment-15222177 ] Gregory Chanan commented on SOLR-8892: -- perhaps we should have a flag to enable this as well for SolrFieldCacheMBean. > Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls > > > Key: SOLR-8892 > URL: https://issues.apache.org/jira/browse/SOLR-8892 > Project: Solr > Issue Type: Bug > Components: JMX, web gui >Affects Versions: 6.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 6.1, trunk > > Attachments: SOLR-8892.patch > > > Discussed this with [~markmiller]. SOLR-8869 allows us to not return field > cache entries from the SolrFieldCacheMBean. It would be nice to be a little > more flexible about this; the reason SOLR-8869 was useful was that the field > cache printing was expensive due to periodic monitoring calls to /jmx. But > turning off the field cache entry printing via SOLR-8869 also turns it off > from the web ui. It can be useful for users and admins to be able to view > the cache entries there, non-periodically, without paying a performance > penalty due to /jmx calls. > The proposal here is to allow SolrInfoMBeans to define statistics to be > returned specificially for jmx purposes and to implement a specific instance > of this for the SolrFieldCacheMBean. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8892) Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls
[ https://issues.apache.org/jira/browse/SOLR-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222174#comment-15222174 ] Gregory Chanan commented on SOLR-8892: -- Here's a patch with a couple of tests. [~markrmil...@gmail.com] can you take a look? Is this along the lines of what you were thinking? > Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls > > > Key: SOLR-8892 > URL: https://issues.apache.org/jira/browse/SOLR-8892 > Project: Solr > Issue Type: Bug > Components: JMX, web gui >Affects Versions: 6.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 6.1, trunk > > Attachments: SOLR-8892.patch > > > Discussed this with [~markmiller]. SOLR-8869 allows us to not return field > cache entries from the SolrFieldCacheMBean. It would be nice to be a little > more flexible about this; the reason SOLR-8869 was useful was that the field > cache printing was expensive due to periodic monitoring calls to /jmx. But > turning off the field cache entry printing via SOLR-8869 also turns it off > from the web ui. It can be useful for users and admins to be able to view > the cache entries there, non-periodically, without paying a performance > penalty due to /jmx calls. > The proposal here is to allow SolrInfoMBeans to define statistics to be > returned specificially for jmx purposes and to implement a specific instance > of this for the SolrFieldCacheMBean. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8892) Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls
[ https://issues.apache.org/jira/browse/SOLR-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-8892: - Attachment: SOLR-8892.patch > Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls > > > Key: SOLR-8892 > URL: https://issues.apache.org/jira/browse/SOLR-8892 > Project: Solr > Issue Type: Bug > Components: JMX, web gui >Affects Versions: 6.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 6.1, trunk > > Attachments: SOLR-8892.patch > > > Discussed this with [~markmiller]. SOLR-8869 allows us to not return field > cache entries from the SolrFieldCacheMBean. It would be nice to be a little > more flexible about this; the reason SOLR-8869 was useful was that the field > cache printing was expensive due to periodic monitoring calls to /jmx. But > turning off the field cache entry printing via SOLR-8869 also turns it off > from the web ui. It can be useful for users and admins to be able to view > the cache entries there, non-periodically, without paying a performance > penalty due to /jmx calls. > The proposal here is to allow SolrInfoMBeans to define statistics to be > returned specificially for jmx purposes and to implement a specific instance > of this for the SolrFieldCacheMBean. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-8892) Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls
Gregory Chanan created SOLR-8892: Summary: Allow SolrInfoMBeans to return different statistics for /jmx vs web ui calls Key: SOLR-8892 URL: https://issues.apache.org/jira/browse/SOLR-8892 Project: Solr Issue Type: Bug Components: JMX, web gui Affects Versions: 6.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Fix For: 6.1, trunk Discussed this with [~markmiller]. SOLR-8869 allows us to not return field cache entries from the SolrFieldCacheMBean. It would be nice to be a little more flexible about this; the reason SOLR-8869 was useful was that the field cache printing was expensive due to periodic monitoring calls to /jmx. But turning off the field cache entry printing via SOLR-8869 also turns it off from the web ui. It can be useful for users and admins to be able to view the cache entries there, non-periodically, without paying a performance penalty due to /jmx calls. The proposal here is to allow SolrInfoMBeans to define statistics to be returned specificially for jmx purposes and to implement a specific instance of this for the SolrFieldCacheMBean. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-7115) Speed up FieldCache.CacheEntry toString by setting initial StringBuilder capacity
[ https://issues.apache.org/jira/browse/LUCENE-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan resolved LUCENE-7115. Resolution: Fixed Fix Version/s: 6.1 trunk Thanks for the review, Mark, committed to trunk and 6.1 > Speed up FieldCache.CacheEntry toString by setting initial StringBuilder > capacity > - > > Key: LUCENE-7115 > URL: https://issues.apache.org/jira/browse/LUCENE-7115 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: trunk, 6.1 > > Attachments: LUCENE-7115.patch > > > Solr can end up printing a lot of these objects via the JmxMonitoriedMap, see > SOLR-8869 and SOLR-6747 as examples. > From looking at some profiles, a lot of time and memory are spent resizing > the StringBuilder, which doesn't set the initial capacity. > On my cluster, the strings are a bit over 200 chars; I set the initial > capacity to 250 and ran tests calling toString 1000 times. Tests > consistently show 10-15% improvement when setting the initial capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-8869) Optionally disable printing field cache entries in SolrFieldCacheMBean
[ https://issues.apache.org/jira/browse/SOLR-8869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan resolved SOLR-8869. -- Resolution: Fixed Fix Version/s: trunk 6.1 Thanks for the review, Mark, committed to 6.1 and trunk. > Optionally disable printing field cache entries in SolrFieldCacheMBean > -- > > Key: SOLR-8869 > URL: https://issues.apache.org/jira/browse/SOLR-8869 > Project: Solr > Issue Type: Improvement >Affects Versions: 6.1, trunk >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Fix For: 6.1, trunk > > Attachments: SOLR-8869.patch > > > Even with SOLR-6747, we are seeing some pretty load / memory allocation due > to the JmxMonitoredMap. A majority of this seems to be printing the field > cache entries. We should allow admins to disable printing the field cache > entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-8869) Optionally disable printing field cache entries in JmxMonitoredMap
Gregory Chanan created SOLR-8869: Summary: Optionally disable printing field cache entries in JmxMonitoredMap Key: SOLR-8869 URL: https://issues.apache.org/jira/browse/SOLR-8869 Project: Solr Issue Type: Improvement Affects Versions: 6.1, trunk Reporter: Gregory Chanan Assignee: Gregory Chanan Even with SOLR-6747, we are seeing some pretty load / memory allocation due to the JmxMonitoredMap. A majority of this seems to be printing the field cache entries. We should allow admins to disable printing the field cache entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8869) Optionally disable printing field cache entries in SolrFieldCacheMBean
[ https://issues.apache.org/jira/browse/SOLR-8869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-8869: - Summary: Optionally disable printing field cache entries in SolrFieldCacheMBean (was: Optionally disable printing field cache entries in JmxMonitoredMap) > Optionally disable printing field cache entries in SolrFieldCacheMBean > -- > > Key: SOLR-8869 > URL: https://issues.apache.org/jira/browse/SOLR-8869 > Project: Solr > Issue Type: Improvement >Affects Versions: 6.1, trunk >Reporter: Gregory Chanan >Assignee: Gregory Chanan > > Even with SOLR-6747, we are seeing some pretty load / memory allocation due > to the JmxMonitoredMap. A majority of this seems to be printing the field > cache entries. We should allow admins to disable printing the field cache > entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-7115) Speed up FieldCache.CacheEntry toString by setting initial StringBuilder capacity
Gregory Chanan created LUCENE-7115: -- Summary: Speed up FieldCache.CacheEntry toString by setting initial StringBuilder capacity Key: LUCENE-7115 URL: https://issues.apache.org/jira/browse/LUCENE-7115 Project: Lucene - Core Issue Type: Improvement Components: core/search Reporter: Gregory Chanan Assignee: Gregory Chanan Solr can end up printing a lot of these objects via the JmxMonitoriedMap, see SOLR-8869 and SOLR-6747 as examples. >From looking at some profiles, a lot of time and memory are spent resizing the >StringBuilder, which doesn't set the initial capacity. On my cluster, the strings are a bit over 200 chars; I set the initial capacity to 250 and ran tests calling toString 1000 times. Tests consistently show 10-15% improvement when setting the initial capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7115) Speed up FieldCache.CacheEntry toString by setting initial StringBuilder capacity
[ https://issues.apache.org/jira/browse/LUCENE-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated LUCENE-7115: --- Attachment: LUCENE-7115.patch > Speed up FieldCache.CacheEntry toString by setting initial StringBuilder > capacity > - > > Key: LUCENE-7115 > URL: https://issues.apache.org/jira/browse/LUCENE-7115 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: LUCENE-7115.patch > > > Solr can end up printing a lot of these objects via the JmxMonitoriedMap, see > SOLR-8869 and SOLR-6747 as examples. > From looking at some profiles, a lot of time and memory are spent resizing > the StringBuilder, which doesn't set the initial capacity. > On my cluster, the strings are a bit over 200 chars; I set the initial > capacity to 250 and ran tests calling toString 1000 times. Tests > consistently show 10-15% improvement when setting the initial capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7115) Speed up FieldCache.CacheEntry toString by setting initial StringBuilder capacity
[ https://issues.apache.org/jira/browse/LUCENE-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15202202#comment-15202202 ] Gregory Chanan commented on LUCENE-7115: I uploaded a patch based on a previous version, going to upload a new one shortly. > Speed up FieldCache.CacheEntry toString by setting initial StringBuilder > capacity > - > > Key: LUCENE-7115 > URL: https://issues.apache.org/jira/browse/LUCENE-7115 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Reporter: Gregory Chanan >Assignee: Gregory Chanan > > Solr can end up printing a lot of these objects via the JmxMonitoriedMap, see > SOLR-8869 and SOLR-6747 as examples. > From looking at some profiles, a lot of time and memory are spent resizing > the StringBuilder, which doesn't set the initial capacity. > On my cluster, the strings are a bit over 200 chars; I set the initial > capacity to 250 and ran tests calling toString 1000 times. Tests > consistently show 10-15% improvement when setting the initial capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8869) Optionally disable printing field cache entries in SolrFieldCacheMBean
[ https://issues.apache.org/jira/browse/SOLR-8869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-8869: - Attachment: SOLR-8869.patch Here's a patch and a test. Printing of field cache entries can be disabled via setting the system property "disableSolrFieldCacheMBeanEntryList" to "true" > Optionally disable printing field cache entries in SolrFieldCacheMBean > -- > > Key: SOLR-8869 > URL: https://issues.apache.org/jira/browse/SOLR-8869 > Project: Solr > Issue Type: Improvement >Affects Versions: 6.1, trunk >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: SOLR-8869.patch > > > Even with SOLR-6747, we are seeing some pretty load / memory allocation due > to the JmxMonitoredMap. A majority of this seems to be printing the field > cache entries. We should allow admins to disable printing the field cache > entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-8735) Fix CheckHdfsIndexTest test failure
[ https://issues.apache.org/jira/browse/SOLR-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan resolved SOLR-8735. -- Resolution: Fixed Fix Version/s: 6.0 Thanks [~mdrob], committed to trunk. I held off committing to 5.x because it's not clear to me what's going on there...there's no 5.5.1 or 5.6.0 version in jira. This is safe for 6.0 in any case. > Fix CheckHdfsIndexTest test failure > --- > > Key: SOLR-8735 > URL: https://issues.apache.org/jira/browse/SOLR-8735 > Project: Solr > Issue Type: Bug >Reporter: Mike Drob >Assignee: Gregory Chanan > Labels: jenkins, unit-test > Fix For: 6.0 > > Attachments: SOLR-8735.patch > > > I was going to post this on SOLR-7928 but the "Attach File" button is gone > for some reason. > There are some failures from this on Jenkins -- > https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/3977/ > I think we need to give Solr more time to start up before we begin indexing, > patch coming shortly. > I was not able to find any failures on trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8735) Fix CheckHdfsIndexTest test failure
[ https://issues.apache.org/jira/browse/SOLR-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-8735: - Summary: Fix CheckHdfsIndexTest test failure (was: org.apache.solr.index.hdfs.CheckHdfsIndexTest.doTest test failure) > Fix CheckHdfsIndexTest test failure > --- > > Key: SOLR-8735 > URL: https://issues.apache.org/jira/browse/SOLR-8735 > Project: Solr > Issue Type: Bug >Reporter: Mike Drob >Assignee: Gregory Chanan > Labels: jenkins, unit-test > Attachments: SOLR-8735.patch > > > I was going to post this on SOLR-7928 but the "Attach File" button is gone > for some reason. > There are some failures from this on Jenkins -- > https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/3977/ > I think we need to give Solr more time to start up before we begin indexing, > patch coming shortly. > I was not able to find any failures on trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-8735) org.apache.solr.index.hdfs.CheckHdfsIndexTest.doTest test failure
[ https://issues.apache.org/jira/browse/SOLR-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan reassigned SOLR-8735: Assignee: Gregory Chanan > org.apache.solr.index.hdfs.CheckHdfsIndexTest.doTest test failure > - > > Key: SOLR-8735 > URL: https://issues.apache.org/jira/browse/SOLR-8735 > Project: Solr > Issue Type: Bug >Reporter: Mike Drob >Assignee: Gregory Chanan > Labels: jenkins, unit-test > Attachments: SOLR-8735.patch > > > I was going to post this on SOLR-7928 but the "Attach File" button is gone > for some reason. > There are some failures from this on Jenkins -- > https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/3977/ > I think we need to give Solr more time to start up before we begin indexing, > patch coming shortly. > I was not able to find any failures on trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8735) org.apache.solr.index.hdfs.CheckHdfsIndexTest.doTest test failure
[ https://issues.apache.org/jira/browse/SOLR-8735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15166742#comment-15166742 ] Gregory Chanan commented on SOLR-8735: -- +1. Seems reasonable to put in trunk even if there are no failures that you could find. > org.apache.solr.index.hdfs.CheckHdfsIndexTest.doTest test failure > - > > Key: SOLR-8735 > URL: https://issues.apache.org/jira/browse/SOLR-8735 > Project: Solr > Issue Type: Bug >Reporter: Mike Drob > Labels: jenkins, unit-test > Attachments: SOLR-8735.patch > > > I was going to post this on SOLR-7928 but the "Attach File" button is gone > for some reason. > There are some failures from this on Jenkins -- > https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/3977/ > I think we need to give Solr more time to start up before we begin indexing, > patch coming shortly. > I was not able to find any failures on trunk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[ https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122318#comment-15122318 ] Gregory Chanan commented on SOLR-8415: -- the command is updateacls, not updateAcls. > Provide command to switch between non/secure mode in ZK > --- > > Key: SOLR-8415 > URL: https://issues.apache.org/jira/browse/SOLR-8415 > Project: Solr > Issue Type: Improvement > Components: security, SolrCloud >Reporter: Mike Drob >Assignee: Gregory Chanan > Fix For: 5.5, Trunk > > Attachments: SOLR-8415.branch_5x.patch, SOLR-8415.branch_5x.patch, > SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch, > SOLR-8415.patch, SOLR-8415.patch > > > We have the ability to run both with and without zk acls, but we don't have a > great way to switch between the two modes. Most common use case, I imagine, > would be upgrading from an old version that did not support this to a new > version that does, and wanting to protect all of the existing content in ZK, > but it is conceivable that a user might want to remove ACLs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[ https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-8415: - Attachment: SOLR-8415.branch_5x.patch branch 5 patch seems to be based on a previously posted patch, not what was committed. Attached a version based on the committed version. > Provide command to switch between non/secure mode in ZK > --- > > Key: SOLR-8415 > URL: https://issues.apache.org/jira/browse/SOLR-8415 > Project: Solr > Issue Type: Improvement > Components: security, SolrCloud >Reporter: Mike Drob >Assignee: Gregory Chanan > Fix For: Trunk > > Attachments: SOLR-8415.branch_5x.patch, SOLR-8415.branch_5x.patch, > SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch, > SOLR-8415.patch, SOLR-8415.patch > > > We have the ability to run both with and without zk acls, but we don't have a > great way to switch between the two modes. Most common use case, I imagine, > would be upgrading from an old version that did not support this to a new > version that does, and wanting to protect all of the existing content in ZK, > but it is conceivable that a user might want to remove ACLs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[ https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan resolved SOLR-8415. -- Resolution: Fixed Fix Version/s: 5.5 Thanks Mike. In addition to the previous Trunk commit, also committed to 5.5. > Provide command to switch between non/secure mode in ZK > --- > > Key: SOLR-8415 > URL: https://issues.apache.org/jira/browse/SOLR-8415 > Project: Solr > Issue Type: Improvement > Components: security, SolrCloud >Reporter: Mike Drob >Assignee: Gregory Chanan > Fix For: 5.5, Trunk > > Attachments: SOLR-8415.branch_5x.patch, SOLR-8415.branch_5x.patch, > SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch, > SOLR-8415.patch, SOLR-8415.patch > > > We have the ability to run both with and without zk acls, but we don't have a > great way to switch between the two modes. Most common use case, I imagine, > would be upgrading from an old version that did not support this to a new > version that does, and wanting to protect all of the existing content in ZK, > but it is conceivable that a user might want to remove ACLs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8129) HdfsChaosMonkeyNothingIsSafeTest failures
[ https://issues.apache.org/jira/browse/SOLR-8129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1592#comment-1592 ] Gregory Chanan commented on SOLR-8129: -- +1 for Mark's reasoning. > HdfsChaosMonkeyNothingIsSafeTest failures > - > > Key: SOLR-8129 > URL: https://issues.apache.org/jira/browse/SOLR-8129 > Project: Solr > Issue Type: Bug >Reporter: Yonik Seeley > Attachments: fail.151005_064958, fail.151005_080319 > > > New HDFS chaos test in SOLR-8123 hits a number of types of failures, > including shard inconsistency. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[ https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-8415: - Attachment: SOLR-8415.patch Here's a patch with some minor changes: - Changed UPDATEACL to UPDATEACLS to match the constant value - Changed the constant value from updateAcls to updateacls for consistency and the test seems to fail without it - Added retryOnConnLoss to javadoc I'll commit this assuming the tests pass. > Provide command to switch between non/secure mode in ZK > --- > > Key: SOLR-8415 > URL: https://issues.apache.org/jira/browse/SOLR-8415 > Project: Solr > Issue Type: Improvement > Components: security, SolrCloud >Reporter: Mike Drob >Assignee: Gregory Chanan > Fix For: Trunk > > Attachments: SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch, > SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch > > > We have the ability to run both with and without zk acls, but we don't have a > great way to switch between the two modes. Most common use case, I imagine, > would be upgrading from an old version that did not support this to a new > version that does, and wanting to protect all of the existing content in ZK, > but it is conceivable that a user might want to remove ACLs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-8535) Support forcing define-lucene-javadoc-url to be local
[
https://issues.apache.org/jira/browse/SOLR-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan resolved SOLR-8535.
--
Resolution: Fixed
Committed to 5.5 and 6.0.
> Support forcing define-lucene-javadoc-url to be local
> -
>
> Key: SOLR-8535
> URL: https://issues.apache.org/jira/browse/SOLR-8535
> Project: Solr
> Issue Type: Improvement
> Components: Build
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 5.5, 6.0
>
> Attachments: SOLR-8535.patch
>
>
> Today, unless the version name contains '-SNAPSHOT', the documentation will
> generate javadocs with external links, e.g. http://lucene.apache.org/core/.
> For the same reasons given in SOLR-8353 (for jar checksums), it would be nice
> to support specifying a regex to support internal links for versions that do
> not contain '-SNAPSHOT'. An example from SOLR-8353:
> {quote}
> some of us have a fork of the build that has to deal with changing
> dependencies. Our build systems all get to work according to their needs -
> which is a good thing.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[ https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097367#comment-15097367 ] Gregory Chanan commented on SOLR-8415: -- committed to trunk. [~mdrob] if you post a 5x patch, I'll commit. > Provide command to switch between non/secure mode in ZK > --- > > Key: SOLR-8415 > URL: https://issues.apache.org/jira/browse/SOLR-8415 > Project: Solr > Issue Type: Improvement > Components: security, SolrCloud >Reporter: Mike Drob >Assignee: Gregory Chanan > Fix For: Trunk > > Attachments: SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch, > SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch > > > We have the ability to run both with and without zk acls, but we don't have a > great way to switch between the two modes. Most common use case, I imagine, > would be upgrading from an old version that did not support this to a new > version that does, and wanting to protect all of the existing content in ZK, > but it is conceivable that a user might want to remove ACLs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8535) Support forcing define-lucene-javadoc-url to be local
[
https://issues.apache.org/jira/browse/SOLR-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-8535:
-
Summary: Support forcing define-lucene-javadoc-url to be local (was:
Support regex for define-lucene-javadoc-url)
Changed the title, probably more straightforward to just make it a boolean
property to use local paths.
> Support forcing define-lucene-javadoc-url to be local
> -
>
> Key: SOLR-8535
> URL: https://issues.apache.org/jira/browse/SOLR-8535
> Project: Solr
> Issue Type: Improvement
> Components: Build
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 5.5, 6.0
>
>
> Today, unless the version name contains '-SNAPSHOT', the documentation will
> generate javadocs with external links, e.g. http://lucene.apache.org/core/.
> For the same reasons given in SOLR-8353 (for jar checksums), it would be nice
> to support specifying a regex to support internal links for versions that do
> not contain '-SNAPSHOT'. An example from SOLR-8353:
> {quote}
> some of us have a fork of the build that has to deal with changing
> dependencies. Our build systems all get to work according to their needs -
> which is a good thing.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8535) Support forcing define-lucene-javadoc-url to be local
[
https://issues.apache.org/jira/browse/SOLR-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated SOLR-8535:
-
Attachment: SOLR-8535.patch
> Support forcing define-lucene-javadoc-url to be local
> -
>
> Key: SOLR-8535
> URL: https://issues.apache.org/jira/browse/SOLR-8535
> Project: Solr
> Issue Type: Improvement
> Components: Build
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 5.5, 6.0
>
> Attachments: SOLR-8535.patch
>
>
> Today, unless the version name contains '-SNAPSHOT', the documentation will
> generate javadocs with external links, e.g. http://lucene.apache.org/core/.
> For the same reasons given in SOLR-8353 (for jar checksums), it would be nice
> to support specifying a regex to support internal links for versions that do
> not contain '-SNAPSHOT'. An example from SOLR-8353:
> {quote}
> some of us have a fork of the build that has to deal with changing
> dependencies. Our build systems all get to work according to their needs -
> which is a good thing.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[
https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094505#comment-15094505
]
Gregory Chanan commented on SOLR-8415:
--
{quote}Going secure -> insecure, probably can do against a running
cluster.{quote}
Why probably? Don't you need to update solr.xml?
{quote}Going to a different secure configuration, yea, would need to update
solr.xml. I think that's sufficiently covered in the other sections of the
page, though.{quote}
Which page are you referring to?
https://cwiki.apache.org/confluence/display/solr/ZooKeeper+Access+Control?
Maybe I'm missing something, but that all seems to be about initial setup.
> Provide command to switch between non/secure mode in ZK
> ---
>
> Key: SOLR-8415
> URL: https://issues.apache.org/jira/browse/SOLR-8415
> Project: Solr
> Issue Type: Improvement
> Components: security, SolrCloud
>Reporter: Mike Drob
>Assignee: Gregory Chanan
> Fix For: Trunk
>
> Attachments: SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch
>
>
> We have the ability to run both with and without zk acls, but we don't have a
> great way to switch between the two modes. Most common use case, I imagine,
> would be upgrading from an old version that did not support this to a new
> version that does, and wanting to protect all of the existing content in ZK,
> but it is conceivable that a user might want to remove ACLs as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[
https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094953#comment-15094953
]
Gregory Chanan commented on SOLR-8415:
--
bq. 2) The FunctionalInterface is the right way to do this, but it would be
nice to have in 5.x as well. I'm willing to create a separate patch that drops
it specifically for 5.x
Sure, let's do a separate patch if that's what you prefer.
{quote}3) Naming things is hard, I hadn't considered the possible confusion
there but now that you point it out I can't unsee it. Maybe setAcl and
updateAcls? updateAllAcl? Is it fine to still call the ZkCli command resetacl?
Maybe reinitacl? Original intent was to convey something with gravitas.{quote}
I think setAcl and updateAcls is good. I get the gravitas point, but anything
with "re" I think is going to cause more confusion because it implies some
initial state that may or may not be true. I'd just match the ZkCli command
name to the SolrZkClient function name, so in this case updateAcls. But if you
feel really strongly for something else, let's do that.
> Provide command to switch between non/secure mode in ZK
> ---
>
> Key: SOLR-8415
> URL: https://issues.apache.org/jira/browse/SOLR-8415
> Project: Solr
> Issue Type: Improvement
> Components: security, SolrCloud
>Reporter: Mike Drob
>Assignee: Gregory Chanan
> Fix For: Trunk
>
> Attachments: SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch,
> SOLR-8415.patch
>
>
> We have the ability to run both with and without zk acls, but we don't have a
> great way to switch between the two modes. Most common use case, I imagine,
> would be upgrading from an old version that did not support this to a new
> version that does, and wanting to protect all of the existing content in ZK,
> but it is conceivable that a user might want to remove ACLs as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[
https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094863#comment-15094863
]
Gregory Chanan commented on SOLR-8415:
--
bq. Aside: There has to be a better way to share this than just pasting my
proposed changes in a comment each time.
Hmm, you could just go ahead and change the wiki I guess? Don't know of a
better way.
Patch looks good. Some comments:
1) Provide javadoc for {code}+ public Stat setACL(String path, List acls,
boolean retryOnConnLoss) throws InterruptedException, KeeperException {
{code}
2) Using the @FunctionalInterface stuff means I can't commit this to 5.x, are
you okay with that?
3) The set vs reset in SolrZkClient is kind of confusing. As it stands, set
means a single node, reset means recursive. That's not the common usage of the
words, e.g. we don't have clean vs reclean to mean a single node vs recursively
(there it's delete vs clean). I don't know which terminology to use; reset
seems to imply changing from ACLs that existed (either from secure-> other
secure or secure->unsecure), while set seems to imply changing from unsecure to
secure. This is really a problem with ZooKeeper lacking declarative APIs (what
you actually want is an API that says "after this runs, the ACLs are this" --
you don't really care how it actually happens). Given that, what makes the
most sense to me is to just call everything "set", since this matches the ZK
API that you are calling. Maybe instead of setAcl vs resetACLs you should have
setAcl vs setAcls or setAcl vs setAclsRecursively. Thoughts?
> Provide command to switch between non/secure mode in ZK
> ---
>
> Key: SOLR-8415
> URL: https://issues.apache.org/jira/browse/SOLR-8415
> Project: Solr
> Issue Type: Improvement
> Components: security, SolrCloud
>Reporter: Mike Drob
>Assignee: Gregory Chanan
> Fix For: Trunk
>
> Attachments: SOLR-8415.patch, SOLR-8415.patch, SOLR-8415.patch,
> SOLR-8415.patch
>
>
> We have the ability to run both with and without zk acls, but we don't have a
> great way to switch between the two modes. Most common use case, I imagine,
> would be upgrading from an old version that did not support this to a new
> version that does, and wanting to protect all of the existing content in ZK,
> but it is conceivable that a user might want to remove ACLs as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[ https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093095#comment-15093095 ] Gregory Chanan commented on SOLR-8415: -- One thing I'm a bit unclear on from the docs: what is the recommended strategy for changing the permissions? Stop the servers, change the solr.xml, run the ZkCLI command, start the servers? Would be good to specify that. What is the current plan with the patch? Do you think it's ready to go or are you still working on more testing? > Provide command to switch between non/secure mode in ZK > --- > > Key: SOLR-8415 > URL: https://issues.apache.org/jira/browse/SOLR-8415 > Project: Solr > Issue Type: Improvement > Components: security, SolrCloud >Reporter: Mike Drob >Assignee: Gregory Chanan > Fix For: Trunk > > Attachments: SOLR-8415.patch, SOLR-8415.patch > > > We have the ability to run both with and without zk acls, but we don't have a > great way to switch between the two modes. Most common use case, I imagine, > would be upgrading from an old version that did not support this to a new > version that does, and wanting to protect all of the existing content in ZK, > but it is conceivable that a user might want to remove ACLs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-8535) Support regex for define-lucene-javadoc-url
Gregory Chanan created SOLR-8535:
Summary: Support regex for define-lucene-javadoc-url
Key: SOLR-8535
URL: https://issues.apache.org/jira/browse/SOLR-8535
Project: Solr
Issue Type: Improvement
Components: Build
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Fix For: 5.5, 6.0
Today, unless the version name contains '-SNAPSHOT', the documentation will
generate javadocs with external links, e.g. http://lucene.apache.org/core/.
For the same reasons given in SOLR-8353 (for jar checksums), it would be nice
to support specifying a regex to support internal links for versions that do
not contain '-SNAPSHOT'. An example from SOLR-8353:
{quote}
some of us have a fork of the build that has to deal with changing
dependencies. Our build systems all get to work according to their needs -
which is a good thing.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[
https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15069982#comment-15069982
]
Gregory Chanan commented on SOLR-8415:
--
{quote}It's pretty straightforward to do this with access to writing some java
classes, but at that point I'm not sure who our audience is.{quote}
The point was the proposed documentation was incorrect. Our audience is
someone who wants to switch from one secure acl regime to another. If you
don't think that's a popular enough use case to warrant documentation then I'd
suggest getting rid of that part including the incorrect information.
> Provide command to switch between non/secure mode in ZK
> ---
>
> Key: SOLR-8415
> URL: https://issues.apache.org/jira/browse/SOLR-8415
> Project: Solr
> Issue Type: Improvement
> Components: security, SolrCloud
>Reporter: Mike Drob
>Assignee: Gregory Chanan
> Fix For: Trunk
>
> Attachments: SOLR-8415.patch, SOLR-8415.patch
>
>
> We have the ability to run both with and without zk acls, but we don't have a
> great way to switch between the two modes. Most common use case, I imagine,
> would be upgrading from an old version that did not support this to a new
> version that does, and wanting to protect all of the existing content in ZK,
> but it is conceivable that a user might want to remove ACLs as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[ https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068609#comment-15068609 ] Gregory Chanan commented on SOLR-8415: -- bq. Is this true? Let's say you wanted to switch from secure setup old: (old acls, old credentials) to secure setup new (new acls, new credentials). You can call resetacls with (old acls + new acls, old credentials). Then call reset acls with (new acls, new credentials). That requires an intermediate step, but it isn't unsecure. maybe change your test to do this (or do both this and the secure/non-secure version, should be simple to do both probably). > Provide command to switch between non/secure mode in ZK > --- > > Key: SOLR-8415 > URL: https://issues.apache.org/jira/browse/SOLR-8415 > Project: Solr > Issue Type: Improvement > Components: security, SolrCloud >Reporter: Mike Drob >Assignee: Gregory Chanan > Fix For: Trunk > > Attachments: SOLR-8415.patch, SOLR-8415.patch > > > We have the ability to run both with and without zk acls, but we don't have a > great way to switch between the two modes. Most common use case, I imagine, > would be upgrading from an old version that did not support this to a new > version that does, and wanting to protect all of the existing content in ZK, > but it is conceivable that a user might want to remove ACLs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[
https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15068778#comment-15068778
]
Gregory Chanan commented on SOLR-8415:
--
{quote}Optional path or required path? Could still default to / if no path
given, or could make the path required for consistency. Or could accept
multiple paths.
I think operating on / will be the most common use case, so it would make sense
to default to it, but I'll defer to you on this.{quote}
Whatever you think is best.
{quote}Why don't you support retryOnConnLoss?
Not sure what this means.{quote}
See a bunch of the other commands in SolrZkClient, like makePath. They support
a retryOnConnLoss parameter, which would be useful here.
{quote}The existing test does this. Set acls on /, test on
/collections/collection1{quote}
My mistake. I'd check "/" as well, that sort of thing is easy to screw up.
> Provide command to switch between non/secure mode in ZK
> ---
>
> Key: SOLR-8415
> URL: https://issues.apache.org/jira/browse/SOLR-8415
> Project: Solr
> Issue Type: Improvement
> Components: security, SolrCloud
>Reporter: Mike Drob
>Assignee: Gregory Chanan
> Fix For: Trunk
>
> Attachments: SOLR-8415.patch, SOLR-8415.patch
>
>
> We have the ability to run both with and without zk acls, but we don't have a
> great way to switch between the two modes. Most common use case, I imagine,
> would be upgrading from an old version that did not support this to a new
> version that does, and wanting to protect all of the existing content in ZK,
> but it is conceivable that a user might want to remove ACLs as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[
https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067043#comment-15067043
]
Gregory Chanan commented on SOLR-8415:
--
Some comments on the proposed documentation:
{code}but will not protect the already existing data{code}
"data" is ambiguous. ZooKeeper metadata? ZNodes?
{code}it ma be necessary{code}
may be necessary
{code}use an unsecure intermediate step.{code}
Is this true? Let's say you wanted to switch from secure setup old: (old acls,
old credentials) to secure setup new (new acls, new credentials). You can
call resetacls with (old acls + new acls, old credentials). Then call reset
acls with (new acls, new credentials). That requires an intermediate step, but
it isn't unsecure.
> Provide command to switch between non/secure mode in ZK
> ---
>
> Key: SOLR-8415
> URL: https://issues.apache.org/jira/browse/SOLR-8415
> Project: Solr
> Issue Type: Improvement
> Components: security, SolrCloud
>Reporter: Mike Drob
> Fix For: Trunk
>
> Attachments: SOLR-8415.patch, SOLR-8415.patch
>
>
> We have the ability to run both with and without zk acls, but we don't have a
> great way to switch between the two modes. Most common use case, I imagine,
> would be upgrading from an old version that did not support this to a new
> version that does, and wanting to protect all of the existing content in ZK,
> but it is conceivable that a user might want to remove ACLs as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[ https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan reassigned SOLR-8415: Assignee: Gregory Chanan > Provide command to switch between non/secure mode in ZK > --- > > Key: SOLR-8415 > URL: https://issues.apache.org/jira/browse/SOLR-8415 > Project: Solr > Issue Type: Improvement > Components: security, SolrCloud >Reporter: Mike Drob >Assignee: Gregory Chanan > Fix For: Trunk > > Attachments: SOLR-8415.patch, SOLR-8415.patch > > > We have the ability to run both with and without zk acls, but we don't have a > great way to switch between the two modes. Most common use case, I imagine, > would be upgrading from an old version that did not support this to a new > version that does, and wanting to protect all of the existing content in ZK, > but it is conceivable that a user might want to remove ACLs as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8415) Provide command to switch between non/secure mode in ZK
[
https://issues.apache.org/jira/browse/SOLR-8415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15067054#comment-15067054
]
Gregory Chanan commented on SOLR-8415:
--
{code}+} else if (line.getOptionValue(CMD).equals(RESETACL)) {
+ zkClient.resetACLs("/");
{code}
should take a path, like CLEAN
{code}
+List children = getChildren(znode, null, true);
{code}
catch NoNodeException, like CLEAN
{code}
+ getSolrZooKeeper().setACL(znode, getZkACLProvider().getACLsToAdd(znode),
-1);
{code}
Will this work if the version of the znode is set?
Why don't you support retryOnConnLoss?
Would be good to test that the acls get applied recursively
> Provide command to switch between non/secure mode in ZK
> ---
>
> Key: SOLR-8415
> URL: https://issues.apache.org/jira/browse/SOLR-8415
> Project: Solr
> Issue Type: Improvement
> Components: security, SolrCloud
>Reporter: Mike Drob
>Assignee: Gregory Chanan
> Fix For: Trunk
>
> Attachments: SOLR-8415.patch, SOLR-8415.patch
>
>
> We have the ability to run both with and without zk acls, but we don't have a
> great way to switch between the two modes. Most common use case, I imagine,
> would be upgrading from an old version that did not support this to a new
> version that does, and wanting to protect all of the existing content in ZK,
> but it is conceivable that a user might want to remove ACLs as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-8446) Allow failonerror to be configured for unit tests
Gregory Chanan created SOLR-8446: Summary: Allow failonerror to be configured for unit tests Key: SOLR-8446 URL: https://issues.apache.org/jira/browse/SOLR-8446 Project: Solr Issue Type: Improvement Components: Tests Reporter: Gregory Chanan Assignee: Gregory Chanan Currently, failonerror is hard coded to false for the "test" task at the top level scope. For jenkins runs, it would be useful to be able to configure this because: 1) unit tests runs are flaky 2) jenkins can detect test failures even if the the test task itself passes and mark the build yellow (which happens if failonerror is true) Therefore, this allows some nicer visualization of the jenkins history, i.e.: green if everything is good yellow if unit tests are failing (most likely flaky) red if compile / precommit, etc are broken -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8446) Allow failonerror to be configured for unit tests
[ https://issues.apache.org/jira/browse/SOLR-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated SOLR-8446: - Attachment: SOLR-8446.patch Here's a trivial patch that makes failonerror configurable for the top-level "test" task. This works for the use case above but depending on what you want to happen, you need to understand the ant structure a bit. For example: 1) If the test task fails, no test report will be generated. So you may have to specify -Dtests.ifNoTests=ignore as well or the task will still fail. If all the lucene tests pass and the solr tests fail (or visa versa), the task will succeed even without specifying -Dtests.ifNoTests=ignore because the passing subdir will generate the report. 2) This setting affects the entire task, so calling "ant test" can pass even if say, compilation is broken. You may want to specify something like "ant compile compile-test test" or similar to avoid this. > Allow failonerror to be configured for unit tests > - > > Key: SOLR-8446 > URL: https://issues.apache.org/jira/browse/SOLR-8446 > Project: Solr > Issue Type: Improvement > Components: Tests >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: SOLR-8446.patch > > > Currently, failonerror is hard coded to false for the "test" task at the top > level scope. For jenkins runs, it would be useful to be able to configure > this because: > 1) unit tests runs are flaky > 2) jenkins can detect test failures even if the the test task itself passes > and mark the build yellow (which happens if failonerror is true) > Therefore, this allows some nicer visualization of the jenkins history, i.e.: > green if everything is good > yellow if unit tests are failing (most likely flaky) > red if compile / precommit, etc are broken -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8416) Solr collection creation API should return after all cores are alive
[ https://issues.apache.org/jira/browse/SOLR-8416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061319#comment-15061319 ] Gregory Chanan commented on SOLR-8416: -- I'd see what hbase or other systems do. > Solr collection creation API should return after all cores are alive > - > > Key: SOLR-8416 > URL: https://issues.apache.org/jira/browse/SOLR-8416 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Reporter: Michael Sun > Attachments: SOLR-8416.patch > > > Currently the collection creation API returns once all cores are created. In > large cluster the cores may not be alive for some period of time after cores > are created. For any thing requested during that period, Solr appears > unstable and can return failure. Therefore it's better the collection > creation API waits for all cores to become alive and returns after that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8416) Solr collection creation API should return after all cores are alive
[ https://issues.apache.org/jira/browse/SOLR-8416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061258#comment-15061258 ] Gregory Chanan commented on SOLR-8416: -- The patch claims to be looking at all shards being active but is actually looking at all replicas being active, right? It's also inconsistent with the other creation commands now, e.g. CREATESHARD/ADDREPLICA will return as soon as the replicas are created while this will wait until all the replicas are active. >From a client perspective, what do you actually want? I don't think it's that >all replicas are active at one time; given a large enough cluster that can be >unlikely. Some possibilities: 1) all the replicas were able to become active (you'd have to track this separately) 2) the collection is "usable" from the client -- each shard has a leader that is live and active? > Solr collection creation API should return after all cores are alive > - > > Key: SOLR-8416 > URL: https://issues.apache.org/jira/browse/SOLR-8416 > Project: Solr > Issue Type: Bug > Components: SolrCloud >Reporter: Michael Sun > Attachments: SOLR-8416.patch > > > Currently the collection creation API returns once all cores are created. In > large cluster the cores may not be alive for some period of time after cores > are created. For any thing requested during that period, Solr appears > unstable and can return failure. Therefore it's better the collection > creation API waits for all cores to become alive and returns after that. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8178) Solr should support jetty's obfuscated password for ssl truststore
[ https://issues.apache.org/jira/browse/SOLR-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057379#comment-15057379 ] Gregory Chanan commented on SOLR-8178: -- Do those end up in 'ps' output? > Solr should support jetty's obfuscated password for ssl truststore > -- > > Key: SOLR-8178 > URL: https://issues.apache.org/jira/browse/SOLR-8178 > Project: Solr > Issue Type: Improvement >Reporter: Noble Paul >Assignee: Noble Paul > Fix For: 5.5 > > Attachments: SOLR-8178.patch, SOLR-8178.patch > > > Jetty supports obfuscating password that used for SSL keystore and > truststore. Solr should support that as well -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8378) Add upconfig and downconfig commands to the bin/solr script and managed-only schema to configsets
[ https://issues.apache.org/jira/browse/SOLR-8378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047606#comment-15047606 ] Gregory Chanan commented on SOLR-8378: -- Some thoughts: I'm on board with having useful commands in one place rather than requiring end users know about zkcli. That said, I don't think adding more uncategorized comands to the same script is the correct way to go. In our distribution (CDH) we have had script that does a bunch of different actions on solr/zk and I've found it's pretty confusing to users what command actually goes where. Ideally the users wouldn't have to know that sort of information (at least when starting up, but I think quickstart is a different enough use case to warrant special consideration), but that's just not practical -- consider if the configs znode has ACLs enabled -- you need to pass a reasonable endpoint-specific error message back to the user, you have to have an end-point specific mechanism to pass kerberos credentials (does this script work in a secure environment)?, etc. So what will happen if we continue along this path is we'll have a bunch of different useful commands where it is unclear to users what information they actually need to provide without looking it up each time. Heck, I wrote a lot of the commands in our distribution and I get confused :). So, my suggestion is that we break up the commands into "subtopics" based on the endpoint (the solr http endpoint can be an unnamed default). So long story short, I'd argue for naming these: zk upconfig zk downconfig or something like that. > Add upconfig and downconfig commands to the bin/solr script and managed-only > schema to configsets > - > > Key: SOLR-8378 > URL: https://issues.apache.org/jira/browse/SOLR-8378 > Project: Solr > Issue Type: Improvement > Components: scripts and tools >Affects Versions: 5.4, Trunk >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8378.patch, SOLR-8378.patch, SOLR-8378.patch, > SOLR-8378.patch, SOLR-8378.patch > > > It would be convenient to be able to upload and download arbitrary configsets > to Zookeeper. > This _might_ be the last thing we need before not requiring users be aware of > zkcli, which is awkward. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
