[ANNOUNCE] Apache Solr 8.8.0 released

[Noble Paul]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Solr is the popular, blazing fast, open source NoSQL search platform from
the Apache Lucene project. Its major features include powerful full-text
search, hit highlighting, faceted search and analytics, rich document
parsing, geospatial search, extensive REST APIs as well as parallel SQL.
Solr is enterprise grade, secure and highly scalable, providing fault
tolerant distributed search and indexing, and powers the search and
navigation features of many of the world's largest internet sites.

The release is available for immediate download at:

 https://lucene.apache.org/solr/downloads.html

Please read CHANGES.txt for a detailed list of changes:

 https://lucene.apache.org/solr/8_8_0/changes/Changes.html Solr 8.8.0

Release Highlights:

Reducing overseer bottlenecks using per-replica states. More stability and
lesser load on large cluster that use this feature.

Better restart and collection creation performance.

Interleaving support in Learning To Rank

A summary of important changes is published in the Solr Reference Guide at:

 https://lucene.apache.org/solr/guide/8_8/solr-upgrade-notes.html.

For the most exhaustive list, see the full release notes at
https://lucene.apache.org/solr/8_8_0/changes/Changes.html

or

by viewing the CHANGES.txt file accompanying the distribution. Solr's
release notes usually don't include Lucene layer changes. Lucene's release
notes are at

https://lucene.apache.org/core/8_8_0/changes/Changes.html

- -
Noble Paul
-BEGIN PGP SIGNATURE-
Version: FlowCrypt Email Encryption 8.0.0
Comment: Seamlessly send and receive encrypted email

wsFzBAEBCgAGBQJgF/RnACEJEMOP9ew/z9s+FiEEz85fu5IMPHRc7uCEw4/1
7D/P2z6fzRAAm4AKbeIGWfPK+0nsrZCAPaDucGZYVL0lPQr3eF4jnmhi60dF
Sv9rD5Mq5ZSTTuJlpwoaxowxVp4M1tV1vmCdfBRkgoUD3dwS/snryr/AK69R
zdjjV/BABtcMNA7cMYIrkolGl37g4kI1alLfU36Uf/3M0NfUcw0keW1XuMOr
uV7AzXhZGw4eL4LJt7I7gXJs1kgE6/sPSmoKBVckKisrruiUSYmH9r/EhXXU
YB8cxd5tenMrchbjcOquC9X2JJjB++/LyJw3mFNIO5W3UpjqwtI8IGDo1Sxl
fM32FuAWVVDZsiBKXuRzsIO/iEPfgZFfTcoSJkD0Rt/Q6gJPZIuBmiUFaYfs
9fzufNDuXdPKFEndSHfwdPMJwvk3XA5+xYzhkcQH+3FKOPmYXkvLolOC3j+r
ZtbgI421jDIahpVPbFtgUPB2dM3mw34B73wP5MIOHHxz22tVKe6PBOeihccK
mOr0r1tZHR+11aijYf+Nlhv3hpbpRoDbQ7pRkRyu53Od47p6itZAi60TFFIJ
bDw26wZRNRrEuYhriJUeM7ahvJNlcE6VaO0szUDL5g/x2Oa9jKMHPpsUF9pS
9HbJWcnflxq0iU+sfdv7Eoxzv6zkXMTUsbpT2XjKcZZN5jd2rWV3JfiU6FiZ
jpqJBHzwGan9qKKswNKyDKhoa2jPdSYIbqQ=
=NbSI
-END PGP SIGNATURE-

Re: Survey on ManagedResources feature

[Noble Paul]

The end point is served by restlet. So, your rules are not going to be
honored. The rules work only if it is served by a Solr request handler

On Wed, Aug 12, 2020, 12:46 AM Jason Gerlowski 
wrote:

> Hey Noble,
>
> Can you explain what you mean when you say it's not secured?  Just for
> those of us who haven't been following the discussion so far?  On the
> surface of things users taking advantage of our RuleBasedAuth plugin
> can secure this API like they can any other HTTP API.  Or are you
> talking about some other security aspect here?
>
> Jason
>
> On Tue, Aug 11, 2020 at 9:55 AM Noble Paul  wrote:
> >
> > Hi all,
> > The end-point for Managed resources is not secured. So it needs to be
> > fixed/eliminated.
> >
> > I would like to know what is the level of adoption for that feature
> > and if it is a critical feature for users.
> >
> > Another possibility is to offer a replacement for the feature using a
> > different API
> >
> > Your feedback will help us decide on what a potential solution should be
> >
> > --
> > -
> > Noble Paul
>

Survey on ManagedResources feature

[Noble Paul]

Hi all,
The end-point for Managed resources is not secured. So it needs to be
fixed/eliminated.

I would like to know what is the level of adoption for that feature
and if it is a critical feature for users.

Another possibility is to offer a replacement for the feature using a
different API

Your feedback will help us decide on what a potential solution should be

-- 
-
Noble Paul

Re: [ANNOUNCE] Apache Solr 8.6.0 released

[Noble Paul]

ude powerful
> > > > full-text
> > > > > > search, hit highlighting, faceted search, dynamic clustering,
> > database
> > > > > > integration, rich document handling, and geospatial search. Solr is
> > > > highly
> > > > > > scalable, providing fault tolerant distributed search and
> > indexing, and
> > > > > > powers the search and navigation features of many of the world's
> > > > largest
> > > > > > internet sites.
> > > > > >
> > > > > >
> > > > > > Solr 8.6.0 is available for immediate download at:
> > > > > >
> > > > > >
> > > > > >   <https://lucene.apache.org/solr/downloads.html>
> > > > > >
> > > > > >
> > > > > > ### Solr 8.6.0 Release Highlights:
> > > > > >
> > > > > >
> > > > > >  * Cross-Collection Join Queries: Join queries can now work
> > > > > > cross-collection, even when shared or when spanning nodes.
> > > > > >
> > > > > >  * Search: Performance improvement for some types of queries when
> > exact
> > > > > > hit count isn't needed by using BlockMax WAND algorithm.
> > > > > >
> > > > > >  * Streaming Expression: Percentiles and standard deviation
> > > > aggregations
> > > > > > added to stats, facet and time series.  Streaming expressions
> > added to
> > > > > > /export handler.  Drill Streaming Expression for efficient and
> > accurate
> > > > > > high cardinality aggregation.
> > > > > >
> > > > > >  * Package manager: Support for cluster (CoreContainer) level
> > plugins.
> > > > > >
> > > > > >  * Health Check: HealthCheckHandler can now require that all cores
> > are
> > > > > > healthy before returning OK.
> > > > > >
> > > > > >  * Zookeeper read API: A read API at /api/cluster/zk/* to fetch
> > raw ZK
> > > > > > data and view contents of a ZK directory.
> > > > > >
> > > > > >  * Admin UI: New panel with security info in admin UI's dashboard.
> > > > > >
> > > > > >  * Query DSL: Support for {param:ref} and {bool: {excludeTags:""}}
> > > > > >
> > > > > >  * Ref Guide: Major redesign of Solr's documentation.
> > > > > >
> > > > > >
> > > > > > Please read CHANGES.txt for a full list of new features and
> > changes:
> > > > > >
> > > > > >
> > > > > >   <https://lucene.apache.org/solr/8_6_0/changes/Changes.html>
> > > > > >
> > > > > >
> > > > > > Solr 8.6.0 also includes features, optimizations  and bugfixes in
> > the
> > > > > > corresponding Apache Lucene release:
> > > > > >
> > > > > >
> > > > > >   <https://lucene.apache.org/core/8_6_0/changes/Changes.html>
> > > > > >
> > > > > >
> > > > > > Note: The Apache Software Foundation uses an extensive mirroring
> > > > network
> > > > > > for
> > > > > >
> > > > > > distributing releases. It is possible that the mirror you are
> > using may
> > > > > > not have
> > > > > >
> > > > > > replicated the release yet. If that is the case, please try another
> > > > mirror.
> > > > > >
> > > > > > This also applies to Maven access.
> > > > > >
> > > >
> >



-- 
-
Noble Paul

Re: [EXTERNAL] Getting rid of Master/Slave nomenclature in Solr

[Noble Paul]

Do we even call it the master/slave mode? I thought we had 2 modes

* Standalone mode
* SolrCloud mode

On Wed, Jun 24, 2020 at 3:00 AM Tomás Fernández Löbbe
 wrote:
>
> I agree in general with what Trey and Jan said and have suggested. I
> personally like to use "leader/follower". It's true that somewhat collides
> with SolrCloud terminology, but that's not a problem IMO, now that replica
> types exist, the “role” of the replica (leader vs. non-leader/follower)
> doesn’t specify the internals of how they behave, the replica type defines
> that. So, in a non-SolrCloud world, they would still be leader/followers
> regardless of how they perform that role.
>
> I also agree that the name of the role is not that important, more the
> "mode" of the architecture needs to be renamed. We tend to refer to
> "SolrCloud mode" and "Master/Slave mode", the main part in all this (IMO)
> is to change that "mode" name. I kind of like Trey's suggestion of "Managed
> Clustering" vs. "Manual Clustering" Mode (Or "managed" vs "manual"), but
> still haven't made up my mind (especially the fact that "manual" usually
> doesn't really mean "manual", is just "you build your tools”)…
>
> On Fri, Jun 19, 2020 at 1:38 PM Walter Underwood 
> wrote:
>
> > > On Jun 19, 2020, at 7:48 AM, Phill Campbell
> >  wrote:
> > >
> > > Delegator - Handler
> > >
> > > A common pattern we are all aware of. Pretty simple.
> >
> > The Solr master does not delegate and the slave does not handle.
> > The master is a server that handles replication requests from the
> > slave.
> >
> > Delegator/handler is a common pattern, but it is not the pattern
> > that describes traditional Solr replication.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> >



-- 
-
Noble Paul

Re: Getting rid of Master/Slave nomenclature in Solr

[Noble Paul]

Looking at the code I see a 692 occurrences of the word "slave".
Mostly variable names and ref guide docs.

The word "slave" is present in the responses as well. Any change in
the request param/response payload is backward incompatible.

I have no objection to changing the names in ref guide and other
internal variables. Going ahead with backward incompatible changes is
painful. If somebody has the appetite to take it up, it's OK

If we must change, master/follower can be a good enough option.

master (noun): A man in charge of an organization or group.
master(adj) : having or showing very great skill or proficiency.
master(verb): acquire complete knowledge or skill in (a subject,
technique, or art).
master (verb): gain control of; overcome.

I hope nobody has a problem with the term "master"

On Thu, Jun 18, 2020 at 3:19 PM Ilan Ginzburg  wrote:
>
> Would master/follower work?
>
> Half the rename work while still getting rid of the slavery connotation...
>
>
> On Thu 18 Jun 2020 at 07:13, Walter Underwood  wrote:
>
> > > On Jun 17, 2020, at 4:00 PM, Shawn Heisey  wrote:
> > >
> > > It has been interesting watching this discussion play out on multiple
> > open source mailing lists.  On other projects, I have seen a VERY high
> > level of resistance to these changes, which I find disturbing and
> > surprising.
> >
> > Yes, it is nice to see everyone just pitch in and do it on this list.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> >



-- 
-
Noble Paul

Re: Getting rid of Master/Slave nomenclature in Solr

[Noble Paul]

I really do not see a reason why a master/slave terminology is a problem.
We do not have slavery anywhere in the world. Should we also remove it from
the dictionary?

The old mode is going to go away anyway. Why waste time bikeshedding on
this?

On Thu, Jun 18, 2020, 12:04 PM Trey Grainger  wrote:

> @Shawn,
>
> Ok, yeah, apologies, my semantics were wrong.
>
> I was thinking that a TLog replica is a follower role only and becomes an
> NRT replica if it gets elected leader. From a pure semantics standpoint,
> though, I guess technically the TLog replica doesn't "become" an NRT
> replica, but just "acts the same" as if it was an NRT replica when it gets
> elected as leader. From the docs regarding TLog replicas: "This type of
> replica maintains a transaction log but does not index document changes
> locally... When this type of replica needs to update its index, it does so
> by replicating the index from the leader... If it does become a leader, it
> will behave the same as if it was a NRT type of replica."
>
> The Tlog replicas are a bit of a red herring to the point I was making,
> though, which is that Pull Replicas in SolrCloud mode and Slaves in
> non-SolrCloud mode both just pull the index from the leader/master and as
> opposed to updates being pushed the other way. As such, I don't see a
> meaningful distinction between master/slave and leader/follower behavior in
> non-SolrCloud mode vs. SolrCloud mode for the specific functionality we're
> talking about renaming (Solr cores that pull indices from other Solr
> cores).
>
> At any rate, this is not a hill I care to die on. My belief is that it's
> better to have consistent terminology for what I see as essentially the
> same functionality. I respect that others disagree and would rather
> introduce new terminology to clearly distinguish between modes. Regardless
> of the naming decided on, I'm in support of removing the master/slave
> nomenclature.
>
> Trey Grainger
> Founder, Searchkernel
> https://searchkernel.com
>
> On Wed, Jun 17, 2020 at 7:00 PM Shawn Heisey  wrote:
>
> > On 6/17/2020 2:36 PM, Trey Grainger wrote:
> > > 2) TLOG - which can only serve in the role of follower
> >
> > This is inaccurate.  TLOG can become leader.  If that happens, then it
> > functions exactly like an NRT leader.
> >
> > I'm aware that saying the following is bikeshedding ... but I do think
> > it would be as mistake to use any existing SolrCloud terminology for
> > non-cloud deployments, including the word "replica".  The top contenders
> > I have seen to replace master/slave in Solr are primary/secondary and
> > publisher/subscriber.
> >
> > It has been interesting watching this discussion play out on multiple
> > open source mailing lists.  On other projects, I have seen a VERY high
> > level of resistance to these changes, which I find disturbing and
> > surprising.
> >
> > Thanks,
> > Shawn
> >
>

Fwd: [ANNOUNCE] Apache Solr 7.7.3 released

[Noble Paul]

-- Forwarded message -
From: Noble Paul 
Date: Tue, Apr 28, 2020, 4:40 PM
Subject: [ANNOUNCE] Apache Solr 7.7.3 released
To: 



-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

28 April 2020, Apache Solr™ 7.7.3 available

The Lucene PMC is pleased to announce the release of Apache Solr 7.7.3.

Solr is the popular, blazing fast, open source NoSQL search platform
from the Apache Lucene project. Its major features include powerful
full-text search, hit highlighting, faceted search, dynamic clustering,
database integration, rich document handling, and geospatial search.
Solr is highly scalable, providing fault tolerant distributed search and
indexing, and powers the search and navigation features of many of the
world's largest internet sites.

Solr 7.7.3 is available for immediate download at:
 <https://lucene.apache.org/solr/downloads.html>

Solr 7.7.3 Release Highlights:

SOLR-13779: Use the safe fork of simple-xml for clustering contrib
* SOLR-13718: SPLITSHARD (async) with failures in underlying sub-operations
can result in data loss
* SOLR-12291: prematurely reporting not yet finished async Collections API
call as completed when collection's replicas are collocated at least at one
node
* SOLR-13828: Improve ExecutePlanAction error handling
* SOLR-13472: Forwarded requests should skip authorization on receiving
nodes
* SOLR-13793: HttpSolrCall now maintains internal request count
(_forwardedCount) for remote queries and limits them to
 the number of replicas. This avoids making too many cascading calls to
remote servers, which, if not restricted, can
 bring down nodes containing the said collection
* SOLR-13971: Velocity response writer's resource loading now possible only
through startup parameters. Also, removed velocity
 response writer from _default configset
* SOLR-14025: VelocityResponseWriter has been hardened - only trusted
configsets can render configset provided
 templates and rendering templates from request parameters has been
removed.
* SOLR-13158: DataImportHandler: Added enable.dih.dataConfigParam system
property to toggle whether the dataConfig param
 is permitted
* SOLR-14259: Fix javabin performance regression fixes

Please read CHANGES.txt for a full list of and changes:
 <https://lucene.apache.org/solr/7_7_3/changes/Changes.html>

Solr 7.7.3 also includes bugfixes in the corresponding Apache Lucene
release:
 <https://lucene.apache.org/core/7_7_3/changes/Changes.html>

Note: The Apache Software Foundation uses an extensive mirroring network
for distributing releases. It is possible that the mirror you are using
may not have replicated the release yet. If that is the case, please try
another mirror. This also applies to Maven access.

- --Noble Paul
-BEGIN PGP SIGNATURE-
Version: FlowCrypt 7.7.7 Gmail Encryption
Comment: Seamlessly send and receive encrypted email

wsFcBAEBCgAGBQJep8/nAAoJEMOP9ew/z9s+lygP/2jvJgW/+VMYwTNMur5x
KX7P1PC7oo7Lm2+F/vM81MOVms1TN7x1W/NQfGWCbLXG1G4AYmUrs+1Z6odo
CW57pyILoLMCWMfie4f5bccBb8b/lEK2jtQiOgQCD2xz7RWNHy749wxOAIN3
r5vdjsNDEULQdNXl3zpHm7A268djS90DBWuxKmDPZaSVpvBhhOzIuYIUUWuK
zrqYYAh4SVUtNDLPIrIk8075d2b4pxedvyG5TYFjLJDAX+WWRopTzdaa93zl
kyZnA/C5GQRkD1eBiU6V0JOi8LF1o2BZpdLx/lgsVUsEAqYCHUdlqd926qwS
Jwlkzl1LBc1T7sC49hXtOC7k9n/OiTZuZhkMahlT3UIECcUslFi7cKrnhZdQ
zng5UG9W/HgmzJPpK7SgJ6uAcRevwc/j9qD7helBCKF29XDCdP8jqBM9Gxbz
ma1Amgae7C4RN8I1yTLbyy+1E4fGLt+lQ5t+rQcBFb8lV9jRv8cAiulVufAF
Cb4kHACApuexfvDlFhCCHKV4n6xTHJSLMA9XvNXuilvO3ErDvFz3M+Of+SVM
rmDnvxlzV6z5tlrooNQ3Fvb5eoewa2TfvYMCG0VRQN4wzg0hfU69aXer5wt6
8ZUgEV+Omz8ZVK4C80a1CMyXPacK8yoO0qpUXmMqo8721uIbCrY5trVawMcB
Ounp
=R60o
-END PGP SIGNATURE-
-BEGIN PGP PUBLIC KEY BLOCK-
Version: FlowCrypt 7.7.7 Gmail Encryption
Comment: Seamlessly send and receive encrypted email

xsFNBFXEng4BEACtCX8PXTX+ZVmGbaxPDZSNjQxnJcnMvEaN/UoDeGVOogqT
mZn+KsGemG5vfgojeC3XlTL5OtqknygdGRGZ/A/BE0q15yk5j2YEezfhoJfO
XJSGwF/cPcOyFhhMgm/MjolzmHoUADo863u9xtYiOH0Rsgaa8BGtmfM/Z55M
6TkyGhzUPpKQdbre8+aAfMWRj42WOLVRen/2Fx5w9+ItdtZqFWgqV7k/J6N3
Ys1LgYf5p8kDniB2M98V+ZCwxivhTRbUoUKkNRssGImFb08UBbTyw4bRGRiM
Z9lSZlfIaLh3HHYQBtlMqPrjlAafuKS7K3EC+eF4j31YrwDjziLxi704OaIB
NUWIhdMa2FC7i0knUMllp8bjZ4TDyvbSFqOy62uM/wYm2QqrcEui22EtskbZ
GN/pyBns+hHj5LOL9mIrFQ7MhFcHYQ/nis++Kol9dGF0KftWnsGCFeLeUm9D
4x/Bxr2e6YYTPeoWOpQfejBKlAzaX7I3M3ERpn2xHFffa//MxkW1Jc/3snCS
icViQB7JKbLI6GF4z5Hn5okhib9yOQqLMcoIYVqddyolZjL+n2bdF0UCD5N8
caTMkMXIWd2Gq/U4rdioXJ5GQJ8FTsrcACYSmDOsFDKN6p6KnP0qkJbgXnz3
6fXPPYe5xKsMipp7zI8FJ3oV6DDqhE5ginh/YQARAQABzTBOb2JsZSBQYXVs
IChDT0RFIFNJR05JTkcgS0VZKSA8bm9ibGVAYXBhY2hlLm9yZz7CwXcEEwEK
ACEFAlXEng4CGwMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AACgkQw4/17D/P
2z4GFA//QLVVe6Jc1YjsF+xtWov3iuycENyyPNVlvoTqFPMChOTeYg1pLL4D
SGF3xYRvyzL2fl9dn+pBZSEdxrF1+9MqtbW/FCVPHCdIIJZRhAaBjHRIJES0
Hif8nXQzHzuy9BwDFpyFZMz+Y+TH42W3ATlg77HaFhjNeAaeRbknE8JYB5p9
vGZSkOFJwS4rI+QcIrpvO97WyGILWIDfZr/LFCx8tf4/umSS0TVu/12cIoFN
kB1gg3DqI8bQaF3DG7m3rB8Lg2CxQXPl44aYESdhH38ll2ntDJrswWzbgENy
koOza8xOIl1+KbJyZy2aL0XJhPawuUGfeHHK4bljGdRSER

Re: [ANNOUNCE] Apache Solr 8.3.1 released

[Noble Paul]

Thanks ishan

On Wed, Dec 4, 2019, 3:32 PM Ishan Chattopadhyaya 
wrote:

> ## 3 December 2019, Apache Solr™ 8.3.1 available
>
> The Lucene PMC is pleased to announce the release of Apache Solr 8.3.1.
>
> Solr is the popular, blazing fast, open source NoSQL search platform
> from the Apache Lucene project. Its major features include powerful
> full-text search, hit highlighting, faceted search, dynamic
> clustering, database integration, rich document handling, and
> geospatial search. Solr is highly scalable, providing fault tolerant
> distributed search and indexing, and powers the search and navigation
> features of many of the world's largest internet sites.
>
> Solr 8.3.1 is available for immediate download at:
>
>   
>
> ### Solr 8.3.1 Release Highlights:
>
>   * JavaBinCodec has concurrent modification of CharArr resulting in
> corrupt internode updates
>   * findRequestType in AuditEvent is more robust
>   * CoreContainer.auditloggerPlugin is volatile now
>   * Velocity response writer's resource loading now possible only
> through startup parameters
>
>
> Please read CHANGES.txt for a full list of changes:
>
>   
>
> Solr 8.3.1 also includes  and bugfixes in the corresponding Apache
> Lucene release:
>
>   
>
> Note: The Apache Software Foundation uses an extensive mirroring network
> for
> distributing releases. It is possible that the mirror you are using may
> not have
> replicated the release yet. If that is the case, please try another mirror.
> This also applies to Maven access.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Convert javabin to json

[Noble Paul]

obj = new JavabinCodec().unmarshal();
Utils#writeJson()

On Thu, Nov 28, 2019 at 11:54 AM Wei  wrote:
>
> Hi,
>
> Is there a reliable way to convert solr's javabin response to json format?
> We use solrj client with wt=javabin, but want to convert the received
> javabin response to json for passing to client.  We don't want to use
> wt=json as javabin is more efficient.  We tried the noggit jsonutil
>
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/noggit/JSONUtil.java
>
> but seems it is not able to convert parts of the query response such as
> facet.  Are there any other options available?
>
> Thanks,
> Wei



-- 
-----
Noble Paul

Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?

[Noble Paul]

Thanks Colvin.
Can you share the details in the ticket?

I plan to debug this today.

It's unlikely to be a synchronization issue because
serialization/deserialization usually happens in single thread.

On Sun, Nov 24, 2019, 4:09 AM Colvin Cowie 
wrote:

> https://issues.apache.org/jira/browse/SOLR-13963
>
> I'll see about modifying the test I have to fit in with the existing tests,
> and if there's a better option then open to whatever
>
> On Sat, 23 Nov 2019 at 16:43, Colvin Cowie 
> wrote:
>
> > I've found the problem, JavaBinCodec has a CharArr,* arr*, which is
> > modified in two different locations, but only one of which is protected
> > with a synchronized block
> >
> > getStringProvider(), which is used when you call getValue() rather than
> > getRawValue() on the string based SolrInputFields, synchronizes:
> >
> >
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L966
> > but  _readStr() doesn't:
> >
> >
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L930
> >
> > Adding a synchronized block into _readStr() fixes the problem. At least
> as
> > far as my test goes.
> >
> > I'll raise a JIRA issue and can provide a patch with the synchronized
> > block, but not sure what test(s) should be updated / added to cover this?
> >
> >
> >
> > On Thu, 21 Nov 2019 at 18:23, Colvin Cowie 
> > wrote:
> >
> >> *> the difference is because the _default config has the dynamic schema
> >> building in it, which I assume is pushing it down a different code
> path. *
> >>
> >> Also to add to that, I assumed initially that this just meant that it
> was
> >> working because the corrupted field names would just cause it to create
> a
> >> field with the dodgy name (since that's the idea for the dynamic
> schema),
> >> but checking the documents on retrieval showed they all had the right
> field
> >> names...
> >> So I assume it's a result of going into a different branch of code
> >> instead.
> >>
> >>
> >> On an unrelated matter, I saw this in the logs when running with
> embedded
> >> zookeeper... I don't think I've seen it mentioned anywhere else, so I
> will
> >> raise an issue for it
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> *2019-11-21 17:25:14.292 INFO  (main) [   ] o.a.s.c.SolrZkServer
> STARTING
> >> EMBEDDED STANDALONE ZOOKEEPER SERVER at port 99832019-11-21 17:25:14.792
> >> INFO  (main) [   ] o.a.s.c.ZkContainer Zookeeper
> >> client=localhost:99832019-11-21 17:25:18.833 WARN  (Thread-13) [   ]
> >> o.a.z.s.a.AdminServerFactory Unable to load jetty, not starting
> >> JettyAdminServer => java.lang.NoClassDefFoundError:
> >> org/eclipse/jetty/server/Connector at java.lang.Class.forName0(Native
> >> Method)java.lang.NoClassDefFoundError:
> org/eclipse/jetty/server/Connector
> >> at java.lang.Class.forName0(Native Method) ~[?:1.8.0_191] at
> >> java.lang.Class.forName(Class.java:264) ~[?:1.8.0_191] at
> >>
> org.apache.zookeeper.server.admin.AdminServerFactory.createAdminServer(AdminServerFactory.java:43)
> >> ~[?:?] at
> >>
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:136)
> >> ~[?:?] at
> org.apache.solr.cloud.SolrZkServer$1.run(SolrZkServer.java:121)
> >> ~[?:?]Caused by: java.lang.ClassNotFoundException:
> >> org.eclipse.jetty.server.Connector at
> >>
> org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:577)
> >> ~[jetty-webapp-9.4.19.v20190610.jar:9.4.19.v20190610] at
> >> java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_191]
> ... 5
> >> more2019-11-21 17:25:19.365 INFO  (main) [   ]
> o.a.s.c.c.ConnectionManager
> >> Waiting for client to connect to ZooKeeper2019-11-21 17:25:19.396 INFO
> >>  (zkConnectionManagerCallback-7-thread-1) [   ]
> o.a.s.c.c.ConnectionManager
> >> zkClient has connected2019-11-21 17:25:19.396 INFO  (main) [   ]
> >> o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper*
> >>
> >> On Thu, 21 Nov 2019 at 17:30, Colvin Cowie 
> >> wrote:
> >>
> >>> I've been a bit snowed under, but I've found the difference is because
> >>> the _default config has the dynamic schema building in it, which I
> assume
> >>> is pushing it down a different code path.
> >>>
> >>>>>> default="${update.autoCreateFields:true}"
> >>>
> >>>
> processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
> >>>
> >>> I'm using the vanilla Solr 8.3.0 binary8.3.0
> >>> 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:15:22
> with
> >>> Eclipse OpenJ9 Eclipse OpenJ9 VM 1.8.0_232 openj9-0.17.0
> >>> and I've checked with Oracle Corporation Java HotSpot(TM) 64-Bit Server
> >>> VM 1.8.0_191 25.191-b12 as well
> >>>
> >>> I've put a testcase and configsets in Google Drive:
> >>> https://drive.google.com/open?id=1ibKNWvowT8cXTwSa3bcTwKYLSRNur86U
> >>> The configsets are a copy 

Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?

[Noble Paul]

Sure, looking forward to that
thnaks

On Thu, Nov 21, 2019 at 8:06 AM Colvin Cowie  wrote:
>
> Hi, I'll share it when I'm back at work tomorrow.
>
> I've found that the issue appears to not be reproducible using the
> solrconfig.xml in the _default example, so it must be something in ours (or
> something missing from it) that is making the problem appear, in
> combination with the code change.
>
> Thanks
>
>
> On Wednesday, 20 November 2019, Noble Paul  wrote:
>
> > Can you share the test please
> >
> > On Thu, Nov 21, 2019 at 7:02 AM Noble Paul  wrote:
> > >
> > > Thanks Colvin, I'll take a look
> > >
> > > On Thu, Nov 21, 2019 at 4:24 AM Colvin Cowie 
> > wrote:
> > > >
> > > > I've identified the change which has caused the problem to
> > materialize, but
> > > > it shouldn't itself cause a problem.
> > > >
> > > > https://github.com/apache/lucene-solr/commit/
> > e45e8127d5c17af4e4b87a0a4eaf0afaf4f9ff4b#diff-
> > 7f7f485122d8257bd5d3210c092b967fR52
> > > > for https://issues.apache.org/jira/browse/SOLR-13682
> > > >
> > > > In writeMap, the new BiConsumer unwraps the SolrInputField using
> > getValue
> > > > rather than getRawValue (which the JavaBinCodec calls):
> > > >
> > > >
> > > > *  if (o instanceof SolrInputField) {o = ((SolrInputField)
> > > > o).getValue();  }*
> > > > As a result the JavaBinCodec will now be hitting different writer
> > methods
> > > > based on the value retrieved from the SolrInputField, rather than just
> > > > writing the org.apache.solr.common.util.JavaBinCodec.writeKnownType(
> > Object)
> > > >
> > > >
> > > > *if (val instanceof SolrInputField) {  return
> > > > writeKnownType(((SolrInputField) val).getRawValue());}*
> > > > https://github.com/apache/lucene-solr/blob/branch_8_3/
> > solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L362
> > > >
> > > > SolrInputField getValue uses
> > > > org.apache.solr.common.util.ByteArrayUtf8CharSequence.
> > convertCharSeq(Object)
> > > > while getRawValue just returns whatever value the SolrInputField has,
> > so
> > > > the EntryWriter in the JavaBinCodec hits different paths from the ones
> > > > which must non-deterministically produce garbage data when getValue()
> > is
> > > > used.
> > > >
> > > > Changing *getValue()* to *getRawValue()* in the SolrInputDocument's
> > > > *writeMap()* appears to "fix" the problem. (With getValue() the test I
> > have
> > > > reliably fails within 50 iterations of indexing 2500 documents, with
> > > > getRawValue() it succeeds for the 500 iterations I'm running it for)
> > > >
> > > > I'll see about providing a test that can be shared that demonstrates
> > the
> > > > problem, and see if we can find what is going wrong in the codec...
> > > >
> > > >
> > > > On Tue, 19 Nov 2019 at 13:48, Colvin Cowie  > >
> > > > wrote:
> > > >
> > > > > Hello
> > > > >
> > > > > Apologies for the lack of actual detail in this, we're still digging
> > into
> > > > > it ourselves. I will provide more detail, and maybe some logs, once
> > I have
> > > > > a better idea of what is actually happening.
> > > > > But I thought I might as well ask if anyone knows of changes that
> > were
> > > > > made in the Solr 8.3 release that are likely to have caused an issue
> > like
> > > > > this?
> > > > >
> > > > > We were on Solr 8.1.1 for several months and moved to 8.2.0 for
> > about 2
> > > > > weeks before moving to 8.3.0 last week.
> > > > > We didn't see this issue at all on the previous releases. Since
> > moving to
> > > > > 8.3 we have had a consistent (but non-deterministic) set of failing
> > tests,
> > > > > on Windows and Linux.
> > > > >
> > > > > The issue we are seeing as that during updates, the data we have
> > sent is
> > > > > *sometimes* corrupted, as though a buffer has been used incorrectly.
> > For
> > > > > example if the well formed data went was
> > > > > *'fieldName':"this is a long string"*
> > > > > The error we see from

Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?

[Noble Paul]

Can you share the test please

On Thu, Nov 21, 2019 at 7:02 AM Noble Paul  wrote:
>
> Thanks Colvin, I'll take a look
>
> On Thu, Nov 21, 2019 at 4:24 AM Colvin Cowie  
> wrote:
> >
> > I've identified the change which has caused the problem to materialize, but
> > it shouldn't itself cause a problem.
> >
> > https://github.com/apache/lucene-solr/commit/e45e8127d5c17af4e4b87a0a4eaf0afaf4f9ff4b#diff-7f7f485122d8257bd5d3210c092b967fR52
> > for https://issues.apache.org/jira/browse/SOLR-13682
> >
> > In writeMap, the new BiConsumer unwraps the SolrInputField using getValue
> > rather than getRawValue (which the JavaBinCodec calls):
> >
> >
> > *  if (o instanceof SolrInputField) {o = ((SolrInputField)
> > o).getValue();  }*
> > As a result the JavaBinCodec will now be hitting different writer methods
> > based on the value retrieved from the SolrInputField, rather than just
> > writing the org.apache.solr.common.util.JavaBinCodec.writeKnownType(Object)
> >
> >
> > *if (val instanceof SolrInputField) {  return
> > writeKnownType(((SolrInputField) val).getRawValue());}*
> > https://github.com/apache/lucene-solr/blob/branch_8_3/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java#L362
> >
> > SolrInputField getValue uses
> > org.apache.solr.common.util.ByteArrayUtf8CharSequence.convertCharSeq(Object)
> > while getRawValue just returns whatever value the SolrInputField has, so
> > the EntryWriter in the JavaBinCodec hits different paths from the ones
> > which must non-deterministically produce garbage data when getValue() is
> > used.
> >
> > Changing *getValue()* to *getRawValue()* in the SolrInputDocument's
> > *writeMap()* appears to "fix" the problem. (With getValue() the test I have
> > reliably fails within 50 iterations of indexing 2500 documents, with
> > getRawValue() it succeeds for the 500 iterations I'm running it for)
> >
> > I'll see about providing a test that can be shared that demonstrates the
> > problem, and see if we can find what is going wrong in the codec...
> >
> >
> > On Tue, 19 Nov 2019 at 13:48, Colvin Cowie 
> > wrote:
> >
> > > Hello
> > >
> > > Apologies for the lack of actual detail in this, we're still digging into
> > > it ourselves. I will provide more detail, and maybe some logs, once I have
> > > a better idea of what is actually happening.
> > > But I thought I might as well ask if anyone knows of changes that were
> > > made in the Solr 8.3 release that are likely to have caused an issue like
> > > this?
> > >
> > > We were on Solr 8.1.1 for several months and moved to 8.2.0 for about 2
> > > weeks before moving to 8.3.0 last week.
> > > We didn't see this issue at all on the previous releases. Since moving to
> > > 8.3 we have had a consistent (but non-deterministic) set of failing tests,
> > > on Windows and Linux.
> > >
> > > The issue we are seeing as that during updates, the data we have sent is
> > > *sometimes* corrupted, as though a buffer has been used incorrectly. For
> > > example if the well formed data went was
> > > *'fieldName':"this is a long string"*
> > > The error we see from Solr might be that
> > > unknown field * 'fieldNamis a long string" *
> > >
> > > And variations of that kind of behaviour, were part of the data is missing
> > > or corrupted. The data we are indexing does include fields which store
> > > (escaped) serialized JSON strings - if that might have any bearing - but
> > > the error isn't always on those fields.
> > > For example, given a valid document that looks like this (I've replaced
> > > the values by hand, so if the json is messed up here, that's not 
> > > relevant:)
> > > when returned with the json response writer:
> > >
> > >
> > >
> > >
> > > *{"id": "abcd","testField": "blah","jsonField":
> > > "{\"thing\":{\"abcd\":\"value\",\"xyz\":[\"abc\",\"def\",\"ghi\"],\"nnn\":\"xyz\"},\"stuff\":[{\"qqq\":\"rrr\"}],\"ttt\":0,\"mmm\":\"Some
> > > string\",\"someBool\":true}"}*
> > > We've had errors during indexing like:
> > > *unknown field
> > > 'testField:"value","xyz":["abc","def","ghi&q

Re: Possible data corruption in JavaBinCodec in Solr 8.3 during distributed update?

[Noble Paul]

 the other shards... But that's not been
> > totally verified.
> >
> > We've also only encountered the problem on one of the collections we build
> > (the data within each collection is generally the same though. The ids are
> > slightly different - but still strings. The main difference is that this
> > problematic index is built using an Iterator to *solrj
> > org.apache.solr.client.solrj.SolrClient.add(String,
> > Iterator)* - the *SolrInputDocument*s are not being
> > reused in the client, I checked that -, while the other index is built by
> > streaming CSVs to Solr.)
> >
> >
> > We will look into it further, but if anyone has any ideas of what might
> > have changed in 8.3 from 8.1 / 8.2 that could cause this, that would be
> > helpful.
> >
> > Cheers
> > Colvin
> >
> >



-- 
-
Noble Paul

Re: Performance Issue since Solr 7.7 with wt=javabin

[Noble Paul]

How are you consuming the output? Are you using solrj?

On Tue, Jun 18, 2019, 1:27 AM Andy Reek  wrote:

> Hi Solr team,
>
>
> we are using Solr in version 7.1 as search engine in our online shop (SAP
> Hybris). And as a task I needed to migrate to the most recent Solr in
> version 7 (7.7). Doing this I faced extreme performance issues. After
> debugging and testing different setups I found out, that they were caused
> by the parameter wt=javabin. These issues begin to raise since version 7.7,
> in 7.6 it is still working as fast as in 7.1.
>
>
> Just an example: Doing a simple query for *.* and wt=javabin in 7.6: 0.2
> seconds and in 7.7: 34 seconds!
>
>
> The configuration of the schema.xml and solrconfig.xml are equal in both
> versions. Version 8.1 has the same effect as 7.7. Using something other
> than wt=javabin (e.g. wt=xml) will work fast in every version - which is
> our current workaround.
>
>
>
> To reproduce this issue I have attached my used configsets folder plus
> some test data. This all can be tested with docker and wget:
>
>
> Solr 7.6:
>
> docker run -d --name solr7.6 -p 8983:8983 --rm -v
> $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
> solr:7.6-slim solr-create -c mycore -d
> /opt/solr/server/solr/configsets/myconfig
> docker cp $PWD/data.json solr7.6:/opt/solr/data.json
> docker exec -it --user solr solr7.6 bin/post -c mycore data.json
> wget "http://localhost:8983/solr/mycore/select?q=*:*=javabin=;
> (0.2s)
>
> Solr 7.7:
> docker run -d --name solr7.7 -p 18983:8983 --rm -v
> $PWD/configsets/default:/opt/solr/server/solr/configsets/myconfig:ro
> solr:7.7-slim solr-create -c mycore -d
> /opt/solr/server/solr/configsets/myconfig
> docker cp $PWD/data.json solr7.7:/opt/solr/data.json
> docker exec -it --user solr solr7.7 bin/post -c mycore data.json
> (34s)
>
> For me it seems like a bug. But if not, then please let me know what I did
> wrong ;-)
>
>
>
> Best Regards,
>
>
>
> *Andy Reek*
>
> Principal Software Developer
>
>
>
> *diva-e* Jena
>
> Mälzerstraße 3, 07745 Jena, Deutschland
>
> T:   +49 (3641) 3678 (223)
>
> F:   +49 (3641) 3678 101
>
> *andy.r...@diva-e.com *
>
>
>
> *www.diva-e.com * follow us: facebook
> , twitter
> , LinkedIn
> *,*
>  *Xing *
>
> 
>
>
>
> *diva-e* AGETO GmbH
>
> Handelsregister: HRB 210399 Amtsgericht Jena
>
> Geschäftsführung: Sascha Sauer, Sirko Schneppe, Axel Jahn
>
>

Re: Solr 7.7.2 Autoscaling policy - Poor performance

[Noble Paul]

It can't be considered a bug , it's just that there are too many
calculations involved as there are a very large no:of nodes. Any further
spped up would require a change in the way it's calculated

On Thu, Sep 5, 2019, 1:30 AM Andrew Kettmann 
wrote:

>
> > there are known perf issues in computing very large clusters
>
> Is there any documentation/open tickets on this that you have handy? If
> that is the case, then we might be back to looking at separate Znodes.
> Right now if we provide a nodeset on collection creation, it is creating
> them quickly. I don't want to make many changes as this is part of our
> production at this time.
>
>
>
>
>
> From: Noble Paul 
>
> Sent: Wednesday, September 4, 2019 12:14 AM
>
> To: solr-user@lucene.apache.org 
>
> Subject: Re: Solr 7.7.2 Autoscaling policy - Poor performance
>
>
>
>
> there are known perf issues in computing very large clusters
>
>
>
> give it a try with the following rules
>
>
>
> "FOO_CUSTOMER":[
>
>   {
>
> "replica":"0",
>
> "sysprop.HELM_CHART":"!FOO_CUSTOMER",
>
> "strict":"true"},
>
>   {
>
> "replica":"<2",
>
> "node":"#ANY",
>
> "strict":"false"}]
>
>
>
> On Wed, Sep 4, 2019 at 1:49 AM Andrew Kettmann
>
>  wrote:
>
> >
>
> > Currently our 7.7.2 cluster has ~600 hosts and each collection is using
> an autoscaling policy based on system property. Our goal is a single core
> per host (container, running on K8S). However as we have rolled more
> containers/collections into the cluster
>  any creation/move actions are taking a huge amount of time. In fact we
> generally hit the 180 second timeout if we don't schedule it as async.
> Though the action gets completed anyway. Looking at the code, it looks like
> for each core it is considering the entire
>  cluster.
>
> >
>
> > Right now our autoscaling policies look like this, note we are feeding a
> sysprop on startup for each collection to map to specific containers:
>
> >
>
> > "FOO_CUSTOMER":[
>
> >   {
>
> > "replica":"#ALL",
>
> > "sysprop.HELM_CHART":"FOO_CUSTOMER",
>
> > "strict":"true"},
>
> >   {
>
> > "replica":"<2",
>
> > "node":"#ANY",
>
> > "strict":"false"}]
>
> >
>
> > Does name based filtering allow wildcards ? Also would that likely fix
> the issue of the time it takes for Solr to decide where cores can go? Or
> any other suggestions for making this more efficient on the Solr overseer?
> We do have dedicated overseer nodes,
>  but the leader maxes out CPU for awhile while it is thinking about this.
>
> >
>
> > We are considering putting each collection into its own zookeeper
> znode/chroot if we can't support this many nodes per overseer. I would like
> to avoid that if possible, but also creating a collection in sub 10 minutes
> would be neat too.
>
> >
>
> > I appreciate any input/suggestions anyone has!
>
> >
>
> > [https://storage.googleapis.com/e24-email-images/e24logonotag.png]<
> https://www.evolve24.com> Andrew Kettmann
>
> > DevOps Engineer
>
> > P: 1.314.596.2836
>
> > [LinkedIn]<https://linkedin.com/company/evolve24> [Twitter] <
> https://twitter.com/evolve24>  [Instagram] <
> https://www.instagram.com/evolve_24>
>
> >
>
> > evolve24 Confidential & Proprietary Statement: This email and any
> attachments are confidential and may contain information that is
> privileged, confidential or exempt from disclosure under applicable law. It
> is intended for the use of the recipients. If you
>  are not the intended recipient, or believe that you have received this
> communication in error, please do not read, print, copy, retransmit,
> disseminate, or otherwise use the information. Please delete this email and
> attachments, without reading, printing,
>  copying, forwarding or saving them, and notify the Sender immediately by
> reply email. No confidentiality or privilege is waived or lost by any
> transmission in error.
>
>
>
>
>
>
>
> --
>
> -
>
> Noble Paul
>
>

Re: Solr 7.7.2 Autoscaling policy - Poor performance

[Noble Paul]

there are known perf issues in computing very large clusters

give it a try with the following rules

"FOO_CUSTOMER":[
  {
"replica":"0",
"sysprop.HELM_CHART":"!FOO_CUSTOMER",
"strict":"true"},
  {
"replica":"<2",
"node":"#ANY",
"strict":"false"}]

On Wed, Sep 4, 2019 at 1:49 AM Andrew Kettmann
 wrote:
>
> Currently our 7.7.2 cluster has ~600 hosts and each collection is using an 
> autoscaling policy based on system property. Our goal is a single core per 
> host (container, running on K8S). However as we have rolled more 
> containers/collections into the cluster any creation/move actions are taking 
> a huge amount of time. In fact we generally hit the 180 second timeout if we 
> don't schedule it as async. Though the action gets completed anyway. Looking 
> at the code, it looks like for each core it is considering the entire cluster.
>
> Right now our autoscaling policies look like this, note we are feeding a 
> sysprop on startup for each collection to map to specific containers:
>
> "FOO_CUSTOMER":[
>   {
> "replica":"#ALL",
> "sysprop.HELM_CHART":"FOO_CUSTOMER",
> "strict":"true"},
>   {
> "replica":"<2",
> "node":"#ANY",
> "strict":"false"}]
>
> Does name based filtering allow wildcards ? Also would that likely fix the 
> issue of the time it takes for Solr to decide where cores can go? Or any 
> other suggestions for making this more efficient on the Solr overseer? We do 
> have dedicated overseer nodes, but the leader maxes out CPU for awhile while 
> it is thinking about this.
>
> We are considering putting each collection into its own zookeeper 
> znode/chroot if we can't support this many nodes per overseer. I would like 
> to avoid that if possible, but also creating a collection in sub 10 minutes 
> would be neat too.
>
> I appreciate any input/suggestions anyone has!
>
> [https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://www.evolve24.com>
>  Andrew Kettmann
> DevOps Engineer
> P: 1.314.596.2836
> [LinkedIn]<https://linkedin.com/company/evolve24> [Twitter] 
> <https://twitter.com/evolve24>  [Instagram] 
> <https://www.instagram.com/evolve_24>
>
> evolve24 Confidential & Proprietary Statement: This email and any attachments 
> are confidential and may contain information that is privileged, confidential 
> or exempt from disclosure under applicable law. It is intended for the use of 
> the recipients. If you are not the intended recipient, or believe that you 
> have received this communication in error, please do not read, print, copy, 
> retransmit, disseminate, or otherwise use the information. Please delete this 
> email and attachments, without reading, printing, copying, forwarding or 
> saving them, and notify the Sender immediately by reply email. No 
> confidentiality or privilege is waived or lost by any transmission in error.



-- 
-
Noble Paul

Re: Upload/use a plugin JAR in ZooKeeper

[Noble Paul]

are you interested in this

SOLR-13534: Dynamic loading to support loading jars from a URL

https://issues.apache.org/jira/browse/SOLR-13534

On Fri, Jul 19, 2019 at 12:18 PM Richard Walker
 wrote:
>
> On 19 Jul 2019, at 12:02 pm, Chee Yee Lim  wrote:
> > Not sure if this is the recommended way, but I managed to use plugin JARs
> > with Solr Cloud.
> >
> > Either include the absolute path to JAR in solrconfig.xml, or put the JAR
> > in a "lib" folder relative to your instanceDir. See the following text from
> > solrconfig.xml.
>
> As I already noted in my original message of 16 July:
>
> > I've been able to get this to work the "simple" way,
> > by putting the JAR in the file system, and specifying
> > basic
> >
> >  
> >  
> >
> > values in solrconfig.xml. No problem doing it this way.
>
> ... and that this is precisely what I do _not_ want to do,
> unless I have to.
>
> I want to use a JAR file uploaded to the collection's znode,
> as the user guide strongly suggests is possible.
> (And also again, no, I don't want to configure/use the Blob Store.)
>


-- 
-
Noble Paul

Re: Config API: delete-requesthandler

[Noble Paul]

yes, this is the way to do it

On Thu, May 26, 2016 at 6:54 AM Jan Høydahl  wrote:
>
> Hi
>
> Have you tried adding all your commands to the same file?
>
> {
>   "add-requesthandler":{"name":"/foo","class":"solr.SearchHandler"},
>   "add-requesthandler":{"name":"/bar","class":"solr.SearchHandler"},
>   "add-requesthandler":{"name":"/baz","class":"solr.SearchHandler"}
> }
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 20. mai 2016 kl. 16.19 skrev Steven White :
> >
> > Hi folks,
> >
> > The code I'm maintaining, uses Solr config API per
> > https://cwiki.apache.org/confluence/display/solr/Config+API to manage
> > request handlers.  My environment has many request handlers, up to 100 in
> > few extreme cases.  When that's the case, it means I will issue 100
> > "delete-requesthandler" followed with 100 "add-requesthandler" requests.
> > This end-to-end operation can be time consuming specially if there is data
> > in the index (searchers are being reset leading to cache and stuff being
> > reloaded).  But even if there isn't data in the index, it will take about
> > 10 min. in my test environment.
> >
> > My question is, how can I optimize this?  Is there any kind of "bulk" or
> > "transaction-base" way of doing schema configuration?  I.e.: send all my
> > deletes, followed with all my adds, and then issue a commit for the change
> > to take effect.  If I switch to SolrJ (does it offer schema management?)
> > will that help?
> >
> > Thanks in advanced.
> >
> > Steve
>


-- 
-
Noble Paul

Re: [ANNOUNCE] Apache Solr 8.0.0 released

[Noble Paul]

Thanks Jim

On Fri, Mar 15, 2019 at 1:39 AM Toke Eskildsen  wrote:
>
> On Thu, 2019-03-14 at 13:16 +0100, jim ferenczi wrote:
> > http://lucene.apache.org/solr/8_0_0/changes/Changes.html
>
> Thank you for the hard work of rolling the release!
> Looking forward to upgrading.
>
> - Toke Eskildsen, Royal Danish Library
>
>


-- 
-----
Noble Paul

Re: Basic Auth Permission

[Noble Paul]

You can't restrict access to static files.

You can only restrict access to Solr content.

However you can use the "blockUnknown" property in your security.json
to restrict access to all files

https://lucene.apache.org/solr/guide/7_5/basic-authentication-plugin.html
--Noble
On Sat, Jun 9, 2018 at 2:43 AM Antony A  wrote:
>
> Hello,
>
> I am trying to get the path/params restricted to users of individual
> collection through Solr UI.
>
> Here is the permission that I have for an user.
>
> {"collection": "collection_name", "path": "/admin/file", "role": ["
> collection_user"]}
>
> I am still not able to restrict another user from accessing other
> collection files like solrconfig, solr-data-config etc.
>
> If it possible to define permission at collection-level to this path?
>
> Thanks,
> Antony



-- 
-
Noble Paul

Re: Adding authentication

[Noble Paul]

yStrategy Error while trying to recover.
> core=formdoc_shard2_replica1:java.util.concurrent.ExecutionException:
> org.apache.solr.common.SolrException: javax.crypto.IllegalBlockSize
> Exception: Invalid input.
> at java.util.concurrent.FutureTask.report(FutureTask.java:133)
> at java.util.concurrent.FutureTask.get(FutureTask.java:203)
> at
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:596)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:353)
> at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:224)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
> at java.util.concurrent.FutureTask.run(FutureTask.java:277)
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:231)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.lang.Thread.run(Thread.java:785)
> Caused by: org.apache.solr.common.SolrException:
> javax.crypto.IllegalBlockSizeException: Invalid input.
> at
> org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys$RSAKeyPair.java:67)
> at
> org.apache.solr.security.PKIAuthenticationPlugin.setHeader(PKIAuthenticationPlugin.java:287)
> at
> org.apache.solr.security.PKIAuthenticationPlugin$HttpHeaderClientConfigurer.process(PKIAuthenticationPlugin.java:257)
> at
> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
> at
> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:166)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:485)
> at
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:481)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:284)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:280)
> ... 5 more
> Caused by: javax.crypto.IllegalBlockSizeException: Invalid input.
> at com.rsa.cryptoj.o.fy.engineDoFinal(Unknown Source)
> at javax.crypto.Cipher.doFinal(Unknown Source)
> at
> org.apache.solr.util.CryptoKeys$RSAKeyPair.encrypt(CryptoKeys$RSAKeyPair.java:63)
> ... 17 more
>
> I appreciate any suggestions you can offer.
>
> Thanks,
> Adam



-- 
-
Noble Paul

Re: Rule-based replication or sharing

[Noble Paul]

Yes, it uses a the autoscaling policies to achieve the same. Please refer
to the documentation here
https://lucene.apache.org/solr/guide/7_5/solrcloud-autoscaling-policy-preferences.html

On Thu, Sep 27, 2018, 02:11 Chuck Reynolds  wrote:

> Noble,
>
> Are you saying in the latest version of Solr that this would work with
> three instances of Solr running on each server?
>
> If so how?
>
> Thanks again for your help.
>
> On 9/26/18, 9:11 AM, "Noble Paul"  wrote:
>
> I'm not sure if it is pertinent to ask you to move to the latest Solr
> which has the policy based replica placement. Unfortunately, I don't
> have any other solution I can think of
>
> On Wed, Sep 26, 2018 at 11:46 PM Chuck Reynolds <
> creyno...@ancestry.com> wrote:
> >
> > Noble,
> >
> > So other than manually moving replicas of shard do you have a
> suggestion of how one might accomplish the multiple availability zone with
> multiple instances of Solr running on each server?
> >
> > Thanks
> >
> > On 9/26/18, 12:56 AM, "Noble Paul"  wrote:
> >
> > The rules suggested by Steve is correct. I tested it locally and
> I got
> > the same errors. That means a bug exists probably.
> > All the new development efforts are invested in the new policy
> feature
> > .
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_solrcloud-2Dautoscaling-2Dpolicy-2Dpreferences.html=DwIFaQ=kKqjBR9KKWaWpMhASkPbOg=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0=yXVYNcm-dqN_lucLyuQI38EZfK4f8l4828Ty53e4plM=D1vfu3bOu_hOGAU2CIKPwqBTPkYiBeK1kOUoFnQZpKA=
> >
> > The old one is going to be deprecated pretty soon. So, I'm not
> sure if
> > we should be investing our resources here
> > On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds <
> creyno...@ancestry.com> wrote:
> > >
> > > Shawn,
> > >
> > > Thanks for the info. We’ve been running this way for the past
> 4 years.
> > >
> > > We were running on very large hardware, 20 physical cores with
> 256 gigs of ram with 3 billion document and it was the only way we could
> take advantage of the hardware.
> > >
> > > Running 1 Solr instance per server never gave us the
> throughput we needed.
> > >
> > > So I somewhat disagree with your statement because our test
> proved otherwise.
> > >
> > > Thanks for the info.
> > >
> > > Sent from my iPhone
> > >
> > > > On Sep 25, 2018, at 4:19 PM, Shawn Heisey <
> apa...@elyograg.org> wrote:
> > > >
> > > >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
> > > >> Each server has three instances of Solr running on it so
> every instance on the server has to be in the same replica set.
> > > >
> > > > You should be running exactly one Solr instance per server.
> When evaluating rules for replica placement, SolrCloud will treat each
> instance as completely separate from all others, including others on the
> same machine.  It will not know that those three instances are on the same
> machine.  One Solr instance can handle MANY indexes.
> > > >
> > > > There is only ONE situation where it makes sense to run
> multiple instances per machine, and in my strong opinion, even that
> situation should not be handled with multiple instances. That situation is
> this:  When running one instance would require a REALLY large heap.
> Garbage collection pauses can become extreme in that situation, so some
> people will run multiple instances that each have a smaller heap, and
> divide their indexes between them. In my opinion, when you have enough
> index data on an instance that it requires a huge heap, instead of running
> two or more instances on one server, it's time to add more servers.
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> >
> >
> >
> > --
> > -
> > Noble Paul
> >
> >
>
>
> --
> -
> Noble Paul
>
>
>

Re: Rule-based replication or sharing

[Noble Paul]

I'm not sure if it is pertinent to ask you to move to the latest Solr
which has the policy based replica placement. Unfortunately, I don't
have any other solution I can think of

On Wed, Sep 26, 2018 at 11:46 PM Chuck Reynolds  wrote:
>
> Noble,
>
> So other than manually moving replicas of shard do you have a suggestion of 
> how one might accomplish the multiple availability zone with multiple 
> instances of Solr running on each server?
>
> Thanks
>
> On 9/26/18, 12:56 AM, "Noble Paul"  wrote:
>
> The rules suggested by Steve is correct. I tested it locally and I got
> the same errors. That means a bug exists probably.
> All the new development efforts are invested in the new policy feature
> 
> .https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F4_solrcloud-2Dautoscaling-2Dpolicy-2Dpreferences.html=DwIFaQ=kKqjBR9KKWaWpMhASkPbOg=J-2s3b-3-OTA0o6bGDhJXAQlB5Y3s4rOUxlh_78DJl0=yXVYNcm-dqN_lucLyuQI38EZfK4f8l4828Ty53e4plM=D1vfu3bOu_hOGAU2CIKPwqBTPkYiBeK1kOUoFnQZpKA=
>
> The old one is going to be deprecated pretty soon. So, I'm not sure if
> we should be investing our resources here
> On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds  
> wrote:
> >
> > Shawn,
> >
> > Thanks for the info. We’ve been running this way for the past 4 years.
> >
> > We were running on very large hardware, 20 physical cores with 256 gigs 
> of ram with 3 billion document and it was the only way we could take 
> advantage of the hardware.
> >
> > Running 1 Solr instance per server never gave us the throughput we 
> needed.
> >
> > So I somewhat disagree with your statement because our test proved 
> otherwise.
> >
> > Thanks for the info.
> >
> > Sent from my iPhone
> >
> > > On Sep 25, 2018, at 4:19 PM, Shawn Heisey  wrote:
> > >
> > >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
> > >> Each server has three instances of Solr running on it so every 
> instance on the server has to be in the same replica set.
> > >
> > > You should be running exactly one Solr instance per server.  When 
> evaluating rules for replica placement, SolrCloud will treat each instance as 
> completely separate from all others, including others on the same machine.  
> It will not know that those three instances are on the same machine.  One 
> Solr instance can handle MANY indexes.
> > >
> > > There is only ONE situation where it makes sense to run multiple 
> instances per machine, and in my strong opinion, even that situation should 
> not be handled with multiple instances. That situation is this:  When running 
> one instance would require a REALLY large heap.  Garbage collection pauses 
> can become extreme in that situation, so some people will run multiple 
> instances that each have a smaller heap, and divide their indexes between 
> them. In my opinion, when you have enough index data on an instance that it 
> requires a huge heap, instead of running two or more instances on one server, 
> it's time to add more servers.
> > >
> > > Thanks,
> > > Shawn
> > >
>
>
>
> --
> -
> Noble Paul
>
>


-- 
-
Noble Paul

Re: Rule-based replication or sharing

[Noble Paul]

The rules suggested by Steve is correct. I tested it locally and I got
the same errors. That means a bug exists probably.
All the new development efforts are invested in the new policy feature
.https://lucene.apache.org/solr/guide/7_4/solrcloud-autoscaling-policy-preferences.html

The old one is going to be deprecated pretty soon. So, I'm not sure if
we should be investing our resources here
On Wed, Sep 26, 2018 at 1:23 PM Chuck Reynolds  wrote:
>
> Shawn,
>
> Thanks for the info. We’ve been running this way for the past 4 years.
>
> We were running on very large hardware, 20 physical cores with 256 gigs of 
> ram with 3 billion document and it was the only way we could take advantage 
> of the hardware.
>
> Running 1 Solr instance per server never gave us the throughput we needed.
>
> So I somewhat disagree with your statement because our test proved otherwise.
>
> Thanks for the info.
>
> Sent from my iPhone
>
> > On Sep 25, 2018, at 4:19 PM, Shawn Heisey  wrote:
> >
> >> On 9/25/2018 9:21 AM, Chuck Reynolds wrote:
> >> Each server has three instances of Solr running on it so every instance on 
> >> the server has to be in the same replica set.
> >
> > You should be running exactly one Solr instance per server.  When 
> > evaluating rules for replica placement, SolrCloud will treat each instance 
> > as completely separate from all others, including others on the same 
> > machine.  It will not know that those three instances are on the same 
> > machine.  One Solr instance can handle MANY indexes.
> >
> > There is only ONE situation where it makes sense to run multiple instances 
> > per machine, and in my strong opinion, even that situation should not be 
> > handled with multiple instances. That situation is this:  When running one 
> > instance would require a REALLY large heap.  Garbage collection pauses can 
> > become extreme in that situation, so some people will run multiple 
> > instances that each have a smaller heap, and divide their indexes between 
> > them. In my opinion, when you have enough index data on an instance that it 
> > requires a huge heap, instead of running two or more instances on one 
> > server, it's time to add more servers.
> >
> > Thanks,
> > Shawn
> >



-- 
-
Noble Paul

Re: Solr Autoscaling multi-AZ rules

[Noble Paul]

The meaning of Replication Factor is screwed up. Replication factor is
a number. RF=3 means there are 3 replicas for each shard.

I understand that {"replica": "<7", "node":"#ANY"} may result in two
replicas of the same shard ending up on the same node. However, the
other rule should prevent this: {"replica": "<2", "shard": "#EACH",
"node": "#ANY"}
So by using both rules, that should mean "no more than six replicas on
a node, where all the replicas on that node represent distinct
shards". Right?

Yes you are right

On Fri, Feb 23, 2018 at 7:17 AM, Jeff Wartes <jwar...@whitepages.com> wrote:
>
> I managed to miss this reply earlier, but:
>
> Shard: A logical segment of a collection
> Replica: A physical core, representing a particular Shard
> Replication Factor (RF): A set of Replicas, such that a single Replica exists 
> for each Shard in a Collection.
> Availability Zone (AZ): A partitioned set of nodes such that a physical or 
> hardware failure in one AZ should not affect another AZ. AZ could mean 
> distinct racks in a data center, or distinct  data centers, but I happen to 
> specifically mean the AWS definition here: 
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions-availability-zones
>
> So an RF2 collection with 2 shards means I have four Replicas in my 
> collection, two shard1 and two shard2. If it's RF3, then I have six: three 
> shard1 and three shard2.
> I'm using "Distinct RF" as a shorthand for "a single replica for every shard 
> in the collection".
> In the RF2 example above, if I have two Availability Zones, I would want a 
> Distinct RF in each AZ. So, a replica for shard1 and shard2 in AZ1, and a 
> replica for shard1 and shard2 in AZ2. I would *not* want, say, both shard1 
> replicas in AZ1 because then a failure of AZ1 could leave me with no replicas 
> for shard1 and an incomplete collection.
> If I had RF6 and two AZs, I would want three Distinct RFs in each AZ. (three 
> replicas for each shard, per AZ)
>
> I understand that {"replica": "<7", "node":"#ANY"} may result in two replicas 
> of the same shard ending up on the same node. However, the other rule should 
> prevent this: {"replica": "<2", "shard": "#EACH", "node": "#ANY"}
> So by using both rules, that should mean "no more than six replicas on a 
> node, where all the replicas on that node represent distinct shards". Right?
>
>
>
> On 2/12/18, 12:18 PM, "Noble Paul" <noble.p...@gmail.com> wrote:
>
> >>Goal: No node should have more than 6 shards
>
> This is not possible today
>
>  {"replica": "<7", "node":"#ANY"} , means don't put more than 7
> replicas of the collection (irrespective of the shards) in a given
> node
>
> what do you mean by distinct 'RF' ? I think we are screwing up the
> terminologies a bit here
>
> On Wed, Feb 7, 2018 at 1:38 PM, Jeff Wartes <jwar...@whitepages.com> 
> wrote:
> > I’ve been messing around with the Solr 7.2 autoscaling framework this 
> week. Some things seem trivial, but I’m also running into questions and 
> issues. If anyone else has experience with this stuff, I’d be glad to hear 
> it. Specifically:
> >
> >
> > Context:
> > -One collection, consisting of 42 shards, where up to 6 shards can fit 
> on a single node. (which means 7 nodes per Replication Factor)
> > -Three AZs, each with its own ip_2 value.
> >
> > Goals:
> >
> > Goal: Fully utilize available nodes.
> > Cluster Preference: {“maximize”: "cores”}
> >
> > Goal: No node should have more than one replica of a given shard
> > Rule: {"replica": "<2", "shard": "#EACH", "node": "#ANY"}
> >
> > Goal: No node should have more than 6 shards
> > Rule: {"replica": "<7", "node":"#ANY"}
> >
> > Goal: Where possible, distinct RFs should each exist in an AZ.
> > (Example1: I’d like 7 nodes with a complete RF in AZ 1 and 7 nodes with 
> a complete RF in AZ 2, and not end up with, say, both shard2 replicas in AZ 1)
> > (Example2: If I have 14 nodes in AZ 1 and 7 in AZ 2, I should have two 
> full RFs in AZ 1 and one in AZ 2)
> > Rule: ???
> >
> > I could have multiple non-strict rules perhaps? Like:
> >

Re: Solr Autoscaling multi-AZ rules

[Noble Paul]

>>Goal: No node should have more than 6 shards

This is not possible today

 {"replica": "<7", "node":"#ANY"} , means don't put more than 7
replicas of the collection (irrespective of the shards) in a given
node

what do you mean by distinct 'RF' ? I think we are screwing up the
terminologies a bit here

On Wed, Feb 7, 2018 at 1:38 PM, Jeff Wartes <jwar...@whitepages.com> wrote:
> I’ve been messing around with the Solr 7.2 autoscaling framework this week. 
> Some things seem trivial, but I’m also running into questions and issues. If 
> anyone else has experience with this stuff, I’d be glad to hear it. 
> Specifically:
>
>
> Context:
> -One collection, consisting of 42 shards, where up to 6 shards can fit on a 
> single node. (which means 7 nodes per Replication Factor)
> -Three AZs, each with its own ip_2 value.
>
> Goals:
>
> Goal: Fully utilize available nodes.
> Cluster Preference: {“maximize”: "cores”}
>
> Goal: No node should have more than one replica of a given shard
> Rule: {"replica": "<2", "shard": "#EACH", "node": "#ANY"}
>
> Goal: No node should have more than 6 shards
> Rule: {"replica": "<7", "node":"#ANY"}
>
> Goal: Where possible, distinct RFs should each exist in an AZ.
> (Example1: I’d like 7 nodes with a complete RF in AZ 1 and 7 nodes with a 
> complete RF in AZ 2, and not end up with, say, both shard2 replicas in AZ 1)
> (Example2: If I have 14 nodes in AZ 1 and 7 in AZ 2, I should have two full 
> RFs in AZ 1 and one in AZ 2)
> Rule: ???
>
> I could have multiple non-strict rules perhaps? Like:
> {"replica": "<2", "shard": "#EACH", "ip_2": "1", "strict":false}
> {"replica": "<3", "shard": "#EACH", "ip_2": "1", "strict":false}
> {"replica": "<4", "shard": "#EACH", "ip_2": "1", "strict":false}
> {"replica": "<2", "shard": "#EACH", "ip_2": "2", "strict":false}
> {"replica": "<3", "shard": "#EACH", "ip_2": "2", "strict":false}
> {"replica": "<4", "shard": "#EACH", "ip_2": "2", "strict":false}
> etc
> So having more than one RF in an AZ is a technical “violation”, but if 
> placement minimizes non-strict violations, replicas would tend to get placed 
> correctly.
>
>
> Given a working set of rules, I’m still having trouble with two things:
>
>   1.  I’ve manually created the “.system” collection, as it didn’t seem to 
> get created automatically. However, autoscaling activity is not getting 
> logged to it.
>   2.  I can’t seem to figure out how to scale up.
>  *   I’d presumed editing the collection’s “replicationFactor” would do 
> the trick, but it does not.
>  *   The “node-up” trigger will serve to replace lost replicas, but won’t 
> otherwise take advantage of additional capacity.
>
>i.  
> There’s a UTILIZENODE command in 7.2, but it appears that’s still something 
> you need to trigger manually.
>
> Anyone played with this stuff?



-- 
-
Noble Paul

Re: [ANNOUNCE] Apache Solr 7.1.0 released

[Noble Paul]

Thanks Shalin

On Wed, Oct 18, 2017 at 2:06 AM, Susheel Kumar <susheel2...@gmail.com> wrote:
> Thank you, Yonik. Able to download directly.
>
> On Tue, Oct 17, 2017 at 11:29 AM, Yonik Seeley <ysee...@gmail.com> wrote:
>
>> It pointed to 7.1.0 for me perhaps a browser cache issue?
>> Anyway, you can go directly as well:
>> http://www.apache.org/dyn/closer.lua/lucene/solr/7.1.0
>>
>> -Yonik
>>
>>
>> On Tue, Oct 17, 2017 at 11:25 AM, Susheel Kumar <susheel2...@gmail.com>
>> wrote:
>> > Thanks, Shalin.
>> >
>> > But the download mirror still has 7.0.1 not 7.1.0.
>> >
>> > http://www.apache.org/dyn/closer.lua/lucene/solr/7.0.1
>> >
>> >
>> >
>> >
>> > On Tue, Oct 17, 2017 at 5:28 AM, Shalin Shekhar Mangar
>> > <shalinman...@gmail.com> wrote:
>> >>
>> >> 17 October 2017, Apache Solr™ 7.1.0 available
>> >>
>> >> The Lucene PMC is pleased to announce the release of Apache Solr 7.1.0
>> >>
>> >> Solr is the popular, blazing fast, open source NoSQL search platform
>> >> from the Apache Lucene project. Its major features include powerful
>> >> full-text search, hit highlighting, faceted search, dynamic
>> >> clustering, database integration, rich document (e.g., Word, PDF)
>> >> handling, and geospatial search. Solr is highly scalable, providing
>> >> fault tolerant distributed search and indexing, and powers the search
>> >> and navigation features of many of the world's largest internet sites.
>> >>
>> >> Solr 7.1.0 is available for immediate download at:
>> >>
>> >> http://lucene.apache.org/solr/mirrors-solr-latest-redir.html
>> >>
>> >> See http://lucene.apache.org/solr/7_1_0/changes/Changes.html for a
>> >> full list of details.
>> >>
>> >> Solr 7.1.0 Release Highlights:
>> >>
>> >> * Critical Security Update: Fix for CVE-2017-12629 which is a working
>> >> 0-day exploit reported on the public mailing list. See
>> >> https://s.apache.org/FJDl
>> >>
>> >> * Auto-scaling: Solr can now move replicas automatically when a new
>> >> node is added or an existing node is removed using the auto scaling
>> >> policy framework introduced in 7.0
>> >>
>> >> * Auto-scaling: The 'autoAddReplicas' feature which was limited to
>> >> shared file systems is now available for all file systems. It has been
>> >> ported to use the new autoscaling framework internally.
>> >>
>> >> * Auto-scaling: New set-trigger, remove-trigger, set-listener,
>> >> remove-listener, suspend-trigger, resume-trigger APIs
>> >>
>> >> * Auto-scaling: New /autoscaling/history API to show past autoscaling
>> >> actions and cluster events
>> >>
>> >> * New JSON based Query DSL for Solr that extends JSON Request API to
>> >> also support all query parsers and their nested parameters
>> >>
>> >> * JSON Facet API: min/max aggregations are now supported on
>> >> single-valued date fields
>> >>
>> >> * Lucene's Geo3D (surface of sphere & ellipsoid) is now supported on
>> >> spatial RPT fields by setting spatialContextFactory="Geo3D".
>> >> Furthermore, this is the first time Solr has out of the box support
>> >> for polygons
>> >>
>> >> * Expanded support for statistical stream evaluators such as various
>> >> distributions, rank correlations, distances and more.
>> >>
>> >> * Multiple other optimizations and bug fixes
>> >>
>> >> You are encouraged to thoroughly read the "Upgrade Notes" at
>> >> http://lucene.apache.org/solr/7_1_0/changes/Changes.html or in the
>> >> CHANGES.txt file accompanying the release.
>> >>
>> >> Solr 7.1 also includes many other new features as well as numerous
>> >> optimizations and bugfixes of the corresponding Apache Lucene release.
>> >>
>> >> Please report any feedback to the mailing lists
>> >> (http://lucene.apache.org/solr/discussion.html)
>> >>
>> >> Note: The Apache Software Foundation uses an extensive mirroring
>> >> network for distributing releases. It is possible that the mirror you
>> >> are using may not have replicated the release yet. If that is the
>> >> case, please try another mirror. This also goes for Maven access.
>> >>
>> >> --
>> >> Regards,
>> >> Shalin Shekhar Mangar.
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
>> >>
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>



-- 
-
Noble Paul

Re: Rule-based Replica Placement not working with Solr 6.5.1

[Noble Paul]

did you try the rule
shard:shard1,port:8983

this ensures that all replicas of shard1 is allocated in the node w/ port 8983.

if it doesn't , it's a bug. Please open  aticket

On Tue, May 23, 2017 at 7:10 PM, Bernd Fehling
<bernd.fehl...@uni-bielefeld.de> wrote:
> After some analysis it turns out that they compare apples with oranges :-(
>
> Inside "tryAPermutationOfRules" the rule is called with rules.get() and
> the next step is calling rule.compare(), but they don't compare the nodes
> against the rule (or rules). They compare the nodes against each other.
>
> E.g. server1:8983, server2:7574, server1:7574,...
> What do you think will happen if comparing server1:8983 against server2:7574 
> (and so on)???
> It will _NEVER_ match!!!
>
> Regards
> Bernd
>
>
> Am 23.05.2017 um 08:54 schrieb Bernd Fehling:
>> No, that is way off, because:
>> 1. you have no "tag" defined.
>>shard and replica can be omitted and they will default to wildcard,
>>but a "tag" must be defined.
>> 2. replica must be an integer or a wildcard.
>>
>> Regards
>> Bernd
>>
>> Am 23.05.2017 um 01:17 schrieb Damien Kamerman:
>>> If you want all the replicas for shard1 on the same port then I think the
>>> rule is: 'shard:shard1,replica:port:8983'
>>>
>>> On 22 May 2017 at 18:47, Bernd Fehling <bernd.fehl...@uni-bielefeld.de>
>>> wrote:
>>>
>>>> I tried many settings with "Rule-based Replica Placement" on Solr 6.5.1
>>>> and came to the conclusion that it is not working at all.
>>>>
>>>> My test setup is 6 nodes on 3 servers (port 8983 and 7574 on each server).
>>>>
>>>> The call to create a new collection is
>>>> "http://localhost:8983/solr/admin/collections?action=CREATE=boss;
>>>> collection.configName=boss_configs=3=2&
>>>> maxShardsPerNode=1=shard:shard1,replica:<2,port:8983"
>>>>
>>>> With "rule=shard:shard1,replica:<2,port:8983" I expect that shard1 has
>>>> only nodes with port 8983 _OR_ it shoud fail due to "strict mode" because
>>>> the fuzzy operator "~" it not set.
>>>>
>>>> The result of the call is:
>>>> shard1 --> server2:7574 / server1:8983
>>>> shard2 --> server1:7574 / server3:8983
>>>> shard3 --> server2:8983 / server3:7574
>>>>
>>>> The expected result should be (at least!!!) shard1 --> server_x:8983 /
>>>> server_y:8983
>>>> where "_x" and "_y" can be anything between 1 and 3 but must be different.
>>>>
>>>> I think the problem is somewhere in "class ReplicaAssigner" with
>>>> "tryAllPermutations"
>>>> and "tryAPermutationOfRules".
>>>>
>>>> Regards
>>>> Bernd
>>>>
>>>



-- 
-
Noble Paul

Re: Using BasicAuth with SolrJ Code

[Noble Paul]

I tested this with the

following code

SolrRequest req = new QueryRequest(new SolrQuery("*:*"));//create a new
req.setBasicAuthCredentials("solr", "SolrRocks");
client.setDefaultCollection(COLLECTION);
NamedList res = cluster.getSolrClient().request(req);
System.out.println("successful"+ Utils.toJSON(res.asMap(5)));

seems to work for me.

which version of Solr and SolrJ are you using

On Sun, Apr 16, 2017 at 11:40 AM, Zheng Lin Edwin Yeo
<edwinye...@gmail.com> wrote:
> Ok, thank you.
>
> Regards,
> Edwin
>
> On 15 April 2017 at 08:05, Noble Paul <noble.p...@gmail.com> wrote:
>
>> I'll test with this and let you know
>>
>> On Apr 13, 2017 23:06, "Zheng Lin Edwin Yeo" <edwinye...@gmail.com> wrote:
>>
>> > The security.json which I'm using is the default one that is available
>> from
>> > the Solr Documentation https://cwiki.apache.org/confluence/display/
>> > solr/Basic+Authentication+Plugin.
>> >
>> > {
>> > "authentication":{
>> >"blockUnknown": true,
>> >"class":"solr.BasicAuthPlugin",
>> >"credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
>> > Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
>> > },
>> > "authorization":{
>> >"class":"solr.RuleBasedAuthorizationPlugin",
>> >"user-role":{"solr":"admin"},
>> >"permissions":[{"name":"security-edit",
>> >   "role":"admin"}]
>> > }}
>> >
>> >
>> > Regards,
>> > Edwin
>> >
>> > On 13 April 2017 at 19:53, Noble Paul <noble.p...@gmail.com> wrote:
>> >
>> > > That looks good. can you share the security.json (commenting out
>> > > anything that's sensitive of course)
>> > >
>> > > On Wed, Apr 12, 2017 at 5:10 PM, Zheng Lin Edwin Yeo
>> > > <edwinye...@gmail.com> wrote:
>> > > > This is what I get when I run the code.
>> > > >
>> > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
>> RemoteSolrException:
>> > > Error
>> > > > from server at http://localhost:8983/solr/testing: Expected mime
>> type
>> > > > application/octet-stream but got text/html. 
>> > > > 
>> > > > 
>> > > > Error 401 require authentication
>> > > > 
>> > > > HTTP ERROR 401
>> > > > Problem accessing /solr/testing/update. Reason:
>> > > > require authentication
>> > > > 
>> > > > 
>> > > >
>> > > > at
>> > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
>> > > executeMethod(HttpSolrClient.java:578)
>> > > > at
>> > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
>> > > HttpSolrClient.java:279)
>> > > > at
>> > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
>> > > HttpSolrClient.java:268)
>> > > > at org.apache.solr.client.solrj.SolrRequest.process(
>> > > SolrRequest.java:149)
>> > > > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
>> > > > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
>> > > > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
>> > > > at testing.indexing(testing.java:2939)
>> > > > at testing.main(testing.java:329)
>> > > > Exception in thread "main"
>> > > > org.apache.solr.client.solrj.impl.HttpSolrClient$
>> RemoteSolrException:
>> > > Error
>> > > > from server at http://localhost:8983/solr/testing: Expected mime
>> type
>> > > > application/octet-stream but got text/html. 
>> > > > 
>> > > > 
>> > > > Error 401 require authentication
>> > > > 
>> > > > HTTP ERROR 401
>> > > > Problem accessing /solr/testing/update. Reason:
>> > > > require authentication
>> > > > 
>> > > > 
>> > > >
>> > > > at
>> > > > org.apache.solr.client.solrj.impl.HttpSolrClient.
>> > > executeMethod(HttpSolrClient.java:578)
>> > > > at
>> > > > org.apache.solr.client.so

Re: Using BasicAuth with SolrJ Code

[Noble Paul]

I'll test with this and let you know

On Apr 13, 2017 23:06, "Zheng Lin Edwin Yeo" <edwinye...@gmail.com> wrote:

> The security.json which I'm using is the default one that is available from
> the Solr Documentation https://cwiki.apache.org/confluence/display/
> solr/Basic+Authentication+Plugin.
>
> {
> "authentication":{
>"blockUnknown": true,
>"class":"solr.BasicAuthPlugin",
>"credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0=
> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
> },
> "authorization":{
>"class":"solr.RuleBasedAuthorizationPlugin",
>"user-role":{"solr":"admin"},
>"permissions":[{"name":"security-edit",
>   "role":"admin"}]
> }}
>
>
> Regards,
> Edwin
>
> On 13 April 2017 at 19:53, Noble Paul <noble.p...@gmail.com> wrote:
>
> > That looks good. can you share the security.json (commenting out
> > anything that's sensitive of course)
> >
> > On Wed, Apr 12, 2017 at 5:10 PM, Zheng Lin Edwin Yeo
> > <edwinye...@gmail.com> wrote:
> > > This is what I get when I run the code.
> > >
> > > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error
> > > from server at http://localhost:8983/solr/testing: Expected mime type
> > > application/octet-stream but got text/html. 
> > > 
> > > 
> > > Error 401 require authentication
> > > 
> > > HTTP ERROR 401
> > > Problem accessing /solr/testing/update. Reason:
> > > require authentication
> > > 
> > > 
> > >
> > > at
> > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > executeMethod(HttpSolrClient.java:578)
> > > at
> > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > HttpSolrClient.java:279)
> > > at
> > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > HttpSolrClient.java:268)
> > > at org.apache.solr.client.solrj.SolrRequest.process(
> > SolrRequest.java:149)
> > > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
> > > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
> > > at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
> > > at testing.indexing(testing.java:2939)
> > > at testing.main(testing.java:329)
> > > Exception in thread "main"
> > > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error
> > > from server at http://localhost:8983/solr/testing: Expected mime type
> > > application/octet-stream but got text/html. 
> > > 
> > > 
> > > Error 401 require authentication
> > > 
> > > HTTP ERROR 401
> > > Problem accessing /solr/testing/update. Reason:
> > > require authentication
> > > 
> > > 
> > >
> > > at
> > > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > executeMethod(HttpSolrClient.java:578)
> > > at
> > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > HttpSolrClient.java:279)
> > > at
> > > org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> > HttpSolrClient.java:268)
> > > at org.apache.solr.client.solrj.SolrRequest.process(
> > SolrRequest.java:149)
> > > at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
> > > at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
> > > at testing.indexing(testing.java:3063)
> > > at testing.main(testing.java:329)
> > >
> > > Regards,
> > > Edwin
> > >
> > >
> > > On 12 April 2017 at 14:28, Noble Paul <noble.p...@gmail.com> wrote:
> > >
> > >> can u paste the stacktrace here
> > >>
> > >> On Tue, Apr 11, 2017 at 1:19 PM, Zheng Lin Edwin Yeo
> > >> <edwinye...@gmail.com> wrote:
> > >> > I found from StackOverflow  that we should declare it this way:
> > >> > http://stackoverflow.com/questions/43335419/using-
> > >> basicauth-with-solrj-code
> > >> >
> > >> >
> > >> > SolrRequest req = new QueryRequest(new SolrQuery("*:*"));//create a
> > new
> > >> > request object
> > >> > req.setBasicAuthCredentials(userName, password);
> > 

Re: Using BasicAuth with SolrJ Code

[Noble Paul]

That looks good. can you share the security.json (commenting out
anything that's sensitive of course)

On Wed, Apr 12, 2017 at 5:10 PM, Zheng Lin Edwin Yeo
<edwinye...@gmail.com> wrote:
> This is what I get when I run the code.
>
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
> from server at http://localhost:8983/solr/testing: Expected mime type
> application/octet-stream but got text/html. 
> 
> 
> Error 401 require authentication
> 
> HTTP ERROR 401
> Problem accessing /solr/testing/update. Reason:
> require authentication
> 
> 
>
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:578)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
> at testing.indexing(testing.java:2939)
> at testing.main(testing.java:329)
> Exception in thread "main"
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
> from server at http://localhost:8983/solr/testing: Expected mime type
> application/octet-stream but got text/html. 
> 
> 
> Error 401 require authentication
> 
> HTTP ERROR 401
> Problem accessing /solr/testing/update. Reason:
> require authentication
> 
> 
>
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:578)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
> at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:484)
> at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:463)
> at testing.indexing(testing.java:3063)
> at testing.main(testing.java:329)
>
> Regards,
> Edwin
>
>
> On 12 April 2017 at 14:28, Noble Paul <noble.p...@gmail.com> wrote:
>
>> can u paste the stacktrace here
>>
>> On Tue, Apr 11, 2017 at 1:19 PM, Zheng Lin Edwin Yeo
>> <edwinye...@gmail.com> wrote:
>> > I found from StackOverflow  that we should declare it this way:
>> > http://stackoverflow.com/questions/43335419/using-
>> basicauth-with-solrj-code
>> >
>> >
>> > SolrRequest req = new QueryRequest(new SolrQuery("*:*"));//create a new
>> > request object
>> > req.setBasicAuthCredentials(userName, password);
>> > solrClient.request(req);
>> >
>> > Is that correct?
>> >
>> > For this, the NullPointerException is not coming out, but the SolrJ is
>> > still not able to get authenticated. I'm still getting Error Code 401
>> even
>> > after putting in this code.
>> >
>> > Any advice on which part of the SolrJ code should we place this code in?
>> >
>> > Regards,
>> > Edwin
>> >
>> >
>> > On 10 April 2017 at 23:50, Zheng Lin Edwin Yeo <edwinye...@gmail.com>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> I have just set up the Basic Authentication Plugin in Solr 6.4.2 on
>> >> SolrCloud, and I am trying to modify my SolrJ code so that the code can
>> go
>> >> through the authentication and do the indexing.
>> >>
>> >> I tried using the following code from the Solr Documentation
>> >> https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+
>> >> Plugin.
>> >>
>> >> SolrRequest req ;//create a new request object
>> >> req.setBasicAuthCredentials(userName, password);
>> >> solrClient.request(req);
>> >>
>> >> However, the code complains that the req is not initialized.
>> >>
>> >> If I initialized it, it will be initialize as null.
>> >>
>> >> SolrRequest req = null;//create a new request object
>> >> req.setBasicAuthCredentials(userName, password);
>> >> solrClient.request(req);
>> >>
>> >> This will caused a null pointer exception.
>> >> Exception in thread "main" java.lang.NullPointerException
>> >>
>> >> How should we go about putting these codes, so that the error can be
>> >> prevented?
>> >>
>> >> Regards,
>> >> Edwin
>> >>
>> >>
>>
>>
>>
>> --
>> -
>> Noble Paul
>>



-- 
-
Noble Paul

Re: Using BasicAuth with SolrJ Code

[Noble Paul]

can u paste the stacktrace here

On Tue, Apr 11, 2017 at 1:19 PM, Zheng Lin Edwin Yeo
<edwinye...@gmail.com> wrote:
> I found from StackOverflow  that we should declare it this way:
> http://stackoverflow.com/questions/43335419/using-basicauth-with-solrj-code
>
>
> SolrRequest req = new QueryRequest(new SolrQuery("*:*"));//create a new
> request object
> req.setBasicAuthCredentials(userName, password);
> solrClient.request(req);
>
> Is that correct?
>
> For this, the NullPointerException is not coming out, but the SolrJ is
> still not able to get authenticated. I'm still getting Error Code 401 even
> after putting in this code.
>
> Any advice on which part of the SolrJ code should we place this code in?
>
> Regards,
> Edwin
>
>
> On 10 April 2017 at 23:50, Zheng Lin Edwin Yeo <edwinye...@gmail.com> wrote:
>
>> Hi,
>>
>> I have just set up the Basic Authentication Plugin in Solr 6.4.2 on
>> SolrCloud, and I am trying to modify my SolrJ code so that the code can go
>> through the authentication and do the indexing.
>>
>> I tried using the following code from the Solr Documentation
>> https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+
>> Plugin.
>>
>> SolrRequest req ;//create a new request object
>> req.setBasicAuthCredentials(userName, password);
>> solrClient.request(req);
>>
>> However, the code complains that the req is not initialized.
>>
>> If I initialized it, it will be initialize as null.
>>
>> SolrRequest req = null;//create a new request object
>> req.setBasicAuthCredentials(userName, password);
>> solrClient.request(req);
>>
>> This will caused a null pointer exception.
>> Exception in thread "main" java.lang.NullPointerException
>>
>> How should we go about putting these codes, so that the error can be
>> prevented?
>>
>> Regards,
>> Edwin
>>
>>



-- 
-
Noble Paul

Re: Using a library from blob-store without "add-runtimelib"

[Noble Paul]

you could add an entry to solrconfig.xml



I have not tested it muself, but it should work

On Mon, Oct 31, 2016 at 11:38 PM, Samuel García Martínez
<samuelgmarti...@gmail.com> wrote:
> Hi!
>
> I'm wondering if there's any possibility to tell the current collection to
> use a library stored in the blobstore using the solrconfig.xml instead of
> having to do a /config request.
>
> Same way local filesystem libraries are added in solrconfig, it could exist
> for blobstore libraries. Since it's stored as an overlay property and merge
> in memory, not loaded until first request I think it's a reasonable
> approach or am I missing something?
>
> Thanks!



-- 
-----
Noble Paul

Re: Custom authentication plugin & inter-node auth

[Noble Paul]

It should be related to

SOLR-9692: blockUnknown property makes inter-node communication impossible

This is fixed in 6.3



On Tue, Nov 1, 2016 at 8:44 AM, devanshic02 <dink...@gmail.com> wrote:
>
> Hi
>
> I am facing same issue
> I have a custom authentication plugin and it is calling  admin/info/key when
> the PKIAuthenticationPlugin is attempting to authenticate an inter-node
> request. Can you let me know how you resolved the same
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Custom-authentication-plugin-inter-node-auth-tp4294675p4303933.html
> Sent from the Solr - User mailing list archive at Nabble.com.




-- 
-----
Noble Paul

Re: Custom auth plugin not loaded in SolrCloud

[Noble Paul]

yes, runtime lib cannot be used for loading container level plugins
yet. Eventually they must. You can open a ticket

On Mon, Jan 4, 2016 at 1:07 AM, tine-2 <kristine.jet...@kreuzwerker.de> wrote:
> Hi,
>
> are there any news on this? Was anyone able to get it to work?
>
> Cheers,
>
> tine
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Custom-auth-plugin-not-loaded-in-SolrCloud-tp4245670p4248340.html
> Sent from the Solr - User mailing list archive at Nabble.com.



-- 
-----
Noble Paul

Re: API accessible without authentication even though Basic Auth Plugin is enabled

[Noble Paul]

A 5.3.2 release is coming up which will back port the fixes introduced in
5.4
On Dec 17, 2015 10:25 PM, "tine-2" <kristine.jet...@kreuzwerker.de> wrote:

> Noble Paul നോബിള്‍  नोब्ळ् wrote
> > It works as designed.
> >
> > Protect the read path [...]
>
> Works like described in 5.4.0, didn't work in 5.3.1, s.
> https://issues.apache.org/jira/browse/SOLR-8408
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/API-accessible-without-authentication-even-though-Basic-Auth-Plugin-is-enabled-tp4244940p4246099.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Security Problems

[Noble Paul]

I have opened https://issues.apache.org/jira/browse/SOLR-8429

On Wed, Dec 16, 2015 at 9:32 PM, Noble Paul <noble.p...@gmail.com> wrote:
> I don't this behavior is intuitive. It is very easy to misunderstand
>
> I would rather just add a flag to "authentication" plugin section
> which says "blockUnauthenticated" : true
>
> which means all unauthenticated requests must be blocked.
>
>
>
>
> On Tue, Dec 15, 2015 at 7:09 PM, Jan Høydahl <jan@cominvent.com> wrote:
>> Yes, that’s why I believe it should be:
>> 1) if only authentication is enabled, all users must authenticate and all 
>> authenticated users can do anything.
>> 2) if authz is enabled, then all users must still authenticate, and can by 
>> default do nothing at all, unless assigned proper roles
>> 3) if a user is assigned the default “read” rule, and a collection adds a 
>> custom “/myselect” handler, that one is unavailable until the user gets it 
>> assigned
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>> 14. des. 2015 kl. 14.15 skrev Noble Paul <noble.p...@gmail.com>:
>>>
>>> ". If all paths were closed by default, forgetting to configure a path
>>> would not result in a security breach like today."
>>>
>>> But it will still mean that unauthorized users are able to access,
>>> like guest being able to post to "/update". Just authenticating is not
>>> enough without proper authorization
>>>
>>> On Mon, Dec 14, 2015 at 3:59 PM, Jan Høydahl <jan@cominvent.com> wrote:
>>>>> 1) "read" should cover all the paths
>>>>
>>>> This is very fragile. If all paths were closed by default, forgetting to 
>>>> configure a path would not result in a security breach like today.
>>>>
>>>> /Jan
>>>
>>>
>>>
>>> --
>>> -
>>> Noble Paul
>>
>
>
>
> --
> -
> Noble Paul



-- 
-
Noble Paul

Re: Security Problems

[Noble Paul]

I don't this behavior is intuitive. It is very easy to misunderstand

I would rather just add a flag to "authentication" plugin section
which says "blockUnauthenticated" : true

which means all unauthenticated requests must be blocked.




On Tue, Dec 15, 2015 at 7:09 PM, Jan Høydahl <jan@cominvent.com> wrote:
> Yes, that’s why I believe it should be:
> 1) if only authentication is enabled, all users must authenticate and all 
> authenticated users can do anything.
> 2) if authz is enabled, then all users must still authenticate, and can by 
> default do nothing at all, unless assigned proper roles
> 3) if a user is assigned the default “read” rule, and a collection adds a 
> custom “/myselect” handler, that one is unavailable until the user gets it 
> assigned
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>> 14. des. 2015 kl. 14.15 skrev Noble Paul <noble.p...@gmail.com>:
>>
>> ". If all paths were closed by default, forgetting to configure a path
>> would not result in a security breach like today."
>>
>> But it will still mean that unauthorized users are able to access,
>> like guest being able to post to "/update". Just authenticating is not
>> enough without proper authorization
>>
>> On Mon, Dec 14, 2015 at 3:59 PM, Jan Høydahl <jan@cominvent.com> wrote:
>>>> 1) "read" should cover all the paths
>>>
>>> This is very fragile. If all paths were closed by default, forgetting to 
>>> configure a path would not result in a security breach like today.
>>>
>>> /Jan
>>
>>
>>
>> --
>> -
>> Noble Paul
>



-- 
-
Noble Paul

Re: Security Problems

[Noble Paul]

". If all paths were closed by default, forgetting to configure a path
would not result in a security breach like today."

But it will still mean that unauthorized users are able to access,
like guest being able to post to "/update". Just authenticating is not
enough without proper authorization

On Mon, Dec 14, 2015 at 3:59 PM, Jan Høydahl <jan@cominvent.com> wrote:
>> 1) "read" should cover all the paths
>
> This is very fragile. If all paths were closed by default, forgetting to 
> configure a path would not result in a security breach like today.
>
> /Jan



-- 
-----
Noble Paul

Re: Solr5.3.1 solrcloud Enabling Basic AUthentication

[Noble Paul]

You don't need to submit a sha256, Solr will do itself. Just use the
provided commands

please refer this
https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin

On Mon, Dec 14, 2015 at 6:56 AM, soledede_w...@ehsy.com <
soledede_w...@ehsy.com> wrote:

> I want to restrict Admin UI , I know can config the security.json
>
> but how.
>
>
> if use sha256(password+salt) hash) ,how to submit the salt for solr
> server.
>
> who can give me a simple example
> Thanks
>
> --
> soledede_w...@ehsy.com
>



-- 
-----
Noble Paul

Re: Security Problems

[Noble Paul]

This could have multiple solutions

1) "read" should cover all the paths
2) system properties are a strict NO . This can be strictly a property
of the Authentication plugin. So , you can use the API to modify the
property.

On Sat, Nov 21, 2015 at 3:57 AM, Jan Høydahl <jan@cominvent.com> wrote:
>> ideally we should have a simple permission name called "all" (which we
>> don't have)
>>
>> so that one rule should be enough
>>
>> "name":"all",
>> "role":"somerole"
>>
>> Open a ticket and we should fix it for 5.4.0
>> It should also include  the admin paths as well
>
> Yes, that would be convenient.
>
> I still don’t like the existing "open-by-default” security mode of Solr. It 
> is very fragile to mis-configuration without people noticing. Take the 
> well-known permission “read” for instance. It protects /select and /get. But 
> it won’t protect /query, /browse, /export, /spell, /suggest, /tvrh, /terms, 
> /clustering or /elevate, all which also expose sensitive info.
>
> How about allowing to choose between three different security modes?
>
> -Dsolr.security.mode=open  : As today - paths not configured are wide 
> open
> -Dsolr.security.mode=authenticated : Paths not configured are open to any 
> authenticated user
> -Dsolr.security.mode=explicit  : Paths not configured are closed to all. 
> All acccess is explicitly configured
>
> /Jan



-- 
-
Noble Paul

Re: API accessible without authentication even though Basic Auth Plugin is enabled

[Noble Paul]

It works as designed.

Protect the read path using the following command
curl  http://localhost:8983/solr/admin/authorization -H
'Content-type:application/json' -d '{ set-permission : {name : read,
role : admin}}'
Then, you will have the right experience

In this case /select is not protected. So an unauthenticated request
must be able to access /select path. authentication layer has no idea
whether it is a protected resource or not. So, when no credentials
headers are sent it sets the user principal as null and lets the
request go through. Whereas in the case of wrong credentials, the
choices are 1) fail the request or 2) forward the request as if the
principal is null . #2 would be bad user experience because the
Authorization layer would say principal is null (unauthenicated) and
the user would not know that the credentials were wrong.

On Sat, Dec 12, 2015 at 5:14 AM, Chris Hostetter
<hossman_luc...@fucit.org> wrote:
>
> Ugh ... no sure WTF is going on here, but that's for reporting it with
> clear steps to reproduce...
>
> https://issues.apache.org/jira/browse/SOLR-8408
>
> : Date: Fri, 11 Dec 2015 20:43:46 +0100
> : From: Kristine Jetzke <kristine.jet...@gmx.de>
> : Reply-To: solr-user@lucene.apache.org
> : To: solr-user@lucene.apache.org
> : Subject: API accessible without authentication even though Basic Auth Plugin
> : is enabled
> :
> : Hi,
> :
> : I noticed that it is possible to access the API even if the Basic Auth 
> plugin is enabled. Is that a known issue/done on purpose? I didn’t find 
> anything in JIRA or the docs.
> :
> : What I did:
> : - Started zookeeper on port 2181 and uploaded security.json from 
> https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin 
> <https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin>
> : - Started Solr cluster using cloud example: bin/solr start -e cloud -c -z 
> localhost:2181
> : - Executed the following commands:
> : - curl -u solr:SolrRocks 
> 'http://localhost:8983/solr/gettingstarted_shard1_replica1/select?q=*%3A*=json=true':
>  Returns 200 as expected
> : - curl -u solr:wrongPassword 
> 'http://localhost:8983/solr/gettingstarted_shard1_replica1/select?q=*%3A*=json=true':
>  Returns 401 as expected
> : - curl 
> 'http://localhost:8983/solr/gettingstarted_shard1_replica1/select?q=*%3A*=json=true':
>  Returns 200 even though no Authorization header is set.
> :
> : I don’t understand why the last part works like it does. If I don’t give 
> credentials, I would expect that the behavior is the same as with invalid 
> credentials. Is there a special reason why it behaves like this? I’m 
> wondering because I’m working on a custom authentication plugin and was 
> looking into the existing ones to understand how they work.
> :
> : Thanks,
> :
> : tine
>
> -Hoss
> http://www.lucidworks.com/



-- 
-
Noble Paul

Re: how to secure standalone solr

[Noble Paul]

For standalone Solr , Kerberos is the only option for authentication.
If you have  a SolrCloud setup, you have other options

https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin
https://cwiki.apache.org/confluence/display/solr/Rule-Based+Authorization+Plugin

On Fri, Dec 11, 2015 at 11:02 PM, Don Bosco Durai <bo...@apache.org> wrote:
>>Anyone told me how to secure standalone solr .
> Recently there were few discussion on this. In short, it is not tested and 
> there doesn’t seem to a plan to test it.
>
>>1.)using Kerberos Plugin is a good practice or any other else.
> The answer depends how you are using it. Where you are deploying it, who is 
> accessing it, whether you want to restrict by access type (read/write), what 
> authentication environment (LDAP/AD, Kerberos, etc) you already have.
>
> Depending upon your use cases and environment, you may have one or more 
> options.
>
> Bosco
>
>
>
>
>
>
> On 12/11/15, 4:27 AM, "Mugeesh Husain" <muge...@gmail.com> wrote:
>
>>Hello,
>>
>>Anyone told me how to secure standalone solr .
>>
>>1.)using Kerberos Plugin is a good practice or any other else.
>>
>>
>>
>>--
>>View this message in context: 
>>http://lucene.472066.n3.nabble.com/how-to-secure-standalone-solr-tp4244866.html
>>Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
-
Noble Paul

Re: Authorization API versus zkcli.sh

[Noble Paul]

Oakley,

1) ideally you should only upload the first empty security.json. In
that case there is no need to add the version attributes. Thereafter
you are supposed to use the API
2) Just in case you need to upload the security.json, please remove
that attribute

shalin:
The version is added inside the json is because we have no idea
whether an edit caused authentication or authorization to be reloaded
. By adding separate versions I'm able to make that distinction

On Fri, Dec 11, 2015 at 9:02 PM, Oakley, Craig (NIH/NLM/NCBI) [C]
<craig.oak...@nih.gov> wrote:
> So, when one has finished constructing the desired security.json (by means of 
> Authentication and Authorization commands) and then run "zkcli.sh -cmd 
> getfile" to get this security.json in order for it to be used as a template: 
> one should edit the template to remove this "":{"v":85} clause (and the comma 
> which precedes it): correct?
>
> I notice that the documented minimal security.json which simply creates the 
> solr:SolrRocks login:pswd does not have such a clause: so I assume that the 
> lack of such a clause is not an error.
>
> 
> From: Anshum Gupta [ans...@anshumgupta.net]
> Sent: Friday, December 11, 2015 9:48 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Authorization API versus zkcli.sh
>
> yes, that's the assumption. The reason why there's a version there is to
> optimize on reloads i.e. Authentication and Authorization plugins are
> reloaded only when the version number is changed. e.g.
> * Start with Ver 1 for both authentication and authorization
> * Make changes to Authentication, the version for this section is updated
> to the znode version, while the version for the authorization section is
> not changed. This forces the authentication plugin to be reloaded but not
> the authorization plugin. Similarly for authorization.
>
> It's a way to optimize the reloads without splitting the definition into 2
> znodes, which is also an option.
>
>
> On Fri, Dec 11, 2015 at 8:06 PM, Shalin Shekhar Mangar <
> shalinman...@gmail.com> wrote:
>
>> Shouldn't this be the znode version? Why put a version in
>> security.json? Or is the idea that the user will upload security.json
>> only once and then use the security APIs for all further changes?
>>
>> On Fri, Dec 11, 2015 at 11:51 AM, Noble Paul <noble.p...@gmail.com> wrote:
>> > Please do not put any number. That number is used by the system to
>> > optimize loading/reloading plugins. It is not relevant for the user.
>> >
>> > On Thu, Dec 10, 2015 at 11:52 PM, Oakley, Craig (NIH/NLM/NCBI) [C]
>> > <craig.oak...@nih.gov> wrote:
>> >> Looking at security.json in Zookeeper, I notice that both the
>> authentication section and the authorization section ends with something
>> like
>> >>
>> >> "":{"v":47}},
>> >>
>> >> Am I correct in thinking that this 47 (in this case) is a version
>> number, and that ANY number could be used in the file uploaded to
>> security.json using "zkcli.sh -putfile"?
>> >>
>> >> Or is this some sort of checksum whose value must match some unclear
>> criteria?
>> >>
>> >>
>> >> -Original Message-
>> >> From: Anshum Gupta [mailto:ans...@anshumgupta.net]
>> >> Sent: Sunday, December 06, 2015 8:42 AM
>> >> To: solr-user@lucene.apache.org
>> >> Subject: Re: Authorization API versus zkcli.sh
>> >>
>> >> There's nothing cluster specific in security.json if you're using those
>> >> plugins. It is totally safe to just take the file from one cluster and
>> >> upload it for another for things to work.
>> >>
>> >> On Sat, Dec 5, 2015 at 3:38 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
>> >> craig.oak...@nih.gov> wrote:
>> >>
>> >>> Looking through
>> >>>
>> cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
>> >>> one notices that security.json is initially created by zkcli.sh, and
>> then
>> >>> modified by means of the Authentication API and the Authorization API.
>> By
>> >>> and large, this sounds like a good way to accomplish such tasks,
>> assuming
>> >>> that these APIs do some error checking to prevent corruption of
>> >>> security.json
>> >>>
>> >>> I was wondering about cases where one is cloning an existing Solr
>> >>> instance, such as when creating an instanc

Re: Authorization API versus zkcli.sh

[Noble Paul]

Please do not put any number. That number is used by the system to
optimize loading/reloading plugins. It is not relevant for the user.

On Thu, Dec 10, 2015 at 11:52 PM, Oakley, Craig (NIH/NLM/NCBI) [C]
<craig.oak...@nih.gov> wrote:
> Looking at security.json in Zookeeper, I notice that both the authentication 
> section and the authorization section ends with something like
>
> "":{"v":47}},
>
> Am I correct in thinking that this 47 (in this case) is a version number, and 
> that ANY number could be used in the file uploaded to security.json using 
> "zkcli.sh -putfile"?
>
> Or is this some sort of checksum whose value must match some unclear criteria?
>
>
> -Original Message-
> From: Anshum Gupta [mailto:ans...@anshumgupta.net]
> Sent: Sunday, December 06, 2015 8:42 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Authorization API versus zkcli.sh
>
> There's nothing cluster specific in security.json if you're using those
> plugins. It is totally safe to just take the file from one cluster and
> upload it for another for things to work.
>
> On Sat, Dec 5, 2015 at 3:38 AM, Oakley, Craig (NIH/NLM/NCBI) [C] <
> craig.oak...@nih.gov> wrote:
>
>> Looking through
>> cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
>> one notices that security.json is initially created by zkcli.sh, and then
>> modified by means of the Authentication API and the Authorization API. By
>> and large, this sounds like a good way to accomplish such tasks, assuming
>> that these APIs do some error checking to prevent corruption of
>> security.json
>>
>> I was wondering about cases where one is cloning an existing Solr
>> instance, such as when creating an instance in Amazon Cloud. If one has a
>> security.json that has been thoroughly tried and successfully tested on
>> another Solr instance, is it possible / safe / not-un-recommended to use
>> zkcli.sh to load the full security.json (as extracted via zkcli.sh from the
>> Zookeeper of the thoroughly tested existing instance)? Or would the
>> official verdict be that the only acceptable way to create security.json is
>> to load a minimal version with zkcli.sh and then to build the remaining
>> components with the Authentication API and the Authorization API (in a
>> script, if one wants to automate the process: although such a script would
>> have to include plain-text passwords)?
>>
>> I figured there is no harm in asking.
>>
>
>
>
> --
> Anshum Gupta



-- 
-
Noble Paul

Re: Security Problems

[Noble Paul]

What is the smallest possible security.json required currently to
protect all possible paths (except those served by Jetty)?



You would need 2 rules
1)
"name":"all-admin",
"collection": null,
"path":"/*"
"role:"somerole"

2) all core handlers
"name":"all-core-handlers",
"path":"/*"
"role":"somerole"


ideally we should have a simple permission name called "all" (which we
don't have)

so that one rule should be enough

"name":"all",
"role":"somerole"

Open a ticket and we should fix it for 5.4.0
It should also include  the admin paths as well


On Thu, Nov 19, 2015 at 6:02 PM, Jan Høydahl <jan@cominvent.com> wrote:
> Would it not be less surprising if ALL requests to Solr required 
> authentication once an AuthenticationPlugin was enabled?
> Then, if no AuthorizationPlugin was active, all authenticated users could do 
> anything.
> But if AuthorizationPlugin was configured, you could only do what your role 
> allows you to?
>
> As it is now it is super easy to forget a path, say you protect /select but 
> not /browse and /query, or someone creates a collection
> with some new endpoints and forgets to update security.json - then that 
> endpoint would be wide open!
>
> What is the smallest possible security.json required currently to protect all 
> possible paths (except those served by Jetty)?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>> 18. nov. 2015 kl. 20.31 skrev Upayavira <u...@odoko.co.uk>:
>>
>> I'm very happy for the admin UI to be served another way - i.e. not
>> direct from Jetty, if that makes the task of securing it easier.
>>
>> Perhaps a request handler specifically for UI resources which would make
>> it possible to secure it all in a more straight-forward way?
>>
>> Upayavira
>>
>> On Wed, Nov 18, 2015, at 01:54 PM, Noble Paul wrote:
>>> As of now the admin-ui calls are not protected. The static calls are
>>> served by jetty and it bypasses the authentication mechanism
>>> completely. If the admin UI relies on some API call which is served by
>>> Solr.
>>> The other option is to revamp the framework to take care of admin UI
>>> (static content) as well. This would be cleaner solution
>>>
>>>
>>> On Wed, Nov 18, 2015 at 2:32 PM, Upayavira <u...@odoko.co.uk> wrote:
>>>> Not sure I quite understand.
>>>>
>>>> You're saying that the cost for the UI is not large, but then suggesting
>>>> we protect just one resource (/admin/security-check)?
>>>>
>>>> Why couldn't we create the permission called 'admin-ui' and protect
>>>> everything under /admin/ui/ for example? Along with the root HTML link
>>>> too.
>>>>
>>>> Upayavira
>>>>
>>>> On Wed, Nov 18, 2015, at 07:46 AM, Noble Paul wrote:
>>>>> The authentication plugin is not expensive if you are talking in the
>>>>> context of admin UI. After all it is used not like 100s of requests
>>>>> per second.
>>>>>
>>>>> The simplest solution would be
>>>>>
>>>>> provide a well known permission name called "admin-ui"
>>>>>
>>>>> ensure that every admin page load makes a call to some resource say
>>>>> "/admin/security-check"
>>>>>
>>>>> Then we can just protect that .
>>>>>
>>>>> The only concern thatI have is the false sense of security it would
>>>>> give to the user
>>>>>
>>>>> But, that is a different point altogether
>>>>>
>>>>> On Wed, Nov 11, 2015 at 1:52 AM, Upayavira <u...@odoko.co.uk> wrote:
>>>>>> Is the authentication plugin that expensive?
>>>>>>
>>>>>> I can help by minifying the UI down to a smaller number of CSS/JS/etc
>>>>>> files :-)
>>>>>>
>>>>>> It may be overkill, but it would also give better experience. And isn't
>>>>>> that what most applications do? Check authentication tokens on every
>>>>>> request?
>>>>>>
>>>>>> Upayavira
>>>>>>
>>>>>> On Tue, Nov 10, 2015, at 07:33 PM, Anshum Gupta wrote:
>>>>>>> The reason why we bypass that is so that we don't hit the authentication
>>>>>>> plu

Re: Security Problems

[Noble Paul]

Everything requires explicit rules, if you wish to protect "/update/*"
create a permission with name "update" and assign a role for the same.
If you don't have an explicit rule, those paths are accessible by all

On Wed, Nov 18, 2015 at 8:10 PM, Jan Høydahl <jan@cominvent.com> wrote:
> I tried out BasicAuthPlugin today.
> Surprised that not admin UI is protected.
> But even more surprised that only /select seems to be protected for not 
> logged in users.
> I can create collections and /update documents without being prompted for pw.
>
> My security.json is https://gist.github.com/janhoy/d18854c75461816fb947
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>> 18. nov. 2015 kl. 14.54 skrev Noble Paul <noble.p...@gmail.com>:
>>
>> As of now the admin-ui calls are not protected. The static calls are
>> served by jetty and it bypasses the authentication mechanism
>> completely. If the admin UI relies on some API call which is served by
>> Solr.
>> The other option is to revamp the framework to take care of admin UI
>> (static content) as well. This would be cleaner solution
>>
>>
>> On Wed, Nov 18, 2015 at 2:32 PM, Upayavira <u...@odoko.co.uk> wrote:
>>> Not sure I quite understand.
>>>
>>> You're saying that the cost for the UI is not large, but then suggesting
>>> we protect just one resource (/admin/security-check)?
>>>
>>> Why couldn't we create the permission called 'admin-ui' and protect
>>> everything under /admin/ui/ for example? Along with the root HTML link
>>> too.
>>>
>>> Upayavira
>>>
>>> On Wed, Nov 18, 2015, at 07:46 AM, Noble Paul wrote:
>>>> The authentication plugin is not expensive if you are talking in the
>>>> context of admin UI. After all it is used not like 100s of requests
>>>> per second.
>>>>
>>>> The simplest solution would be
>>>>
>>>> provide a well known permission name called "admin-ui"
>>>>
>>>> ensure that every admin page load makes a call to some resource say
>>>> "/admin/security-check"
>>>>
>>>> Then we can just protect that .
>>>>
>>>> The only concern thatI have is the false sense of security it would
>>>> give to the user
>>>>
>>>> But, that is a different point altogether
>>>>
>>>> On Wed, Nov 11, 2015 at 1:52 AM, Upayavira <u...@odoko.co.uk> wrote:
>>>>> Is the authentication plugin that expensive?
>>>>>
>>>>> I can help by minifying the UI down to a smaller number of CSS/JS/etc
>>>>> files :-)
>>>>>
>>>>> It may be overkill, but it would also give better experience. And isn't
>>>>> that what most applications do? Check authentication tokens on every
>>>>> request?
>>>>>
>>>>> Upayavira
>>>>>
>>>>> On Tue, Nov 10, 2015, at 07:33 PM, Anshum Gupta wrote:
>>>>>> The reason why we bypass that is so that we don't hit the authentication
>>>>>> plugin for every request that comes in for static content. I think we
>>>>>> could
>>>>>> call the authentication plugin for that but that'd be an overkill. Better
>>>>>> experience ? yes
>>>>>>
>>>>>> On Tue, Nov 10, 2015 at 11:24 AM, Upayavira <u...@odoko.co.uk> wrote:
>>>>>>
>>>>>>> Noble,
>>>>>>>
>>>>>>> I get that a UI which is open source does not benefit from ACL control -
>>>>>>> we're not giving away anything that isn't public (other than perhaps
>>>>>>> info that could be used to identify the version of Solr, or even the
>>>>>>> fact that it *is* solr).
>>>>>>>
>>>>>>> However, from a user experience point of view, requiring credentials to
>>>>>>> see the UI would be more conventional, and therefore lead to less
>>>>>>> confusion. Is it possible for us to protect the UI static files, only
>>>>>>> for the sake of user experience, rather than security?
>>>>>>>
>>>>>>> Upayavira
>>>>>>>
>>>>>>> On Tue, Nov 10, 2015, at 12:01 PM, Noble Paul wrote:
>>>>>>>> The admin UI is a bunch of static pages . We don't let the ACL control
>>>>>>>> static content
>>>>>>>>
>>>>>>>> you must blacklist all the core/collection apis and it is pretty much
>>>>>>>> useless for anyone to access the admin UI (w/o the credentials , of
>>>>>>>> course)
>>>>>>>>
>>>>>>>> On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟 <mabaizh...@126.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> After I configure Authentication with Basic Authentication Plugin and
>>>>>>> Authorization with Rule-Based Authorization Plugin, How can I prevent 
>>>>>>> the
>>>>>>> strangers from visiting my solr by browser? For example, if the stranger
>>>>>>> visit the http://(my host):8983, the browser will pop up a window and
>>>>>>> says "the server http://(my host):8983 requires a username and
>>>>>>> password"
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -
>>>>>>>> Noble Paul
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Anshum Gupta
>>>>
>>>>
>>>>
>>>> --
>>>> -
>>>> Noble Paul
>>
>>
>>
>> --
>> -
>> Noble Paul
>



-- 
-
Noble Paul

Re: Security Problems

[Noble Paul]

As of now the admin-ui calls are not protected. The static calls are
served by jetty and it bypasses the authentication mechanism
completely. If the admin UI relies on some API call which is served by
Solr.
The other option is to revamp the framework to take care of admin UI
(static content) as well. This would be cleaner solution


On Wed, Nov 18, 2015 at 2:32 PM, Upayavira <u...@odoko.co.uk> wrote:
> Not sure I quite understand.
>
> You're saying that the cost for the UI is not large, but then suggesting
> we protect just one resource (/admin/security-check)?
>
> Why couldn't we create the permission called 'admin-ui' and protect
> everything under /admin/ui/ for example? Along with the root HTML link
> too.
>
> Upayavira
>
> On Wed, Nov 18, 2015, at 07:46 AM, Noble Paul wrote:
>> The authentication plugin is not expensive if you are talking in the
>> context of admin UI. After all it is used not like 100s of requests
>> per second.
>>
>> The simplest solution would be
>>
>> provide a well known permission name called "admin-ui"
>>
>> ensure that every admin page load makes a call to some resource say
>> "/admin/security-check"
>>
>> Then we can just protect that .
>>
>> The only concern thatI have is the false sense of security it would
>> give to the user
>>
>> But, that is a different point altogether
>>
>> On Wed, Nov 11, 2015 at 1:52 AM, Upayavira <u...@odoko.co.uk> wrote:
>> > Is the authentication plugin that expensive?
>> >
>> > I can help by minifying the UI down to a smaller number of CSS/JS/etc
>> > files :-)
>> >
>> > It may be overkill, but it would also give better experience. And isn't
>> > that what most applications do? Check authentication tokens on every
>> > request?
>> >
>> > Upayavira
>> >
>> > On Tue, Nov 10, 2015, at 07:33 PM, Anshum Gupta wrote:
>> >> The reason why we bypass that is so that we don't hit the authentication
>> >> plugin for every request that comes in for static content. I think we
>> >> could
>> >> call the authentication plugin for that but that'd be an overkill. Better
>> >> experience ? yes
>> >>
>> >> On Tue, Nov 10, 2015 at 11:24 AM, Upayavira <u...@odoko.co.uk> wrote:
>> >>
>> >> > Noble,
>> >> >
>> >> > I get that a UI which is open source does not benefit from ACL control -
>> >> > we're not giving away anything that isn't public (other than perhaps
>> >> > info that could be used to identify the version of Solr, or even the
>> >> > fact that it *is* solr).
>> >> >
>> >> > However, from a user experience point of view, requiring credentials to
>> >> > see the UI would be more conventional, and therefore lead to less
>> >> > confusion. Is it possible for us to protect the UI static files, only
>> >> > for the sake of user experience, rather than security?
>> >> >
>> >> > Upayavira
>> >> >
>> >> > On Tue, Nov 10, 2015, at 12:01 PM, Noble Paul wrote:
>> >> > > The admin UI is a bunch of static pages . We don't let the ACL control
>> >> > > static content
>> >> > >
>> >> > > you must blacklist all the core/collection apis and it is pretty much
>> >> > > useless for anyone to access the admin UI (w/o the credentials , of
>> >> > > course)
>> >> > >
>> >> > > On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟 <mabaizh...@126.com> wrote:
>> >> > > > Hi,
>> >> > > >
>> >> > > > After I configure Authentication with Basic Authentication Plugin 
>> >> > > > and
>> >> > Authorization with Rule-Based Authorization Plugin, How can I prevent 
>> >> > the
>> >> > strangers from visiting my solr by browser? For example, if the stranger
>> >> > visit the http://(my host):8983, the browser will pop up a window and
>> >> > says "the server http://(my host):8983 requires a username and
>> >> > password"
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > -
>> >> > > Noble Paul
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Anshum Gupta
>>
>>
>>
>> --
>> -
>> Noble Paul



-- 
-
Noble Paul

Re: Solr Search: Access Control / Role based security

[Noble Paul]

I haven't evaluated manifoldCF for this .
However , my preference would be to have a generic mechanism in built
into Solr to restrict user access to certain docs based on some field
values. Relying on external tools make life complex for users who do
not like it.

Our strategy is

* Provide a pluggable framework so that custom external solutions can
be plugged in
* Provide a standard implementation which does not depend upon any
external solutions

any suggestions are welcome


On Wed, Nov 11, 2015 at 12:07 AM, Susheel Kumar <susheel2...@gmail.com> wrote:
> Thanks everyone for the suggestions.
>
> Hi Noble - Were there any thoughts made on utilizing Apache ManifoldCF
> while developing Authentication/Authorization plugins or anything to add
> there.
>
> Thanks,
> Susheel
>
> On Tue, Nov 10, 2015 at 5:01 AM, Alessandro Benedetti <abenede...@apache.org
>> wrote:
>
>> I've been working for a while with Apache ManifoldCF and Enterprise Search
>> in Solr ( with Document level security) .
>> Basically you can add a couple of extra fields , for example :
>>
>> allow_token : containing all the tokens that can view the document
>> deny_token : containing all the tokens that are denied to view the document
>>
>> Apache ManifoldCF provides an integration that add an additional layer, and
>> is able to combine different data sources permission schemes.
>> The Authority Service endpoint will take in input the user name and return
>> all the allow_token values and deny_token.
>> At this point you can append the related filter queries to your queries and
>> be sure that the user will only see what is supposed to see.
>>
>> It's basically an extension of the strategy you were proposing, role based.
>> Of course keep protected your endpoints and avoid users to put custom fq,
>> or all your document security model would be useless :)
>>
>> Cheers
>>
>>
>> On 9 November 2015 at 21:52, Scott Stults <
>> sstu...@opensourceconnections.com
>> > wrote:
>>
>> > Susheel,
>> >
>> > This is perfectly fine for simple use-cases and has the benefit that the
>> > filterCache will help things stay nice and speedy. Apache ManifoldCF
>> goes a
>> > bit further and ties back to your authentication and authorization
>> > mechanism:
>> >
>> >
>> >
>> http://manifoldcf.apache.org/release/trunk/en_US/concepts.html#ManifoldCF+security+model
>> >
>> >
>> > k/r,
>> > Scott
>> >
>> > On Thu, Nov 5, 2015 at 2:26 PM, Susheel Kumar <susheel2...@gmail.com>
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I have seen couple of use cases / need where we want to restrict result
>> > of
>> > > search based on role of a user.  For e.g.
>> > >
>> > > - if user role is admin, any document from the search result will be
>> > > returned
>> > > - if user role is manager, only documents intended for managers will be
>> > > returned
>> > > - if user role is worker, only documents intended for workers will be
>> > > returned
>> > >
>> > > Typical practise is to tag the documents with the roles (using a
>> > > multi-valued field) during indexing and then during search append
>> filter
>> > > query to restrict result based on roles.
>> > >
>> > > Wondering if there is any other better way out there and if this common
>> > > requirement should be added as a Solr feature/plugin.
>> > >
>> > > The current security plugins are more towards making Solr
>> apis/resources
>> > > secure not towards securing/controlling data during search.
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
>> > >
>> > >
>> > > Please share your thoughts.
>> > >
>> > > Thanks,
>> > > Susheel
>> > >
>> >
>> >
>> >
>> > --
>> > Scott Stults | Founder & Solutions Architect | OpenSource Connections,
>> LLC
>> > | 434.409.2780
>> > http://www.opensourceconnections.com
>> >
>>
>>
>>
>> --
>> --
>>
>> Benedetti Alessandro
>> Visiting card : http://about.me/alessandro_benedetti
>>
>> "Tyger, tyger burning bright
>> In the forests of the night,
>> What immortal hand or eye
>> Could frame thy fearful symmetry?"
>>
>> William Blake - Songs of Experience -1794 England
>>



-- 
-
Noble Paul

Re: Security Problems

[Noble Paul]

The authentication plugin is not expensive if you are talking in the
context of admin UI. After all it is used not like 100s of requests
per second.

The simplest solution would be

provide a well known permission name called "admin-ui"

ensure that every admin page load makes a call to some resource say
"/admin/security-check"

Then we can just protect that .

The only concern thatI have is the false sense of security it would
give to the user

But, that is a different point altogether

On Wed, Nov 11, 2015 at 1:52 AM, Upayavira <u...@odoko.co.uk> wrote:
> Is the authentication plugin that expensive?
>
> I can help by minifying the UI down to a smaller number of CSS/JS/etc
> files :-)
>
> It may be overkill, but it would also give better experience. And isn't
> that what most applications do? Check authentication tokens on every
> request?
>
> Upayavira
>
> On Tue, Nov 10, 2015, at 07:33 PM, Anshum Gupta wrote:
>> The reason why we bypass that is so that we don't hit the authentication
>> plugin for every request that comes in for static content. I think we
>> could
>> call the authentication plugin for that but that'd be an overkill. Better
>> experience ? yes
>>
>> On Tue, Nov 10, 2015 at 11:24 AM, Upayavira <u...@odoko.co.uk> wrote:
>>
>> > Noble,
>> >
>> > I get that a UI which is open source does not benefit from ACL control -
>> > we're not giving away anything that isn't public (other than perhaps
>> > info that could be used to identify the version of Solr, or even the
>> > fact that it *is* solr).
>> >
>> > However, from a user experience point of view, requiring credentials to
>> > see the UI would be more conventional, and therefore lead to less
>> > confusion. Is it possible for us to protect the UI static files, only
>> > for the sake of user experience, rather than security?
>> >
>> > Upayavira
>> >
>> > On Tue, Nov 10, 2015, at 12:01 PM, Noble Paul wrote:
>> > > The admin UI is a bunch of static pages . We don't let the ACL control
>> > > static content
>> > >
>> > > you must blacklist all the core/collection apis and it is pretty much
>> > > useless for anyone to access the admin UI (w/o the credentials , of
>> > > course)
>> > >
>> > > On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟 <mabaizh...@126.com> wrote:
>> > > > Hi,
>> > > >
>> > > > After I configure Authentication with Basic Authentication Plugin and
>> > Authorization with Rule-Based Authorization Plugin, How can I prevent the
>> > strangers from visiting my solr by browser? For example, if the stranger
>> > visit the http://(my host):8983, the browser will pop up a window and
>> > says "the server http://(my host):8983 requires a username and
>> > password"
>> > >
>> > >
>> > >
>> > > --
>> > > -
>> > > Noble Paul
>> >
>>
>>
>>
>> --
>> Anshum Gupta



-- 
-
Noble Paul

Re: Security Problems

[Noble Paul]

The admin UI is a bunch of static pages . We don't let the ACL control
static content

you must blacklist all the core/collection apis and it is pretty much
useless for anyone to access the admin UI (w/o the credentials , of
course)

On Tue, Nov 10, 2015 at 7:08 AM, 马柏樟 <mabaizh...@126.com> wrote:
> Hi,
>
> After I configure Authentication with Basic Authentication Plugin and 
> Authorization with Rule-Based Authorization Plugin, How can I prevent the 
> strangers from visiting my solr by browser? For example, if the stranger 
> visit the http://(my host):8983, the browser will pop up a window and says 
> "the server http://(my host):8983 requires a username and password"



-- 
-----
Noble Paul

[ANNOUNCE] Apache Lucene 5.3.1 released

[Noble Paul]

24 September 2015, Apache Solr™ 5.3.1 available


The Lucene PMC is pleased to announce the release of Apache Solr 5.3.1


Solr is the popular, blazing fast, open source NoSQL search platform

from the Apache Lucene project. Its major features include powerful

full-text search, hit highlighting, faceted search, dynamic

clustering, database integration, rich document (e.g., Word, PDF)

handling, and geospatial search. Solr is highly scalable, providing

fault tolerant distributed search and indexing, and powers the search

and navigation features of many of the world's largest internet sites.


This release contains various bug fixes and optimizations since the
5.3.0 release. The release is available for immediate download at:


  http://lucene.apache.org/solr/mirrors-solr-latest-redir.html


Please read CHANGES.txt for a full list of new features and changes:


  https://lucene.apache.org/solr/5_3_1/changes/Changes.html


Solr 5.3.1 includes these bug fixes.


 * security.json is not loaded on server start

 * RuleBasedAuthorization plugin does not respect the
collection-admin-edit permission

 * Fix VelocityResponseWriter template encoding issue. Templates must
be UTF-8 encoded

 * SimplePostTool (also bin/post) -filetypes "*" now works properly in
'web' mode

 * example/files update-script.js to be Java 7 and 8 compatible.

 * SolrJ could not make requests to handlers with '/admin/' prefix

 * Use of timeAllowed can cause incomplete filters to be cached and
incorrect results to be returned on subsequent requests

 * VelocityResponseWriter's $resource.get(key,baseName,locale) to use
specified locale.

 * Fix the exclusion filter so that collections that start with js,
css, img, tpl can be accessed.

 * Resolve XSS issue in Admin UI stats page


Known issues:

 * On Windows, bin/solr.cmd script fails to start correctly when using
relative path with -s parameter. Use absolute path as a workaround .
https://issues.apache.org/jira/browse/SOLR-8073


See the CHANGES.txt file included with the release for a full list of
changes and further details.


Please report any feedback to the mailing lists
(http://lucene.apache.org/solr/discussion.html)


Note: The Apache Software Foundation uses an extensive mirroring
network for distributing releases. It is possible that the mirror you
are using may not have replicated the release yet. If that is the
case, please try another mirror. This also goes for Maven access.

Noble Paul
on behalf of Lucene PMC

[ANNOUNCE] Apache Solr 5.3.1 released

[Noble Paul]

24 September 2015, Apache Solr™ 5.3.1 available


The Lucene PMC is pleased to announce the release of Apache Solr 5.3.1


Solr is the popular, blazing fast, open source NoSQL search platform

from the Apache Lucene project. Its major features include powerful

full-text search, hit highlighting, faceted search, dynamic

clustering, database integration, rich document (e.g., Word, PDF)

handling, and geospatial search. Solr is highly scalable, providing

fault tolerant distributed search and indexing, and powers the search

and navigation features of many of the world's largest internet sites.


This release contains various bug fixes and optimizations since the
5.3.0 release. The release is available for immediate download at:


  http://lucene.apache.org/solr/mirrors-solr-latest-redir.html


Please read CHANGES.txt for a full list of new features and changes:


  https://lucene.apache.org/solr/5_3_1/changes/Changes.html


Solr 5.3.1 includes these bug fixes.


 * security.json is not loaded on server start

 * RuleBasedAuthorization plugin does not respect the
collection-admin-edit permission

 * Fix VelocityResponseWriter template encoding issue. Templates must
be UTF-8 encoded

 * SimplePostTool (also bin/post) -filetypes "*" now works properly in
'web' mode

 * example/files update-script.js to be Java 7 and 8 compatible.

 * SolrJ could not make requests to handlers with '/admin/' prefix

 * Use of timeAllowed can cause incomplete filters to be cached and
incorrect results to be returned on subsequent requests

 * VelocityResponseWriter's $resource.get(key,baseName,locale) to use
specified locale.

 * Fix the exclusion filter so that collections that start with js,
css, img, tpl can be accessed.

 * Resolve XSS issue in Admin UI stats page


Known issues:

 * On Windows, bin/solr.cmd script fails to start correctly when using
relative path with -s parameter. Use absolute path as a workaround .
https://issues.apache.org/jira/browse/SOLR-8073


See the CHANGES.txt file included with the release for a full list of
changes and further details.

Noble Paul
on behalf of Lucene PMC

Re: [ANNOUNCE] Apache Lucene 5.3.1 released

[Noble Paul]

Wrong title

On Thu, Sep 24, 2015 at 10:55 PM, Noble Paul <noble.p...@gmail.com> wrote:
> 24 September 2015, Apache Solr™ 5.3.1 available
>
>
> The Lucene PMC is pleased to announce the release of Apache Solr 5.3.1
>
>
> Solr is the popular, blazing fast, open source NoSQL search platform
>
> from the Apache Lucene project. Its major features include powerful
>
> full-text search, hit highlighting, faceted search, dynamic
>
> clustering, database integration, rich document (e.g., Word, PDF)
>
> handling, and geospatial search. Solr is highly scalable, providing
>
> fault tolerant distributed search and indexing, and powers the search
>
> and navigation features of many of the world's largest internet sites.
>
>
> This release contains various bug fixes and optimizations since the
> 5.3.0 release. The release is available for immediate download at:
>
>
>   http://lucene.apache.org/solr/mirrors-solr-latest-redir.html
>
>
> Please read CHANGES.txt for a full list of new features and changes:
>
>
>   https://lucene.apache.org/solr/5_3_1/changes/Changes.html
>
>
> Solr 5.3.1 includes these bug fixes.
>
>
>  * security.json is not loaded on server start
>
>  * RuleBasedAuthorization plugin does not respect the
> collection-admin-edit permission
>
>  * Fix VelocityResponseWriter template encoding issue. Templates must
> be UTF-8 encoded
>
>  * SimplePostTool (also bin/post) -filetypes "*" now works properly in
> 'web' mode
>
>  * example/files update-script.js to be Java 7 and 8 compatible.
>
>  * SolrJ could not make requests to handlers with '/admin/' prefix
>
>  * Use of timeAllowed can cause incomplete filters to be cached and
> incorrect results to be returned on subsequent requests
>
>  * VelocityResponseWriter's $resource.get(key,baseName,locale) to use
> specified locale.
>
>  * Fix the exclusion filter so that collections that start with js,
> css, img, tpl can be accessed.
>
>  * Resolve XSS issue in Admin UI stats page
>
>
> Known issues:
>
>  * On Windows, bin/solr.cmd script fails to start correctly when using
> relative path with -s parameter. Use absolute path as a workaround .
> https://issues.apache.org/jira/browse/SOLR-8073
>
>
> See the CHANGES.txt file included with the release for a full list of
> changes and further details.
>
>
> Please report any feedback to the mailing lists
> (http://lucene.apache.org/solr/discussion.html)
>
>
> Note: The Apache Software Foundation uses an extensive mirroring
> network for distributing releases. It is possible that the mirror you
> are using may not have replicated the release yet. If that is the
> case, please try another mirror. This also goes for Maven access.
>
> Noble Paul
> on behalf of Lucene PMC



-- 
-
Noble Paul

Re: Solr authentication - Error 401 Unauthorized

[Noble Paul]

It is not that solr is over protected, it is just that the clients,
SolrJ as well as bin/solr are not provided with basic auth
capabilities.

 I have opened a ticket to track this
https://issues.apache.org/jira/browse/SOLR-8048

On Sat, Sep 12, 2015 at 7:14 PM, Dan Davis <dansm...@gmail.com> wrote:
> Noble,
>
> You should also look at this if it is intended to be more than an internal
> API.   Using the minor protections I added to test SOLR-8000, I was able to
> reproduce a problem very like this:
>
> bin/solr healthcheck -z localhost:2181 -c mycollection
>
> Since Solr /select is protected...
>
> On Sat, Sep 12, 2015 at 9:40 AM, Dan Davis <dansm...@gmail.com> wrote:
>
>> It seems that you have secured Solr so thoroughly that you cannot now run
>> bin/solr status!
>>
>> bin/solr has no arguments as yet for providing a username/password - as a
>> mostly user like you I'm not sure of the roadmap.
>>
>> I think you should relax those restrictions a bit and try again.
>>
>> On Fri, Sep 11, 2015 at 5:06 AM, Merlin Morgenstern <
>> merlin.morgenst...@gmail.com> wrote:
>>
>>> I have secured solr cloud via basic authentication.
>>>
>>> Now I am having difficulties creating cores and getting status
>>> information.
>>> Solr keeps telling me that the request is unothorized. However, I have
>>> access to the admin UI after login.
>>>
>>> How do I configure solr to use the basic authentication credentials?
>>>
>>> This is the error message:
>>>
>>> /opt/solr-5.3.0/bin/solr status
>>>
>>> Found 1 Solr nodes:
>>>
>>> Solr process 31114 running on port 8983
>>>
>>> ERROR: Failed to get system information from http://localhost:8983/solr
>>> due
>>> to: org.apache.http.client.ClientProtocolException: Expected JSON response
>>> from server but received: 
>>>
>>> 
>>>
>>> 
>>>
>>> Error 401 Unauthorized
>>>
>>> 
>>>
>>> HTTP ERROR 401
>>>
>>> Problem accessing /solr/admin/info/system. Reason:
>>>
>>> UnauthorizedPowered by
>>> Jetty://
>>>
>>>
>>> 
>>>
>>> 
>>>
>>
>>



-- 
-
Noble Paul

Re: How to secure Admin UI with Basic Auth in Solr 5.3.x

[Noble Paul]

There were some bugs with the 5.3.0 release and 5.3.1 is in the
process of getting released.

try out the option #2 with the RC here

https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-5.3.1-RC1-rev1702389/solr/



On Fri, Sep 11, 2015 at 5:16 PM, Merlin Morgenstern
<merlin.morgenst...@gmail.com> wrote:
> OK, I downgraded to solr 5.2.x
>
> Unfortunatelly still no luck. I followed 2 aproaches:
>
> 1. Secure it the old fashioned way like described here:
> http://stackoverflow.com/questions/28043957/how-to-set-apache-solr-admin-password
>
> 2. Using the Basic Authentication Plugin like described here:
> http://lucidworks.com/blog/securing-solr-basic-auth-permission-rules/
>
> Both aproaches created unsolved problems.
>
> While following option 1, I was able to secure the Admin UI with basic
> authentication, but no longer able to access my application despite the
> fact that it was working on solr 3.x with the same type of authentication
> procedure and credentials.
>
> While following option 2, I was stuck right after uploading the
> security.json file to the zookeeper ensemble. The described behaviour to curl
> http://localhost:8983/solr/admin/authentication responded with a 404 not
> found and then solr could not connect to zookeeper. I had to remove that
> file from zookeeper and restart all solr nodes.
>
> Please could someone lead me the way on how to secure the Admin UI and
> password protect solr cloud? I have a perfectly running system with solr
> 3.x and one core and now taking it to solr cloud 5.2.x into production
> seems to be stoped by simple authorization problems.
>
> Thank you in advane for any help.
>
>
>
> 2015-09-10 20:42 GMT+02:00 Noble Paul <noble.p...@gmail.com>:
>
>> Check this https://cwiki.apache.org/confluence/display/solr/Securing+Solr
>>
>> There a couple of bugs in 5.3.o and a bug fix release is coming up
>> over the next few days.
>>
>> We don't provide any specific means to restrict access to admin UI
>> itself. However we let users specify fine grained ACLs on various
>> operations such collection-admin-edit, read etc
>>
>> On Wed, Sep 9, 2015 at 2:35 PM, Merlin Morgenstern
>> <merlin.morgenst...@gmail.com> wrote:
>> > I just installed solr cloud 5.3.x and found that the way to secure the
>> amin
>> > ui has changed. Aparently there is a new plugin which does role based
>> > authentification and all info on how to secure the admin UI found on the
>> > net is outdated.
>> >
>> > I do not need role based authentification but just simply want to put
>> basic
>> > authentification to the Admin UI.
>> >
>> > How do I configure solr cloud 5.3.x in order to restrict access to the
>> > Admin UI via Basic Authentification?
>> >
>> > Thank you for any help
>>
>>
>>
>> --
>> -
>> Noble Paul
>>



-- 
-
Noble Paul

Re: How to secure Admin UI with Basic Auth in Solr 5.3.x

[Noble Paul]

Check this https://cwiki.apache.org/confluence/display/solr/Securing+Solr

There a couple of bugs in 5.3.o and a bug fix release is coming up
over the next few days.

We don't provide any specific means to restrict access to admin UI
itself. However we let users specify fine grained ACLs on various
operations such collection-admin-edit, read etc

On Wed, Sep 9, 2015 at 2:35 PM, Merlin Morgenstern
<merlin.morgenst...@gmail.com> wrote:
> I just installed solr cloud 5.3.x and found that the way to secure the amin
> ui has changed. Aparently there is a new plugin which does role based
> authentification and all info on how to secure the admin UI found on the
> net is outdated.
>
> I do not need role based authentification but just simply want to put basic
> authentification to the Admin UI.
>
> How do I configure solr cloud 5.3.x in order to restrict access to the
> Admin UI via Basic Authentification?
>
> Thank you for any help



-- 
-----
Noble Paul

Re: Issue Using Solr 5.3 Authentication and Authorization Plugins

[Noble Paul]

There are no download links for 5.3.x branch  till we do a bug fix release

If you wish to download the trunk nightly (which is not same as 5.3.0)
check here 
https://builds.apache.org/job/Solr-Artifacts-trunk/lastSuccessfulBuild/artifact/solr/package/

If you wish to get the binaries for 5.3 branch you will have to make it
(you will need to install svn and ant)

Here are the steps

svn checkout 
http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_5_3/
cd lucene_solr_5_3/solr
ant server



On Fri, Sep 4, 2015 at 4:11 PM, davidphilip cherian
<davidphilipcher...@gmail.com> wrote:
> Hi Kevin/Noble,
>
> What is the download link to take the latest? What are the steps to compile
> it, test and use?
> We also have a use case to have this feature in solr too. Therefore, wanted
> to test and above info would help a lot to get started.
>
> Thanks.
>
>
> On Fri, Sep 4, 2015 at 1:45 PM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
>
>> Thanks, I downloaded the source and compiled it and replaced the jar file
>> in the dist and solr-webapp’s WEB-INF/lib directory.  It does seem to be
>> protecting the Collections API reload command now as long as I upload the
>> security.json after startup of the Solr instances.  If I shutdown and bring
>> the instances back up, the security is no longer in place and I have to
>> upload the security.json again for it to take effect.
>>
>> - Kevin
>>
>> > On Sep 3, 2015, at 10:29 PM, Noble Paul <noble.p...@gmail.com> wrote:
>> >
>> > Both these are committed. If you could test with the latest 5.3 branch
>> > it would be helpful
>> >
>> > On Wed, Sep 2, 2015 at 5:11 PM, Noble Paul <noble.p...@gmail.com> wrote:
>> >> I opened a ticket for the same
>> >> https://issues.apache.org/jira/browse/SOLR-8004
>> >>
>> >> On Wed, Sep 2, 2015 at 1:36 PM, Kevin Lee <kgle...@yahoo.com.invalid>
>> wrote:
>> >>> I’ve found that completely exiting Chrome or Firefox and opening it
>> back up re-prompts for credentials when they are required.  It was
>> re-prompting with the /browse path where authentication was working each
>> time I completely exited and started the browser again, however it won’t
>> re-prompt unless you exit completely and close all running instances so I
>> closed all instances each time to test.
>> >>>
>> >>> However, to make sure I ran it via the command line via curl as
>> suggested and it still does not give any authentication error when trying
>> to issue the command via curl.  I get a success response from all the Solr
>> instances that the reload was successful.
>> >>>
>> >>> Not sure why the pre-canned permissions aren’t working, but the one to
>> the request handler at the /browse path is.
>> >>>
>> >>>
>> >>>> On Sep 1, 2015, at 11:03 PM, Noble Paul <noble.p...@gmail.com> wrote:
>> >>>>
>> >>>> " However, after uploading the new security.json and restarting the
>> >>>> web browser,"
>> >>>>
>> >>>> The browser remembers your login , So it is unlikely to prompt for the
>> >>>> credentials again.
>> >>>>
>> >>>> Why don't you try the RELOAD operation using command line (curl) ?
>> >>>>
>> >>>> On Tue, Sep 1, 2015 at 10:31 PM, Kevin Lee <kgle...@yahoo.com.invalid>
>> wrote:
>> >>>>> The restart issues aside, I’m trying to lockdown usage of the
>> Collections API, but that also does not seem to be working either.
>> >>>>>
>> >>>>> Here is my security.json.  I’m using the “collection-admin-edit”
>> permission and assigning it to the “adminRole”.  However, after uploading
>> the new security.json and restarting the web browser, it doesn’t seem to be
>> requiring credentials when calling the RELOAD action on the Collections
>> API.  The only thing that seems to work is the custom permission “browse”
>> which is requiring authentication before allowing me to pull up the page.
>> Am I using the permissions correctly for the RuleBasedAuthorizationPlugin?
>> >>>>>
>> >>>>> {
>> >>>>>   "authentication":{
>> >>>>>  "class":"solr.BasicAuthPlugin",
>> >>>>>  "credentials": {
>> >>>>>   "admin”:” ",
>> >>>>>   "use

Re: Issue Using Solr 5.3 Authentication and Authorization Plugins

[Noble Paul]

Both these are committed. If you could test with the latest 5.3 branch
it would be helpful

On Wed, Sep 2, 2015 at 5:11 PM, Noble Paul <noble.p...@gmail.com> wrote:
> I opened a ticket for the same
>  https://issues.apache.org/jira/browse/SOLR-8004
>
> On Wed, Sep 2, 2015 at 1:36 PM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
>> I’ve found that completely exiting Chrome or Firefox and opening it back up 
>> re-prompts for credentials when they are required.  It was re-prompting with 
>> the /browse path where authentication was working each time I completely 
>> exited and started the browser again, however it won’t re-prompt unless you 
>> exit completely and close all running instances so I closed all instances 
>> each time to test.
>>
>> However, to make sure I ran it via the command line via curl as suggested 
>> and it still does not give any authentication error when trying to issue the 
>> command via curl.  I get a success response from all the Solr instances that 
>> the reload was successful.
>>
>> Not sure why the pre-canned permissions aren’t working, but the one to the 
>> request handler at the /browse path is.
>>
>>
>>> On Sep 1, 2015, at 11:03 PM, Noble Paul <noble.p...@gmail.com> wrote:
>>>
>>> " However, after uploading the new security.json and restarting the
>>> web browser,"
>>>
>>> The browser remembers your login , So it is unlikely to prompt for the
>>> credentials again.
>>>
>>> Why don't you try the RELOAD operation using command line (curl) ?
>>>
>>> On Tue, Sep 1, 2015 at 10:31 PM, Kevin Lee <kgle...@yahoo.com.invalid> 
>>> wrote:
>>>> The restart issues aside, I’m trying to lockdown usage of the Collections 
>>>> API, but that also does not seem to be working either.
>>>>
>>>> Here is my security.json.  I’m using the “collection-admin-edit” 
>>>> permission and assigning it to the “adminRole”.  However, after uploading 
>>>> the new security.json and restarting the web browser, it doesn’t seem to 
>>>> be requiring credentials when calling the RELOAD action on the Collections 
>>>> API.  The only thing that seems to work is the custom permission “browse” 
>>>> which is requiring authentication before allowing me to pull up the page.  
>>>> Am I using the permissions correctly for the RuleBasedAuthorizationPlugin?
>>>>
>>>> {
>>>>"authentication":{
>>>>   "class":"solr.BasicAuthPlugin",
>>>>   "credentials": {
>>>>"admin”:” ",
>>>>"user": ” "
>>>>}
>>>>},
>>>>"authorization":{
>>>>   "class":"solr.RuleBasedAuthorizationPlugin",
>>>>   "permissions": [
>>>>{
>>>>"name":"security-edit",
>>>>"role":"adminRole"
>>>>},
>>>>{
>>>>"name":"collection-admin-edit”,
>>>>"role":"adminRole"
>>>>},
>>>>{
>>>>"name":"browse",
>>>>"collection": "inventory",
>>>>"path": "/browse",
>>>>"role":"browseRole"
>>>>}
>>>>],
>>>>   "user-role": {
>>>>"admin": [
>>>>    "adminRole",
>>>>"browseRole"
>>>>],
>>>>"user": [
>>>>"browseRole"
>>>>]
>>>>}
>>>>}
>>>> }
>>>>
>>>> Also tried adding the permission using the Authorization API, but no 
>>>> effect, still isn’t protecting the Collections API fr

Re: Issue Using Solr 5.3 Authentication and Authorization Plugins

[Noble Paul]

" However, after uploading the new security.json and restarting the
web browser,"

The browser remembers your login , So it is unlikely to prompt for the
credentials again.

Why don't you try the RELOAD operation using command line (curl) ?

On Tue, Sep 1, 2015 at 10:31 PM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
> The restart issues aside, I’m trying to lockdown usage of the Collections 
> API, but that also does not seem to be working either.
>
> Here is my security.json.  I’m using the “collection-admin-edit” permission 
> and assigning it to the “adminRole”.  However, after uploading the new 
> security.json and restarting the web browser, it doesn’t seem to be requiring 
> credentials when calling the RELOAD action on the Collections API.  The only 
> thing that seems to work is the custom permission “browse” which is requiring 
> authentication before allowing me to pull up the page.  Am I using the 
> permissions correctly for the RuleBasedAuthorizationPlugin?
>
> {
> "authentication":{
>"class":"solr.BasicAuthPlugin",
>"credentials": {
> "admin”:” ",
> "user": ” "
> }
> },
> "authorization":{
>"class":"solr.RuleBasedAuthorizationPlugin",
>"permissions": [
> {
> "name":"security-edit",
> "role":"adminRole"
> },
> {
> "name":"collection-admin-edit”,
> "role":"adminRole"
> },
> {
> "name":"browse",
> "collection": "inventory",
> "path": "/browse",
> "role":"browseRole"
> }
> ],
>"user-role": {
> "admin": [
> "adminRole",
> "browseRole"
> ],
> "user": [
> "browseRole"
> ]
> }
> }
> }
>
> Also tried adding the permission using the Authorization API, but no effect, 
> still isn’t protecting the Collections API from being invoked without a 
> username password.  I do see in the Solr logs that it sees the updates 
> because it outputs the messages “Updating /security.json …”, “Security node 
> changed”, “Initializing authorization plugin: 
> solr.RuleBasedAuthorizationPlugin” and “Authentication plugin class obtained 
> from ZK: solr.BasicAuthPlugin”.
>
> Thanks,
> Kevin
>
>> On Sep 1, 2015, at 12:31 AM, Noble Paul <noble.p...@gmail.com> wrote:
>>
>> I'm investigating why restarts or first time start does not read the
>> security.json
>>
>> On Tue, Sep 1, 2015 at 1:00 PM, Noble Paul <noble.p...@gmail.com> wrote:
>>> I removed that statement
>>>
>>> "If activating the authorization plugin doesn't protect the admin ui,
>>> how does one protect access to it?"
>>>
>>> One does not need to protect the admin UI. You only need to protect
>>> the relevant API calls . I mean it's OK to not protect the CSS and
>>> HTML stuff.  But if you perform an action to create a core or do a
>>> query through admin UI , it automatically will prompt you for
>>> credentials (if those APIs are protected)
>>>
>>> On Tue, Sep 1, 2015 at 12:41 PM, Kevin Lee <kgle...@yahoo.com.invalid> 
>>> wrote:
>>>> Thanks for the clarification!
>>>>
>>>> So is the wiki page incorrect at
>>>> https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin
>>>>  which says that the admin ui will require authentication once the 
>>>> authorization plugin is activated?
>>>>
>>>> "An authorization plugin is also available to configure Solr with 
>>>> permissions to perform various activities in the system. Once activated, 
>>>> access to the Solr Admin UI and all requests will need to be authenticated 
>>>> and users will be required to have

Re: Issue Using Solr 5.3 Authentication and Authorization Plugins

[Noble Paul]

I opened a ticket for the same
 https://issues.apache.org/jira/browse/SOLR-8004

On Wed, Sep 2, 2015 at 1:36 PM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
> I’ve found that completely exiting Chrome or Firefox and opening it back up 
> re-prompts for credentials when they are required.  It was re-prompting with 
> the /browse path where authentication was working each time I completely 
> exited and started the browser again, however it won’t re-prompt unless you 
> exit completely and close all running instances so I closed all instances 
> each time to test.
>
> However, to make sure I ran it via the command line via curl as suggested and 
> it still does not give any authentication error when trying to issue the 
> command via curl.  I get a success response from all the Solr instances that 
> the reload was successful.
>
> Not sure why the pre-canned permissions aren’t working, but the one to the 
> request handler at the /browse path is.
>
>
>> On Sep 1, 2015, at 11:03 PM, Noble Paul <noble.p...@gmail.com> wrote:
>>
>> " However, after uploading the new security.json and restarting the
>> web browser,"
>>
>> The browser remembers your login , So it is unlikely to prompt for the
>> credentials again.
>>
>> Why don't you try the RELOAD operation using command line (curl) ?
>>
>> On Tue, Sep 1, 2015 at 10:31 PM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
>>> The restart issues aside, I’m trying to lockdown usage of the Collections 
>>> API, but that also does not seem to be working either.
>>>
>>> Here is my security.json.  I’m using the “collection-admin-edit” permission 
>>> and assigning it to the “adminRole”.  However, after uploading the new 
>>> security.json and restarting the web browser, it doesn’t seem to be 
>>> requiring credentials when calling the RELOAD action on the Collections 
>>> API.  The only thing that seems to work is the custom permission “browse” 
>>> which is requiring authentication before allowing me to pull up the page.  
>>> Am I using the permissions correctly for the RuleBasedAuthorizationPlugin?
>>>
>>> {
>>>"authentication":{
>>>   "class":"solr.BasicAuthPlugin",
>>>   "credentials": {
>>>"admin”:” ",
>>>"user": ” "
>>>}
>>>},
>>>"authorization":{
>>>   "class":"solr.RuleBasedAuthorizationPlugin",
>>>   "permissions": [
>>>{
>>>"name":"security-edit",
>>>"role":"adminRole"
>>>},
>>>{
>>>"name":"collection-admin-edit”,
>>>"role":"adminRole"
>>>},
>>>{
>>>"name":"browse",
>>>"collection": "inventory",
>>>"path": "/browse",
>>>"role":"browseRole"
>>>}
>>>],
>>>   "user-role": {
>>>"admin": [
>>>"adminRole",
>>>"browseRole"
>>>],
>>>"user": [
>>>    "browseRole"
>>>]
>>>}
>>>}
>>> }
>>>
>>> Also tried adding the permission using the Authorization API, but no 
>>> effect, still isn’t protecting the Collections API from being invoked 
>>> without a username password.  I do see in the Solr logs that it sees the 
>>> updates because it outputs the messages “Updating /security.json …”, 
>>> “Security node changed”, “Initializing authorization plugin: 
>>> solr.RuleBasedAuthorizationPlugin” and “Authentication plugin class 
>>> obtained from ZK: solr.BasicAuthPlugin”.
>>>
>>> Thanks,
>>> Kevin
>>>
>>>> On Sep 1, 2015, at 12:31 AM, Noble 

Re: Issue Using Solr 5.3 Authentication and Authorization Plugins

[Noble Paul]

Admin UI is not protected by any of these permissions. Only if you try
to perform a protected operation , it asks for a password.

I'll investigate the restart problem and report my  findings

On Tue, Sep 1, 2015 at 3:10 AM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
> Anyone else running into any issues trying to get the authentication and 
> authorization plugins in 5.3 working?
>
>> On Aug 29, 2015, at 2:30 AM, Kevin Lee <kgle...@yahoo.com.INVALID> wrote:
>>
>> Hi,
>>
>> I’m trying to use the new basic auth plugin for Solr 5.3 and it doesn’t seem 
>> to be working quite right.  Not sure if I’m missing steps or there is a bug. 
>>  I am able to get it to protect access to a URL under a collection, but am 
>> unable to get it to secure access to the Admin UI.  In addition, after 
>> stopping the Solr and Zookeeper instances, the security.json is still in 
>> Zookeeper, however Solr is allowing access to everything again like the 
>> security configuration isn’t in place.
>>
>> Contents of security.json taken from wiki page, but edited to produce valid 
>> JSON.  Had to move comma after 3rd from last “}” up to just after the last 
>> “]”.
>>
>> {
>> "authentication":{
>>   "class":"solr.BasicAuthPlugin",
>>   "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= 
>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
>> },
>> "authorization":{
>>   "class":"solr.RuleBasedAuthorizationPlugin",
>>   "permissions":[{"name":"security-edit",
>>  "role":"admin"}],
>>   "user-role":{"solr":"admin"}
>> }}
>>
>> Here are the steps I followed:
>>
>> Upload security.json to zookeeper
>> ./zkcli.sh -z localhost:2181,localhost:2182,localhost:2183 -cmd putfile 
>> /security.json ~/solr/security.json
>>
>> Use zkCli.sh from Zookeeper to ensure the security.json is in Zookeeper at 
>> /security.json.  It is there and looks like what was originally uploaded.
>>
>> Start Solr Instances
>>
>> Attempt to create a permission, however get the following error:
>> {
>>  "responseHeader":{
>>"status":400,
>>"QTime":0},
>>  "error":{
>>"msg":"No authorization plugin configured",
>>"code":400}}
>>
>> Upload security.json again.
>> ./zkcli.sh -z localhost:2181,localhost:2182,localhost:2183 -cmd putfile 
>> /security.json ~/solr/security.json
>>
>> Issue the following to try to create the permission again and this time it’s 
>> successful.
>> // Create a permission for mysearch endpoint
>>curl --user solr:SolrRocks -H 'Content-type:application/json' -d 
>> '{"set-permission": {"name":"mycollection-search","collection": 
>> “mycollection","path":”/mysearch","role": "search-user"}}' 
>> http://localhost:8983/solr/admin/authorization
>>
>>{
>>  "responseHeader":{
>>"status":0,
>>"QTime":7}}
>>
>> Issue the following commands to add users
>> curl --user solr:SolrRocks http://localhost:8983/solr/admin/authentication 
>> -H 'Content-type:application/json' -d '{"set-user": {"admin" : “password" }}’
>> curl --user solr:SolrRocks http://localhost:8983/solr/admin/authentication 
>> -H 'Content-type:application/json' -d '{"set-user": {"user" : “password" }}'
>>
>> Issue the following command to add permission to users
>> curl -u solr:SolrRocks -H 'Content-type:application/json' -d '{ 
>> "set-user-role" : {"admin": ["search-user", "admin"]}}' 
>> http://localhost:8983/solr/admin/authorization
>> curl -u solr:SolrRocks -H 'Content-type:application/json' -d '{ 
>> "set-user-role" : {"user": ["search-user"]}}' 
>> http://localhost:8983/solr/admin/authorization
>>
>> After executing the above, access to /mysearch is protected until I restart 
>> the Solr and Zookeeper instances.  However, the admin UI is never protected 
>> like the Wiki page says it should be once activated.
>>
>> https://cwiki.apache.org/confluence/display/solr/Rule-Based+Authorization+Plugin
>>  
>> <https://cwiki.apache.org/confluence/display/solr/Rule-Based+Authorization+Plugin>
>>
>> Why does the authentication and authorization plugin not stay activated 
>> after restart and why is the Admin UI never protected?  Am I missing any 
>> steps?
>>
>> Thanks,
>> Kevin



-- 
-
Noble Paul

Re: Issue Using Solr 5.3 Authentication and Authorization Plugins

[Noble Paul]

I removed that statement

"If activating the authorization plugin doesn't protect the admin ui,
how does one protect access to it?"

One does not need to protect the admin UI. You only need to protect
the relevant API calls . I mean it's OK to not protect the CSS and
HTML stuff.  But if you perform an action to create a core or do a
query through admin UI , it automatically will prompt you for
credentials (if those APIs are protected)

On Tue, Sep 1, 2015 at 12:41 PM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
> Thanks for the clarification!
>
> So is the wiki page incorrect at
> https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin 
> which says that the admin ui will require authentication once the 
> authorization plugin is activated?
>
> "An authorization plugin is also available to configure Solr with permissions 
> to perform various activities in the system. Once activated, access to the 
> Solr Admin UI and all requests will need to be authenticated and users will 
> be required to have the proper authorization for all requests, including 
> using the Admin UI and making any API calls."
>
> If activating the authorization plugin doesn't protect the admin ui, how does 
> one protect access to it?
>
> Also, the issue I'm having is not just at restart.  According to the docs 
> security.json should be uploaded to Zookeeper before starting any of the Solr 
> instances.  However, I tried to upload security.json before starting any of 
> the Solr instances, but it would not pick up the security config until after 
> the Solr instances are already running and then uploading the security.json 
> again.  I can see in the logs at startup that the Solr instances don't see 
> any plugin enabled even though security.json is already in zookeeper and then 
> after they are started and the security.json is uploaded again I see it 
> reconfigure to use the plugin.
>
> Thanks,
> Kevin
>
>> On Aug 31, 2015, at 11:22 PM, Noble Paul <noble.p...@gmail.com> wrote:
>>
>> Admin UI is not protected by any of these permissions. Only if you try
>> to perform a protected operation , it asks for a password.
>>
>> I'll investigate the restart problem and report my  findings
>>
>>> On Tue, Sep 1, 2015 at 3:10 AM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
>>> Anyone else running into any issues trying to get the authentication and 
>>> authorization plugins in 5.3 working?
>>>
>>>> On Aug 29, 2015, at 2:30 AM, Kevin Lee <kgle...@yahoo.com.INVALID> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I’m trying to use the new basic auth plugin for Solr 5.3 and it doesn’t 
>>>> seem to be working quite right.  Not sure if I’m missing steps or there is 
>>>> a bug.  I am able to get it to protect access to a URL under a collection, 
>>>> but am unable to get it to secure access to the Admin UI.  In addition, 
>>>> after stopping the Solr and Zookeeper instances, the security.json is 
>>>> still in Zookeeper, however Solr is allowing access to everything again 
>>>> like the security configuration isn’t in place.
>>>>
>>>> Contents of security.json taken from wiki page, but edited to produce 
>>>> valid JSON.  Had to move comma after 3rd from last “}” up to just after 
>>>> the last “]”.
>>>>
>>>> {
>>>> "authentication":{
>>>> "class":"solr.BasicAuthPlugin",
>>>> "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= 
>>>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
>>>> },
>>>> "authorization":{
>>>> "class":"solr.RuleBasedAuthorizationPlugin",
>>>> "permissions":[{"name":"security-edit",
>>>>"role":"admin"}],
>>>> "user-role":{"solr":"admin"}
>>>> }}
>>>>
>>>> Here are the steps I followed:
>>>>
>>>> Upload security.json to zookeeper
>>>> ./zkcli.sh -z localhost:2181,localhost:2182,localhost:2183 -cmd putfile 
>>>> /security.json ~/solr/security.json
>>>>
>>>> Use zkCli.sh from Zookeeper to ensure the security.json is in Zookeeper at 
>>>> /security.json.  It is there and looks like what was originally uploaded.
>>>>
>>>> Start Solr Instances
>>>>
>>>> Attempt to create a permission, however get the following error:
>>>> {
>>>&g

Re: Issue Using Solr 5.3 Authentication and Authorization Plugins

[Noble Paul]

I'm investigating why restarts or first time start does not read the
security.json

On Tue, Sep 1, 2015 at 1:00 PM, Noble Paul <noble.p...@gmail.com> wrote:
> I removed that statement
>
> "If activating the authorization plugin doesn't protect the admin ui,
> how does one protect access to it?"
>
> One does not need to protect the admin UI. You only need to protect
> the relevant API calls . I mean it's OK to not protect the CSS and
> HTML stuff.  But if you perform an action to create a core or do a
> query through admin UI , it automatically will prompt you for
> credentials (if those APIs are protected)
>
> On Tue, Sep 1, 2015 at 12:41 PM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
>> Thanks for the clarification!
>>
>> So is the wiki page incorrect at
>> https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin 
>> which says that the admin ui will require authentication once the 
>> authorization plugin is activated?
>>
>> "An authorization plugin is also available to configure Solr with 
>> permissions to perform various activities in the system. Once activated, 
>> access to the Solr Admin UI and all requests will need to be authenticated 
>> and users will be required to have the proper authorization for all 
>> requests, including using the Admin UI and making any API calls."
>>
>> If activating the authorization plugin doesn't protect the admin ui, how 
>> does one protect access to it?
>>
>> Also, the issue I'm having is not just at restart.  According to the docs 
>> security.json should be uploaded to Zookeeper before starting any of the 
>> Solr instances.  However, I tried to upload security.json before starting 
>> any of the Solr instances, but it would not pick up the security config 
>> until after the Solr instances are already running and then uploading the 
>> security.json again.  I can see in the logs at startup that the Solr 
>> instances don't see any plugin enabled even though security.json is already 
>> in zookeeper and then after they are started and the security.json is 
>> uploaded again I see it reconfigure to use the plugin.
>>
>> Thanks,
>> Kevin
>>
>>> On Aug 31, 2015, at 11:22 PM, Noble Paul <noble.p...@gmail.com> wrote:
>>>
>>> Admin UI is not protected by any of these permissions. Only if you try
>>> to perform a protected operation , it asks for a password.
>>>
>>> I'll investigate the restart problem and report my  findings
>>>
>>>> On Tue, Sep 1, 2015 at 3:10 AM, Kevin Lee <kgle...@yahoo.com.invalid> 
>>>> wrote:
>>>> Anyone else running into any issues trying to get the authentication and 
>>>> authorization plugins in 5.3 working?
>>>>
>>>>> On Aug 29, 2015, at 2:30 AM, Kevin Lee <kgle...@yahoo.com.INVALID> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I’m trying to use the new basic auth plugin for Solr 5.3 and it doesn’t 
>>>>> seem to be working quite right.  Not sure if I’m missing steps or there 
>>>>> is a bug.  I am able to get it to protect access to a URL under a 
>>>>> collection, but am unable to get it to secure access to the Admin UI.  In 
>>>>> addition, after stopping the Solr and Zookeeper instances, the 
>>>>> security.json is still in Zookeeper, however Solr is allowing access to 
>>>>> everything again like the security configuration isn’t in place.
>>>>>
>>>>> Contents of security.json taken from wiki page, but edited to produce 
>>>>> valid JSON.  Had to move comma after 3rd from last “}” up to just after 
>>>>> the last “]”.
>>>>>
>>>>> {
>>>>> "authentication":{
>>>>> "class":"solr.BasicAuthPlugin",
>>>>> "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= 
>>>>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
>>>>> },
>>>>> "authorization":{
>>>>> "class":"solr.RuleBasedAuthorizationPlugin",
>>>>> "permissions":[{"name":"security-edit",
>>>>>"role":"admin"}],
>>>>> "user-role":{"solr":"admin"}
>>>>> }}
>>>>>
>>>>> Here are the steps I followed:
>>>>>
>>>>> Upload security.json to zookeeper
&

Re: Issue Using Solr 5.3 Authentication and Authorization Plugins

[Noble Paul]

Looks like there is a bug in that . On start/restart the security.json
is not loaded
I shall open a ticket

https://issues.apache.org/jira/browse/SOLR-8000

On Tue, Sep 1, 2015 at 1:01 PM, Noble Paul <noble.p...@gmail.com> wrote:
> I'm investigating why restarts or first time start does not read the
> security.json
>
> On Tue, Sep 1, 2015 at 1:00 PM, Noble Paul <noble.p...@gmail.com> wrote:
>> I removed that statement
>>
>> "If activating the authorization plugin doesn't protect the admin ui,
>> how does one protect access to it?"
>>
>> One does not need to protect the admin UI. You only need to protect
>> the relevant API calls . I mean it's OK to not protect the CSS and
>> HTML stuff.  But if you perform an action to create a core or do a
>> query through admin UI , it automatically will prompt you for
>> credentials (if those APIs are protected)
>>
>> On Tue, Sep 1, 2015 at 12:41 PM, Kevin Lee <kgle...@yahoo.com.invalid> wrote:
>>> Thanks for the clarification!
>>>
>>> So is the wiki page incorrect at
>>> https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin
>>>  which says that the admin ui will require authentication once the 
>>> authorization plugin is activated?
>>>
>>> "An authorization plugin is also available to configure Solr with 
>>> permissions to perform various activities in the system. Once activated, 
>>> access to the Solr Admin UI and all requests will need to be authenticated 
>>> and users will be required to have the proper authorization for all 
>>> requests, including using the Admin UI and making any API calls."
>>>
>>> If activating the authorization plugin doesn't protect the admin ui, how 
>>> does one protect access to it?
>>>
>>> Also, the issue I'm having is not just at restart.  According to the docs 
>>> security.json should be uploaded to Zookeeper before starting any of the 
>>> Solr instances.  However, I tried to upload security.json before starting 
>>> any of the Solr instances, but it would not pick up the security config 
>>> until after the Solr instances are already running and then uploading the 
>>> security.json again.  I can see in the logs at startup that the Solr 
>>> instances don't see any plugin enabled even though security.json is already 
>>> in zookeeper and then after they are started and the security.json is 
>>> uploaded again I see it reconfigure to use the plugin.
>>>
>>> Thanks,
>>> Kevin
>>>
>>>> On Aug 31, 2015, at 11:22 PM, Noble Paul <noble.p...@gmail.com> wrote:
>>>>
>>>> Admin UI is not protected by any of these permissions. Only if you try
>>>> to perform a protected operation , it asks for a password.
>>>>
>>>> I'll investigate the restart problem and report my  findings
>>>>
>>>>> On Tue, Sep 1, 2015 at 3:10 AM, Kevin Lee <kgle...@yahoo.com.invalid> 
>>>>> wrote:
>>>>> Anyone else running into any issues trying to get the authentication and 
>>>>> authorization plugins in 5.3 working?
>>>>>
>>>>>> On Aug 29, 2015, at 2:30 AM, Kevin Lee <kgle...@yahoo.com.INVALID> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I’m trying to use the new basic auth plugin for Solr 5.3 and it doesn’t 
>>>>>> seem to be working quite right.  Not sure if I’m missing steps or there 
>>>>>> is a bug.  I am able to get it to protect access to a URL under a 
>>>>>> collection, but am unable to get it to secure access to the Admin UI.  
>>>>>> In addition, after stopping the Solr and Zookeeper instances, the 
>>>>>> security.json is still in Zookeeper, however Solr is allowing access to 
>>>>>> everything again like the security configuration isn’t in place.
>>>>>>
>>>>>> Contents of security.json taken from wiki page, but edited to produce 
>>>>>> valid JSON.  Had to move comma after 3rd from last “}” up to just after 
>>>>>> the last “]”.
>>>>>>
>>>>>> {
>>>>>> "authentication":{
>>>>>> "class":"solr.BasicAuthPlugin",
>>>>>> "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= 
>>>>>> Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
>&g

Re: User Authentication

[Noble Paul]

did you manage to look at the reference guide?
https://cwiki.apache.org/confluence/display/solr/Securing+Solr

On Mon, Aug 24, 2015 at 9:23 PM, LeZotte, Tom
tom.lezo...@vanderbilt.edu wrote:
 Alex
 I got a super secret release of Solr 5.3.1, wasn’t suppose to say anything.

 Yes I’m running 5.2.1, I will check out the release notes for 5.3.

 Was looking for three types of user authentication, I guess.
 1. the Admin Console
 2. User auth for each Core ( and select and update) on a server.
 3. HTML interface access (example: 
 ajax-solrhttps://github.com/evolvingweb/ajax-solr)

 Thanks

 Tom LeZotte
 Health I.T. - Senior Product Developer
 (p) 615-875-8830






 On Aug 24, 2015, at 10:05 AM, Alexandre Rafalovitch 
 arafa...@gmail.commailto:arafa...@gmail.com wrote:

 Thanks for the email from the future. It is good to start to prepare
 for 5.3.1 now that 5.3 is nearly out.

 Joking aside (and assuming Solr 5.2.1), what exactly are you trying to
 achieve? Solr should not actually be exposed to the users directly. It
 should be hiding in a backend only visible to your middleware. If you
 are looking for a HTML interface that talks directly to Solr after
 authentication, that's not the right way to set it up.

 That said, some security features are being rolled out and you should
 definitely check the release notes for the 5.3.

 Regards,
   Alex.
 
 Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
 http://www.solr-start.com/


 On 24 August 2015 at 10:01, LeZotte, Tom tom.lezo...@vanderbilt.edu wrote:
 Hi Solr Community

 I have been trying to add user authentication to our Solr 5.3.1 RedHat 
 install. I’ve found some examples on user authentication on the Jetty side. 
 But they have failed.

 Does any one have a step by step example on authentication for the admin 
 screen? And a core?


 Thanks

 Tom LeZotte
 Health I.T. - Senior Product Developer
 (p) 615-875-8830










-- 
-
Noble Paul

Re: SOLR 5.3

[Noble Paul]

The release is underway. Incorporating some corrections suggested by
others. Expect an announcement ove rthe next few hours

On Sun, Aug 23, 2015 at 6:44 PM, Arcadius Ahouansou
arcad...@menelic.com wrote:
 Solr-5.3 has been available for download from
 http://mirror.catn.com/pub/apache/lucene/solr/5.3.0/

 The redirection on the web site will probably be fixed before we get the
 official announcement.

 Arcadius.

 On 23 August 2015 at 09:00, William Bell billnb...@gmail.com wrote:

 At lucene.apache.org/solr it says SOLR 5.3 is there, but when I click on
 downloads it shows Solr 5.2.1... ??

 APACHE SOLR™ 5.3.0Solr is the popular, blazing-fast, open source
 enterprise search platform built on Apache Lucene™.

 --
 Bill Bell
 billnb...@gmail.com
 cell 720-256-8076




 --
 Arcadius Ahouansou
 Menelic Ltd | Information is Power
 M: 07908761999
 W: www.menelic.com
 ---



-- 
-
Noble Paul

[ANNOUNCE] Apache Solr 5.2.0 released

[Noble Paul]

Solr is the popular, blazing fast, open source NoSQL search platform
from the Apache Lucene project. Its major features include powerful
full-text search, hit highlighting, faceted search, dynamic
clustering, database integration, rich document (e.g., Word, PDF)
handling, and geospatial search. Solr is highly scalable, providing
fault tolerant distributed search and indexing, and powers the search
and navigation features of many of the world's largest internet sites.

Solr 5.3.0 is available for immediate download at:
http://lucene.apache.org/solr/mirrors-solr-latest-redir.html

Solr 5.3 Release Highlights:

In addition to many other improvements in the security framework, Solr
now includes an AuthenticationPlugin implementing HTTP Basic Auth that
stores credentials securely in ZooKeeper. This is a simple way to
require a username and password for anyone accessing Solr’s admin
screen or APIs.
In built AuthorizationPlugin that provides fine grained control over
implementing ACLs for various resources with permisssion rules which
are stored in ZooKeeper.
The JSON Facet API can now change the domain for facet commands,
essentially doing a block join and moving from parents to children, or
children to parents before calculating the facet data.
Major improvements in performance of the new Facet Module / JSON Facet API.
Query and Range Facets under Pivot Facets. Just like the JSON Facet
API, pivot facets can how nest other facet types such as range and
query facets.
More Like This Query Parser options. The MoreLikeThis QParser now
supports all options provided by the MLT Handler. The query parser is
much more versatile than the handler as it works in cloud mode as well
as anywhere a normal query can be specified.
Added Schema API support in SolrJ
Added Scoring mode for query-time join and block join.
Added Smile response format

For upgrading from 5.2, please look at the Upgrading from Solr 5.2
section in the change log.

Detailed change log:
http://lucene.apache.org/solr/5_3_0/changes/Changes.html

Please report any feedback to the mailing lists
(http://lucene.apache.org/solr/discussion.html)


-- 
-
Noble Paul
www.lucidworks.com

Re: [ANNOUNCE] Apache Solr 5.2.0 released

[Noble Paul]

sorry , screwed up the title

On Tue, Aug 25, 2015 at 8:30 AM, Noble Paul noble.p...@gmail.com wrote:
 Solr is the popular, blazing fast, open source NoSQL search platform
 from the Apache Lucene project. Its major features include powerful
 full-text search, hit highlighting, faceted search, dynamic
 clustering, database integration, rich document (e.g., Word, PDF)
 handling, and geospatial search. Solr is highly scalable, providing
 fault tolerant distributed search and indexing, and powers the search
 and navigation features of many of the world's largest internet sites.

 Solr 5.3.0 is available for immediate download at:
 http://lucene.apache.org/solr/mirrors-solr-latest-redir.html

 Solr 5.3 Release Highlights:

 In addition to many other improvements in the security framework, Solr
 now includes an AuthenticationPlugin implementing HTTP Basic Auth that
 stores credentials securely in ZooKeeper. This is a simple way to
 require a username and password for anyone accessing Solr’s admin
 screen or APIs.
 In built AuthorizationPlugin that provides fine grained control over
 implementing ACLs for various resources with permisssion rules which
 are stored in ZooKeeper.
 The JSON Facet API can now change the domain for facet commands,
 essentially doing a block join and moving from parents to children, or
 children to parents before calculating the facet data.
 Major improvements in performance of the new Facet Module / JSON Facet API.
 Query and Range Facets under Pivot Facets. Just like the JSON Facet
 API, pivot facets can how nest other facet types such as range and
 query facets.
 More Like This Query Parser options. The MoreLikeThis QParser now
 supports all options provided by the MLT Handler. The query parser is
 much more versatile than the handler as it works in cloud mode as well
 as anywhere a normal query can be specified.
 Added Schema API support in SolrJ
 Added Scoring mode for query-time join and block join.
 Added Smile response format

 For upgrading from 5.2, please look at the Upgrading from Solr 5.2
 section in the change log.

 Detailed change log:
 http://lucene.apache.org/solr/5_3_0/changes/Changes.html

 Please report any feedback to the mailing lists
 (http://lucene.apache.org/solr/discussion.html)


 --
 -
 Noble Paul
 www.lucidworks.com



-- 
-
Noble Paul

[ANNOUNCE] Apache Solr 5.3.0 released

[Noble Paul]

Solr is the popular, blazing fast, open source NoSQL search platform
from the Apache Lucene project. Its major features include powerful
full-text search, hit highlighting, faceted search, dynamic
clustering, database integration, rich document (e.g., Word, PDF)
handling, and geospatial search. Solr is highly scalable, providing
fault tolerant distributed search and indexing, and powers the search
and navigation features of many of the world's largest internet sites.

Solr 5.3.0 is available for immediate download at:
http://lucene.apache.org/solr/mirrors-solr-latest-redir.html

Solr 5.3 Release Highlights:

In addition to many other improvements in the security framework, Solr
now includes an AuthenticationPlugin implementing HTTP Basic Auth that
stores credentials securely in ZooKeeper. This is a simple way to
require a username and password for anyone accessing Solr’s admin
screen or APIs.
In built AuthorizationPlugin that provides fine grained control over
implementing ACLs for various resources with permisssion rules which
are stored in ZooKeeper.
The JSON Facet API can now change the domain for facet commands,
essentially doing a block join and moving from parents to children, or
children to parents before calculating the facet data.
Major improvements in performance of the new Facet Module / JSON Facet API.
Query and Range Facets under Pivot Facets. Just like the JSON Facet
API, pivot facets can how nest other facet types such as range and
query facets.
More Like This Query Parser options. The MoreLikeThis QParser now
supports all options provided by the MLT Handler. The query parser is
much more versatile than the handler as it works in cloud mode as well
as anywhere a normal query can be specified.
Added Schema API support in SolrJ
Added Scoring mode for query-time join and block join.
Added Smile response format

For upgrading from 5.2, please look at the Upgrading from Solr 5.2
section in the change log.

Detailed change log:
http://lucene.apache.org/solr/5_3_0/changes/Changes.html

Please report any feedback to the mailing lists
(http://lucene.apache.org/solr/discussion.html)

-- 
-
Noble Paul
www.lucidworks.com

Re: User Authentication

[Noble Paul]

no.
Most of it is in Solr 5.3

On Tue, Aug 25, 2015 at 12:48 AM, Steven White swhite4...@gmail.com wrote:
 Hi Noble,

 Is everything in the link you provided applicable to Solr 5.2.1?

 Thanks

 Steve

 On Mon, Aug 24, 2015 at 2:20 PM, Noble Paul noble.p...@gmail.com wrote:

 did you manage to look at the reference guide?
 https://cwiki.apache.org/confluence/display/solr/Securing+Solr

 On Mon, Aug 24, 2015 at 9:23 PM, LeZotte, Tom
 tom.lezo...@vanderbilt.edu wrote:
  Alex
  I got a super secret release of Solr 5.3.1, wasn’t suppose to say
 anything.
 
  Yes I’m running 5.2.1, I will check out the release notes for 5.3.
 
  Was looking for three types of user authentication, I guess.
  1. the Admin Console
  2. User auth for each Core ( and select and update) on a server.
  3. HTML interface access (example: ajax-solr
 https://github.com/evolvingweb/ajax-solr)
 
  Thanks
 
  Tom LeZotte
  Health I.T. - Senior Product Developer
  (p) 615-875-8830
 
 
 
 
 
 
  On Aug 24, 2015, at 10:05 AM, Alexandre Rafalovitch arafa...@gmail.com
 mailto:arafa...@gmail.com wrote:
 
  Thanks for the email from the future. It is good to start to prepare
  for 5.3.1 now that 5.3 is nearly out.
 
  Joking aside (and assuming Solr 5.2.1), what exactly are you trying to
  achieve? Solr should not actually be exposed to the users directly. It
  should be hiding in a backend only visible to your middleware. If you
  are looking for a HTML interface that talks directly to Solr after
  authentication, that's not the right way to set it up.
 
  That said, some security features are being rolled out and you should
  definitely check the release notes for the 5.3.
 
  Regards,
Alex.
  
  Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:
  http://www.solr-start.com/
 
 
  On 24 August 2015 at 10:01, LeZotte, Tom tom.lezo...@vanderbilt.edu
 wrote:
  Hi Solr Community
 
  I have been trying to add user authentication to our Solr 5.3.1 RedHat
 install. I’ve found some examples on user authentication on the Jetty side.
 But they have failed.
 
  Does any one have a step by step example on authentication for the admin
 screen? And a core?
 
 
  Thanks
 
  Tom LeZotte
  Health I.T. - Senior Product Developer
  (p) 615-875-8830
 
 
 
 
 
 
 



 --
 -
 Noble Paul




-- 
-
Noble Paul

Re: Basic auth

[Noble Paul]

Although I'm not sure why you took this approach instead of
supporting  simple built-in basic auth and let us configure security
the old/easy way

Going with Jetty basic auth is not useful in a large enough  cluster.
Where do you store the credentials and how would you propagate it
across the cluster. When you use Solr, you need a SOlr like way of
managing that. The other problem is inter-node communication. How do
you pass credentials along in that case

I'm guessing it has to do with future requirement of field/doc level security

Acutally that is an orthogonal requirement

I hope you can get rid of the war file soon and start promoting Solr
as a set of libraries so one can easily embed/extend Solr

That is not what we have in mind. We want Solr to be a server which
controls every aspect of its running . We should have the choice of
getting rid of jetty or whatsoever and move to a new system. We only
guarantee to interface/protocol to remain constant



On Tue, Jul 28, 2015 at 2:19 AM, Fadi Mohsen fadi.moh...@gmail.com wrote:
 Thank you, I tested providing my implementation of authentication in 
 security.json, uploaded file to ZK (just considering authentication), started 
 nodes and it worked like a charm.

 That required of course turning off Jetty basic auth.

 Although I'm not sure why you took this approach instead of supporting  
 simple built-in basic auth and let us configure security the old/easy way.

 I'm guessing it has to do with future requirement of field/doc level security.

 I hope you can get rid of the war file soon and start promoting Solr as a set 
 of libraries so one can easily embed/extend Solr, since some (especially me) 
 might consider command line ZK operations are not that continues 
 delivery/automate everything/production friendly.

 It's easy today to spin up a jetty and wire / point out resource classes or 
 wire up CXF alongside to get things playing, but I'm probably missing out of 
 other things since I see many mails usually in consensus of not embedding and 
 rather want people to consider Solr as a stand-alone service, not sure why!
 I'm probably getting out of context here.

 Regards

 On 27 Jul 2015, at 13:17, Noble Paul noble.p...@gmail.com wrote:

 Q.do you know when it would be released?
 5.3 will be released in another 3-4 weeks .

 Q.Are there any requirements of ZK authentication must be there as well?
 NO

 bq.Providing my own security.json + class/implementation to verify
 user/pass should work today with 5.2, right?

 Yes. But, if you modify your credentials or anything in that JSON, you
 will have to restart all your nodes .

 Q.SOLR-7274 pluggable security is already in 5.2 (my requirement is to
 provide user/pass in a secure manner, not as argument on cmd or from
 (our unsecured) ZK but from a configuration restful service,

 I'm not clear what your question is. Basic Auth is a well-known
 standard. We are just implementing that standard. We store all
 credentials  permissions in ZK . That means it is only as secure as
 your ZK . As long as nobody can write to ZK, your system is safe

 On Wed, Jul 22, 2015 at 11:10 PM, Fadi Mohsen fadi.moh...@gmail.com wrote:
 Hi, I have some questions regarding basic auth and proper support in 5.3:

 do you know when it would be released?

 Are there any requirements of ZK authentication must be there as well?

 Do we store the user/pass in ZK?

 SOLR-7274 pluggable security is already in 5.2 (my requirement is to 
 provide user/pass in a secure manner, not as argument on cmd or from (our 
 unsecured) ZK but from a configuration restful service,
 I'm not sure 5.3 release would fit above requirement, can you reflect on 
 this?

 Providing my own security.json + class/implementation to verify user/pass 
 should work today with 5.2, right?

 Thanks
 Fadi

 On 22 Jul 2015, at 14:33, Noble Paul noble.p...@gmail.com wrote:

 Solr 5.3 is coming with proper basic auth support


 https://issues.apache.org/jira/browse/SOLR-7692

 On Wed, Jul 22, 2015 at 5:28 PM, Peter Sturge peter.stu...@gmail.com 
 wrote:
 if you're using Jetty you can use the standard realms mechanism for Basic
 Auth, and it works the same on Windows or UNIX. There's plenty of docs on
 the Jetty site about getting this working, although it does vary somewhat
 depending on the version of Jetty you're running (N.B. I would suggest
 using Jetty 9, and not 8, as 8 is missing some key authentication 
 classes).
 If, when you execute a search query to your Solr instance you get a
 username and password popup, then Jetty's auth is setup. If you don't then
 something's wrong in the Jetty config.

 it's worth noting that if you're doing distributed searches Basic Auth on
 its own will not work for you. This is because Solr sends distributed
 requests to remote instances on behalf of the user, and it has no 
 knowledge
 of the web container's auth mechanics. We got 'round this by customizing
 Solr to receive credentials and use them for authentication to remote

Re: Basic auth

[Noble Paul]

Q.do you know when it would be released?
5.3 will be released in another 3-4 weeks .

Q.Are there any requirements of ZK authentication must be there as well?
NO

bq.Providing my own security.json + class/implementation to verify
user/pass should work today with 5.2, right?

Yes. But, if you modify your credentials or anything in that JSON, you
will have to restart all your nodes .

Q.SOLR-7274 pluggable security is already in 5.2 (my requirement is to
provide user/pass in a secure manner, not as argument on cmd or from
(our unsecured) ZK but from a configuration restful service,

I'm not clear what your question is. Basic Auth is a well-known
standard. We are just implementing that standard. We store all
credentials  permissions in ZK . That means it is only as secure as
your ZK . As long as nobody can write to ZK, your system is safe

On Wed, Jul 22, 2015 at 11:10 PM, Fadi Mohsen fadi.moh...@gmail.com wrote:
 Hi, I have some questions regarding basic auth and proper support in 5.3:

 do you know when it would be released?

 Are there any requirements of ZK authentication must be there as well?

 Do we store the user/pass in ZK?

 SOLR-7274 pluggable security is already in 5.2 (my requirement is to provide 
 user/pass in a secure manner, not as argument on cmd or from (our unsecured) 
 ZK but from a configuration restful service,
 I'm not sure 5.3 release would fit above requirement, can you reflect on this?

 Providing my own security.json + class/implementation to verify user/pass 
 should work today with 5.2, right?

 Thanks
 Fadi

 On 22 Jul 2015, at 14:33, Noble Paul noble.p...@gmail.com wrote:

 Solr 5.3 is coming with proper basic auth support


 https://issues.apache.org/jira/browse/SOLR-7692

 On Wed, Jul 22, 2015 at 5:28 PM, Peter Sturge peter.stu...@gmail.com 
 wrote:
 if you're using Jetty you can use the standard realms mechanism for Basic
 Auth, and it works the same on Windows or UNIX. There's plenty of docs on
 the Jetty site about getting this working, although it does vary somewhat
 depending on the version of Jetty you're running (N.B. I would suggest
 using Jetty 9, and not 8, as 8 is missing some key authentication classes).
 If, when you execute a search query to your Solr instance you get a
 username and password popup, then Jetty's auth is setup. If you don't then
 something's wrong in the Jetty config.

 it's worth noting that if you're doing distributed searches Basic Auth on
 its own will not work for you. This is because Solr sends distributed
 requests to remote instances on behalf of the user, and it has no knowledge
 of the web container's auth mechanics. We got 'round this by customizing
 Solr to receive credentials and use them for authentication to remote
 instances - SOLR-1861 is an old implementation for a previous release, and
 there has been some significant refactoring of SearchHandler since then,
 but the concept works well for distributed queries.

 Thanks,
 Peter



 On Wed, Jul 22, 2015 at 11:18 AM, O. Klein kl...@octoweb.nl wrote:

 Steven White wrote
 Thanks for updating the wiki page.  However, my issue remains, I cannot
 get
 Basic auth working.  Has anyone got it working, on Windows?

 Doesn't work for me on Linux either.



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Basic-auth-tp4218053p4218519.html
 Sent from the Solr - User mailing list archive at Nabble.com.



 --
 -
 Noble Paul



-- 
-
Noble Paul

Re: Basic auth

[Noble Paul]

Solr 5.3 is coming with proper basic auth support


https://issues.apache.org/jira/browse/SOLR-7692

On Wed, Jul 22, 2015 at 5:28 PM, Peter Sturge peter.stu...@gmail.com wrote:
 if you're using Jetty you can use the standard realms mechanism for Basic
 Auth, and it works the same on Windows or UNIX. There's plenty of docs on
 the Jetty site about getting this working, although it does vary somewhat
 depending on the version of Jetty you're running (N.B. I would suggest
 using Jetty 9, and not 8, as 8 is missing some key authentication classes).
 If, when you execute a search query to your Solr instance you get a
 username and password popup, then Jetty's auth is setup. If you don't then
 something's wrong in the Jetty config.

 it's worth noting that if you're doing distributed searches Basic Auth on
 its own will not work for you. This is because Solr sends distributed
 requests to remote instances on behalf of the user, and it has no knowledge
 of the web container's auth mechanics. We got 'round this by customizing
 Solr to receive credentials and use them for authentication to remote
 instances - SOLR-1861 is an old implementation for a previous release, and
 there has been some significant refactoring of SearchHandler since then,
 but the concept works well for distributed queries.

 Thanks,
 Peter



 On Wed, Jul 22, 2015 at 11:18 AM, O. Klein kl...@octoweb.nl wrote:

 Steven White wrote
  Thanks for updating the wiki page.  However, my issue remains, I cannot
  get
  Basic auth working.  Has anyone got it working, on Windows?

 Doesn't work for me on Linux either.



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Basic-auth-tp4218053p4218519.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
-
Noble Paul

Re: Unable to update config file using zkcli or RELOAD

[Noble Paul]

The behavior has changed from Solr 5.0 onwards


Please refer to the How does it work section here
https://cwiki.apache.org/confluence/display/solr/Config+API

TL:DR

* Every node watches the conf set directory it is using
* Updating individual files WILL NOT trigger a config reload. BUt if you
modify the config set dir , it will trigger a reload. This is how config
API works



On Mon, Apr 6, 2015 at 2:34 AM, Shawn Heisey apa...@elyograg.org wrote:

 On 4/5/2015 12:32 AM, Shai Erera wrote:
  So, the questions that I have are:
 
 1. It does look like Solr re-loads cores on configuration changes, is
 that true?
 2. If (1) is YES, do I still need to manually invoke a collection
 RELOAD
 explicitly after updating the configuration?
 3. Can someone explain the errors I see in the log, even though the
 test
 passes and as I indicated, I'm able to index more documents and
 search them?

 My experience with SolrCloud is somewhat limited and even more limited
 in recent versions, but it was my understanding that config/schema
 changes do not result in new behavior without an explicit reload, unless
 you are using the managed schema or managed resources API to make the
 changes.  I don't know much about how the managed APIs actually work.

 Trying to interpret the stacktraces makes my brain hurt.  You probably
 know a lot more about the code for those areas than I do.

 Thanks,
 Shawn




-- 
-
Noble Paul

Re: Solr 4.x to Solr 5 = org.noggit.JSONParser$ParseException

[Noble Paul]

This code is executed every time Solr is initialized and it is unlikely
that it is a bug.
Are you using an older version of noggit.jar by any chance?


On Mon, Feb 23, 2015 at 6:30 PM, Clemens Wyss DEV clemens...@mysign.ch
wrote:

 Just about to upgrade to Solr5. My UnitTests fail:
 13:50:41.178 [main] ERROR org.apache.solr.core.CoreContainer - Error
 creating core [1-de_CH]: null
 java.lang.ExceptionInInitializerError: null
 at
 org.apache.solr.core.SolrConfig.getConfigOverlay(SolrConfig.java:359)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at org.apache.solr.core.SolrConfig.getOverlay(SolrConfig.java:808)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at
 org.apache.solr.core.SolrConfig.getSubstituteProperties(SolrConfig.java:798)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at org.apache.solr.core.Config.init(Config.java:152)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at org.apache.solr.core.Config.init(Config.java:92)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at org.apache.solr.core.SolrConfig.init(SolrConfig.java:180)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at
 org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:158)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at
 org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:80)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at
 org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:61)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at
 org.apache.solr.core.CoreContainer.create(CoreContainer.java:511)
 [solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at
 org.apache.solr.core.CoreContainer.create(CoreContainer.java:488)
 [solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 at
 ch.mysign.search.solr.EmbeddedSolrMode.prepareCore(EmbeddedSolrMode.java:51)
 [target/:na]
 ...
 at
 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
 [.cp/:na]
 Caused by: org.noggit.JSONParser$ParseException: Expected string:
 char=u,position=2 BEFORE='{ u' AFTER='pdateHandler : { autoCo'
 at org.noggit.JSONParser.err(JSONParser.java:223) ~[noggit.jar:na]
 at org.noggit.JSONParser.nextEvent(JSONParser.java:671)
 ~[noggit.jar:na]
 at org.noggit.ObjectBuilder.getObject(ObjectBuilder.java:123)
 ~[noggit.jar:na]
 at
 org.apache.solr.core.ConfigOverlay.clinit(ConfigOverlay.java:213)
 ~[solr-core.jar:5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10]
 ... 56 common frames omitted

 Look like the exception occurs in the ConfigOverlay static block, line 213:
 editable_prop_map =  (Map)new ObjectBuilder(new JSONParser(new
 StringReader(
   MAPPING))).getObject();

 What is happening?




-- 
-
Noble Paul

Re: Backuping SolrCloud

[Noble Paul]

Hi,
There is a ticket for the same .

https://issues.apache.org/jira/browse/SOLR-5750

Would you mind contributing to the discussion there?


On Mon, Nov 24, 2014 at 11:07 AM, ralph tice ralph.t...@gmail.com wrote:

 I have a writeup of how to perform safe backups here:
 https://gist.github.com/ralph-tice/887414a7f8082a0cb828

 There are some tickets around this work to further the ease of
 backups, especially https://issues.apache.org/jira/browse/SOLR-5750

 On Mon, Nov 24, 2014 at 9:45 AM, Vivek Pathak vpat...@orgmeta.com wrote:
  I was able to get very good backup procedure by having cron job perform
 compact on each shard and then copy out the physical shard (the full lucene
 index directory to a backup server)
 
  Updates would need to be stopped during this period.   And restore would
 be as simple as copying in the backed up shard and restarting solr
 
  On Nov 24, 2014, at 9:48 AM, elmerfudd na...@012.net.il wrote:
 
  Hi, I'm looking for a built-in SolrCloud backup mechanism.
  I want to backup my Index (scheduled / manual backups) while Indexing
 and
  searching.
 
  What is the proper way to perform this backup-restore task?
 
 
  Thanks.
 
 
 
 
  --
  View this message in context:
 http://lucene.472066.n3.nabble.com/Backuping-SolrCloud-tp4170624.html
  Sent from the Solr - User mailing list archive at Nabble.com.




-- 
-
Noble Paul

Re: Migrating from master/slave to solrcloud.

[Noble Paul]

yeah , .
start a zookeeper server.

stop your solr servers and start the server with the -Dzkhost system
property

On Sun, Nov 9, 2014 at 11:15 PM, mike st. john mstj...@gmail.com wrote:

 Is there a quick way to go from single index master/slave to solrcloud
 without a full reindex?


 thanks.

 Msj




-- 
-
Noble Paul

Re: Custom JSON

[Noble Paul]

check https://issues.apache.org/jira/browse/SOLR-6633

On Fri, Oct 17, 2014 at 5:35 PM, Alexandre Rafalovitch arafa...@gmail.com
wrote:

 I wonder how hard it would be to write an URP to just copy JSON from the
 request into a store-only field?

 Regards,
  Alex
 On 17/10/2014 1:21 am, Noble Paul noble.p...@gmail.com wrote:

  The original json is is not stored the fields are extracted and the data
 is
  thrown away
 
  On Fri, Oct 17, 2014 at 1:18 AM, Scott Dawson sc.e.daw...@gmail.com
  wrote:
 
   Noble,
   Thanks. You're right. I had some things incorrectly configured but now
 I
   can put structured JSON into Solr using the out-of-the-box
  solrconfig.xml.
  
   One additional question: Is there any way to query Solr and receive the
   original structured JSON document in response? Or does the flattening
   process that happens during indexing obliterate the original structure
  with
   no way to reconstruct it?
  
   Thanks again,
   Scott
  
   On Thu, Oct 16, 2014 at 2:10 PM, Noble Paul noble.p...@gmail.com
  wrote:
  
The end point  /update/json/docs is enabled implicitly in Solr
   irrespective
of the solrconfig.xml
In schemaless the fields are created automatically by solr.
   
If you have all the fields created in your schema.xml it will work .
   
if you  need an id field please use a copy field to create one
   
--Noble
   
On Thu, Oct 16, 2014 at 8:42 PM, Scott Dawson sc.e.daw...@gmail.com
 
wrote:
   
 Hello,
 I'm trying to use the new custom JSON feature described in
 https://issues.apache.org/jira/browse/SOLR-6304. I'm running Solr
4.10.1.
 It seems that the new feature, or more specifically, the
/update/json/docs
 endpoint is not enabled out-of-the-box except in the schema-less
   example.
 Is there some dependence of the feature on schemaless mode? I've
  tried
 pulling the endpoint definition and related pieces of the
 example-schemaless solrconfig.xml and adding those to the
 standard
 solrconfig.xml in the main example but I've run into a cascade of
   issues.
 Right now I'm getting a This IndexSchema is not mutable exception
   when
I
 try to post to the /update/json/docs endpoint.

 My real question is -- what's the easiest way to get this feature
 up
   and
 running quickly and is this documented somewhere? I'm trying to do
 a
quick
 proof-of-concept to verify that we can move from our current flat
  JSON
 ingestion to a more natural use of structured JSON.

 Thanks,
 Scott Dawson

   
   
   
--
-
Noble Paul
   
  
 
 
 
  --
  -
  Noble Paul
 




-- 
-
Noble Paul

Re: Custom JSON

[Noble Paul]

The original json is is not stored the fields are extracted and the data is
thrown away

On Fri, Oct 17, 2014 at 1:18 AM, Scott Dawson sc.e.daw...@gmail.com wrote:

 Noble,
 Thanks. You're right. I had some things incorrectly configured but now I
 can put structured JSON into Solr using the out-of-the-box solrconfig.xml.

 One additional question: Is there any way to query Solr and receive the
 original structured JSON document in response? Or does the flattening
 process that happens during indexing obliterate the original structure with
 no way to reconstruct it?

 Thanks again,
 Scott

 On Thu, Oct 16, 2014 at 2:10 PM, Noble Paul noble.p...@gmail.com wrote:

  The end point  /update/json/docs is enabled implicitly in Solr
 irrespective
  of the solrconfig.xml
  In schemaless the fields are created automatically by solr.
 
  If you have all the fields created in your schema.xml it will work .
 
  if you  need an id field please use a copy field to create one
 
  --Noble
 
  On Thu, Oct 16, 2014 at 8:42 PM, Scott Dawson sc.e.daw...@gmail.com
  wrote:
 
   Hello,
   I'm trying to use the new custom JSON feature described in
   https://issues.apache.org/jira/browse/SOLR-6304. I'm running Solr
  4.10.1.
   It seems that the new feature, or more specifically, the
  /update/json/docs
   endpoint is not enabled out-of-the-box except in the schema-less
 example.
   Is there some dependence of the feature on schemaless mode? I've tried
   pulling the endpoint definition and related pieces of the
   example-schemaless solrconfig.xml and adding those to the standard
   solrconfig.xml in the main example but I've run into a cascade of
 issues.
   Right now I'm getting a This IndexSchema is not mutable exception
 when
  I
   try to post to the /update/json/docs endpoint.
  
   My real question is -- what's the easiest way to get this feature up
 and
   running quickly and is this documented somewhere? I'm trying to do a
  quick
   proof-of-concept to verify that we can move from our current flat JSON
   ingestion to a more natural use of structured JSON.
  
   Thanks,
   Scott Dawson
  
 
 
 
  --
  -
  Noble Paul
 




-- 
-
Noble Paul

Re: embedded documents

[Noble Paul]

The simplest use case is to dump the entire json using split=/f=/** . i am
planning to add an alias for the same (SOLR-6343) .

The nested docs is missing now and we will need to add it. A ticket needs
to be opened


On Mon, Aug 25, 2014 at 6:45 AM, Jack Krupansky j...@basetechnology.com
wrote:

 Thanks, Erik, but... I've read that Jira several times over the past
 month, it is is far too cryptic for me to make any sense out of what it is
 really trying to do. A simpler approach is clearly needed.

 My perception of SOLR-6304 is not that it indexes a single JSON object as
 a single Solr document, but that it generates a collection of separate
 documents, somewhat analogous to Lucene block/child documents, but... not
 quite.

 I understood the request on this message thread to be the flattening of a
 single nested JSON object to a single Solr document.

 IMHO, we need to be trying to make Solr more automatic and more
 approachable, not an even more complicated toolkit.

 -- Jack Krupansky

 -Original Message- From: Erik Hatcher
 Sent: Monday, August 25, 2014 9:32 AM

 To: solr-user@lucene.apache.org
 Subject: Re: embedded documents

 Jack et al - there’s now this, which is available in the any-minute
 release of Solr 4.10: https://issues.apache.org/jira/browse/SOLR-6304

 Erik

 On Aug 25, 2014, at 5:01 AM, Jack Krupansky j...@basetechnology.com
 wrote:

  That's a completely different concept, I think - the ability to return a
 single field value as a structured JSON object in the writer, rather than
 simply loading from a nested JSON object and distributing the key values
 to normal Solr fields.

 -- Jack Krupansky

 -Original Message- From: Bill Bell
 Sent: Sunday, August 24, 2014 7:30 PM
 To: solr-user@lucene.apache.org
 Subject: Re: embedded documents

 See my Jira. It supports it via json.fsuffix=_jsonwt=json

 http://mail-archives.apache.org/mod_mbox/lucene-dev/
 201304.mbox/%3CJIRA.12641293.1365394604231.125944.1365397875874@arcas%3E

 Bill Bell
 Sent from mobile


  On Aug 24, 2014, at 6:43 AM, Jack Krupansky j...@basetechnology.com
 wrote:

 Indexing and query of raw JSON would be a valuable addition to Solr, so
 maybe you could simply explain more precisely your data model and
 transformation rules. For example, when multi-level nesting occurs, what
 does your loader do?

 Maybe if the fielld names were derived by concatenating the full path of
 JSON key names, like titles_json.FR, field_naming nesting could be handled
 in a fully automated manner.

 I had been thinking of filing a Jira proposing exactly that, so that
 even the most deeply nested JSON maps could be supported, although
 combinations of arrays and maps would be problematic.

 -- Jack Krupansky

 -Original Message- From: Michael Pitsounis
 Sent: Wednesday, August 20, 2014 7:14 PM
 To: solr-user@lucene.apache.org
 Subject: embedded documents

 Hello everybody,

 I had a requirement to store complicated json documents in solr.

 i have modified the JsonLoader to accept complicated json documents with
 arrays/objects as values.

 It stores the object/array and then flatten it and  indexes the fields.

 e.g  basic example document

 {
  titles_json:{FR:This is the FR title , EN:This is the EN
 title} ,
  id: 103,
  guid: 3b2f2998-85ac-4a4e-8867-beb551c0b3c6
 }

 It will store titles_json:{FR:This is the FR title , EN:This is
 the
 EN title}
 and then index fields

 titles.FR:This is the FR title
 titles.EN:This is the EN title


 Do you see any problems with this approach?



 Regards,
 Michael Pitsounis





-- 
-
Noble Paul

Re: Can I use multiple cores

[Noble Paul]

Hi Ramprasad,


I have used it in a cluster with millions of users (1 user per core) in
legacy cloud mode .We used the on demand core loading feature where each
Solr had 30,000 cores and at a time only 2000 cores were in memory. You are
just hitting 400 and I don't see much of a problem . What is your h/w bTW?


On Tue, Aug 12, 2014 at 12:10 PM, Ramprasad Padmanabhan 
ramprasad...@gmail.com wrote:

 I need to store in SOLR all data of my clients mailing activitiy

 The data contains meta data like From;To:Date;Time:Subject etc

 I would easily have 1000 Million records every 2 months.

 What I am currently doing is creating cores per client. So I have 400 cores
 already.

 Is this a good idea to do ?

 What is the general practice for creating cores




-- 
-
Noble Paul

Re: Can I use multiple cores

[Noble Paul]

The machines were 32GB ram boxes. You must do the RAM requirement
calculation for your indexes . Just the no:of indexes alone won't be enough
to arrive at the RAM requirement


On Tue, Aug 12, 2014 at 6:59 PM, Ramprasad Padmanabhan 
ramprasad...@gmail.com wrote:

 On 12 August 2014 18:18, Noble Paul noble.p...@gmail.com wrote:

  Hi Ramprasad,
 
 
  I have used it in a cluster with millions of users (1 user per core) in
  legacy cloud mode .We used the on demand core loading feature where each
  Solr had 30,000 cores and at a time only 2000 cores were in memory. You
 are
  just hitting 400 and I don't see much of a problem . What is your h/w
 bTW?
 
 
  On Tue, Aug 12, 2014 at 12:10 PM, Ramprasad Padmanabhan 
  ramprasad...@gmail.com wrote:
 
   I need to store in SOLR all data of my clients mailing activitiy
  
   The data contains meta data like From;To:Date;Time:Subject etc
  
   I would easily have 1000 Million records every 2 months.
  
   What I am currently doing is creating cores per client. So I have 400
  cores
   already.
  
   Is this a good idea to do ?
  
   What is the general practice for creating cores
  
 
 
 I have a single machine 16GB Ram with 16 cpu cores

 What is the h/w you are using




-- 
-
Noble Paul

Re: Hash range to shard assignment

[Noble Paul നോബിള്‍ नोब्ळ्]

That is in the pipeline. within next 3-4 months for sure

On Mon, Sep 23, 2013 at 11:07 PM, lochri loc...@web.de wrote:
 Yes, actually that would be a very comfortable solution.
 Is that planned ? And if so, when will it be released ?



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Hash-range-to-shard-assignment-tp4091204p4091591.html
 Sent from the Solr - User mailing list archive at Nabble.com.



-- 
-
Noble Paul

Re: Hash range to shard assignment

[Noble Paul നോബിള്‍ नोब्ळ्]

Custom routers is an idea that is floated around ad easy to implement.
Just that it is something we resist to add another extension point.

The point is we are planning other features which would obviate the
need for a custom router. Something like splitting a shard by a query.
Will it be a good enough solution for you?





On Mon, Sep 23, 2013 at 2:52 PM, lochri loc...@web.de wrote:
 Thanks for the clarification.

 Still I would think it is sub-optimal to split shards when we actually don't
 know which mailboxes we actually split. It may create splits of small users
 which leads to unnecessary distribution of the smaller users.

 We thought about doing the routing ourself. As far as a I understood we can
 do distributed searches across multiple collections. What do you think about
 this option ?

 For the ideal solution: when will custom routers be supported ?

 Regards,
 Lochri



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Hash-range-to-shard-assignment-tp4091204p4091503.html
 Sent from the Solr - User mailing list archive at Nabble.com.



-- 
-
Noble Paul

Re: Hash range to shard assignment

[Noble Paul നോബിള്‍ नोब्ळ्]

This would need you to plug your own router . It is not yet possible

But , you can split that shard repeatedly and keep the no:of users in that
shard limited


On Fri, Sep 20, 2013 at 3:52 PM, lochri loc...@web.de wrote:

 Hello folks,

 we would like to have control of where certain hash values or ranges are
 being located.
 The reason is that we want to shard per user but we know ahead that one or
 more specific users could grow way faster than others. Therefore we would
 like to locate them on separate shards (which may be on the same server
 initially and can be moved out later).

 So my question: can we control the hash-ranges and hash-range to shard
 assignment in SolrCloud ?

 Regards,
 Lochri




 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Hash-range-to-shard-assignment-tp4091204.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
-
Noble Paul

Re: dataconfig to index ZIP Files

[Noble Paul നോബിള്‍ नोब्ळ्]

IIRC Zip files are not supported


On Mon, Jul 1, 2013 at 10:30 PM, ericrs22 ericr...@yahoo.com wrote:

 To answer the previous Post:

 I was not sure what datasource=binaryFile I took it from a PDF sample
 thinking that would help.

 after setting datasource=null I'm still gett the same errors...

 dataConfig
 dataSource type=BinFileDataSource user=svcSolr
 password=SomePassword /
 document
 entity name=Archive
   processor=FileListEntityProcessor baseDir=E:\ArchiveRoot
 fileName=.zip$ recursive=true rootEntity=false dataSource=null
 onError=skip

 field column=fileSize name=size/
 field column=file
 name=filename/

 /entity

 /document
 /dataConfig

 the logs report this:


 INFO  - 2013-07-01 16:45:57.317;
 org.apache.solr.handler.dataimport.DataImporter; Starting Full Import
 WARN  - 2013-07-01 16:45:57.333;
 org.apache.solr.handler.dataimport.SimplePropertiesWriter; Unable to read:
 dataimport.properties




 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/dataconfig-to-index-ZIP-Files-tp4073965p4074399.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
-
Noble Paul

Re: DataImportHandler: Problems with delta-import and CachedSqlEntityProcessor

[Noble Paul നോബിള്‍ नोब्ळ्]

it is possible to create two separate root entities . one for full-import
and another for delta. for the delta-import you can skip Cache that way



On Thu, Jun 20, 2013 at 1:50 PM, Constantin Wolber 
constantin.wol...@medicalcolumbus.de wrote:

 Hi,

 i searched for a solution for quite some time but did not manage to find
 some real hints on how to fix it.


 I'm using solr 4.3.0 1477023 - simonw - 2013-04-29 15:10:12 running in a
 tomcat 6 container.

 My data import setup is basically the following:

 Data-config.xml:

 entity
 name=article
 dataSource=ds1
 query=SELECT * FROM article
 deltaQuery=SELECT myownid FROM articleHistory WHERE modified_date
 gt; '${dih.last_index_time}
 deltaImportQuery=SELECT * FROM article WHERE
 myownid=${dih.delta.myownid}
 pk=myownid
 field column=myownid name=id/

 entity
 name=supplier
 dataSource=ds2
 query=SELECT * FROM supplier WHERE status=1
 processor=CachedSqlEntityProcessor
 cacheKey=SUPPLIER_ID
 cacheLookup=article.ARTICLE_SUPPLIER_ID
 /entity

 entity
 name=attributes
 dataSource=ds1
 query=SELECT ARTICLE_ID,'Key:'+ATTRIBUTE_KEY+'
 Value:'+ATTRIBUTE_VALUE FROM attributes
 cacheKey=ARTICLE_ID
 cacheLookup=article.myownid
 processor=CachedSqlEntityProcessor
 /entity
 /entity


 Ok now for the problem:

 At first I tried everything without the Cache. But the full-import took a
 very long time. Because the attributes query is pretty slow compared to the
 rest. As a result I got a processing speed of around 150 Documents/s.
 When switching everything to the CachedSqlEntityProcessor the full import
 processed at the speed of 4000 Documents/s

 So full import is running quite fine. Now I wanted to use the delta
 import. When running the delta import I was expecting the ramp up time to
 be about the same as in full import since I need to load the whole table
 supplier and attributes to the cache in the first step. But when looking
 into the log file the weird thing is solr seems to refresh the Cache for
 every single document that is processed. So currently my delta-import is a
 lot slower than the full-import. I even tried to add the deltaImportQuery
 parameter to the entity but it doesn't change the behavior at all (of
 course I know it is not supposed to change anything in the setup I run).

 The following solutions would be possible in my opinion:

 1. Is there any way to tell the config to ignore the Cache when running a
 delta import? That would help already because we are talking about the
 maximum of 500 documents changed in 15 minutes compared to over 5 million
 documents in total.
 2. Get solr to not refresh the cash for every document.

 Best Regards

 Constantin Wolber




-- 
-
Noble Paul

Re: DataImportHandler: Problems with delta-import and CachedSqlEntityProcessor

[Noble Paul നോബിള്‍ नोब्ळ्]

yes. that's right


On Thu, Jun 20, 2013 at 8:16 PM, Constantin Wolber 
constantin.wol...@medicalcolumbus.de wrote:

 Hi,

 i may have been a little to fast with my response.

 After reading a bit more I imagine you meant running the full-import with
 the entity param for the root entity for full import. And running the delta
 import with the entity param for the delta entity. Is that correct?

 Regards

 Constantin


 -Ursprüngliche Nachricht-
 Von: Constantin Wolber [mailto:constantin.wol...@medicalcolumbus.de]
 Gesendet: Donnerstag, 20. Juni 2013 16:42
 An: solr-user@lucene.apache.org
 Betreff: AW: DataImportHandler: Problems with delta-import and
 CachedSqlEntityProcessor

 Hi,

 and thanks for the answer. But I'm a little bit confused about what you
 are suggesting.
 I did not really use the rootEntity attribute before. But from what I read
 in the documentation as far as I can tell that would result in two
 documents (maybe with the same id which would probably result in only one
 document being stored) because one for each root entity.

 It would be great if you could just sketch the setup with the entities I
 provided. Because currently I have no idea on how to do it.

 Regards

 Constantin


 -Ursprüngliche Nachricht-
 Von: Noble Paul നോബിള്‍ नोब्ळ् [mailto:noble.p...@gmail.com]
 Gesendet: Donnerstag, 20. Juni 2013 15:42
 An: solr-user@lucene.apache.org
 Betreff: Re: DataImportHandler: Problems with delta-import and
 CachedSqlEntityProcessor

 it is possible to create two separate root entities . one for full-import
 and another for delta. for the delta-import you can skip Cache that way



 On Thu, Jun 20, 2013 at 1:50 PM, Constantin Wolber 
 constantin.wol...@medicalcolumbus.de wrote:

  Hi,
 
  i searched for a solution for quite some time but did not manage to
  find some real hints on how to fix it.
 
 
  I'm using solr 4.3.0 1477023 - simonw - 2013-04-29 15:10:12 running in
  a tomcat 6 container.
 
  My data import setup is basically the following:
 
  Data-config.xml:
 
  entity
  name=article
  dataSource=ds1
  query=SELECT * FROM article
  deltaQuery=SELECT myownid FROM articleHistory WHERE
  modified_date gt; '${dih.last_index_time}
  deltaImportQuery=SELECT * FROM article WHERE
  myownid=${dih.delta.myownid}
  pk=myownid
  field column=myownid name=id/
 
  entity
  name=supplier
  dataSource=ds2
  query=SELECT * FROM supplier WHERE status=1
  processor=CachedSqlEntityProcessor
  cacheKey=SUPPLIER_ID
  cacheLookup=article.ARTICLE_SUPPLIER_ID
  /entity
 
  entity
  name=attributes
  dataSource=ds1
  query=SELECT ARTICLE_ID,'Key:'+ATTRIBUTE_KEY+'
  Value:'+ATTRIBUTE_VALUE FROM attributes
  cacheKey=ARTICLE_ID
  cacheLookup=article.myownid
  processor=CachedSqlEntityProcessor
  /entity
  /entity
 
 
  Ok now for the problem:
 
  At first I tried everything without the Cache. But the full-import
  took a very long time. Because the attributes query is pretty slow
  compared to the rest. As a result I got a processing speed of around 150
 Documents/s.
  When switching everything to the CachedSqlEntityProcessor the full
  import processed at the speed of 4000 Documents/s
 
  So full import is running quite fine. Now I wanted to use the delta
  import. When running the delta import I was expecting the ramp up time
  to be about the same as in full import since I need to load the whole
  table supplier and attributes to the cache in the first step. But when
  looking into the log file the weird thing is solr seems to refresh the
  Cache for every single document that is processed. So currently my
  delta-import is a lot slower than the full-import. I even tried to add
  the deltaImportQuery parameter to the entity but it doesn't change the
  behavior at all (of course I know it is not supposed to change anything
 in the setup I run).
 
  The following solutions would be possible in my opinion:
 
  1. Is there any way to tell the config to ignore the Cache when
  running a delta import? That would help already because we are talking
  about the maximum of 500 documents changed in 15 minutes compared to
  over 5 million documents in total.
  2. Get solr to not refresh the cash for every document.
 
  Best Regards
 
  Constantin Wolber
 
 


 --
 -
 Noble Paul




-- 
-
Noble Paul

Re: Replication not working

[Noble Paul നോബിള്‍ नोब्ळ्]

can you check with the indexversion command on both mater and slave?

pollInterval is set to 2 minutes. It is usually long . So you may need to
wait for 2 mins for the replication to kick in


On Tue, Jun 11, 2013 at 3:21 PM, thomas.poroc...@der.net wrote:

 Hi all,



 we have a setup with multiple cores, loaded via DataImportHandlers.

 Works fine so far.

 Now we are trying to get the replication working (for one core so far).
 But the automated replication is never happening.

 Manually triggered replication works!



 Environment:

 Solr 4.1 (also tried with 4.3)

 App-Server JBoss 4.3.

 Java 1.6



 There are two JBoss instances running on different ports on the same box
 with their own solr.home directories.



 Configuration is done like described in the documentation:



 requestHandler name=/replication class=solr.ReplicationHandler 

  lst name=master

   str name=enable${de.der.pu.solr.master.enable:false}/str

   str name=replicateAfterstartup/str

   str name=replicateAftercommit/str

   str name=replicateAfteroptimize/str

   str name=confFilesstopwords.txt, solrconfig.xml/str

  /lst

  lst name=slave

  str name=enable${de.der.pu.solr.slave.enable:false}/str

   str
 name=masterUrlhttp://localhost:30006/solr/${solr.core.name}/str

   str name=pollInterfall00:02:00/str

  /lst

   /requestHandler



 Basically it looks all fine from the admin-pages.



 The polling from the slave is going on but nothing happens.

 We have tried to delete slave index completely and restart both servers.
 Reimportet the master data several times and so on..



 On the masters replication page I see:

 - replication enable: true

 - replicateAfter: commit, startup

 - confFiles: stopwords.txt, solrconfig.xml



 On slave side I see:

 -masters version 1370612995391  53   2.56 MB

 -master url:  http://localhost:30006/solr/contacts

 -poling enable: true



 And master settings like on master side...



 When I enter
 http://localhost:30006/solr/contacts/replication?command=detailswt=json
 indent=true in the browser the response seems ok:

 {
   responseHeader:{
 status:0,
 QTime:0},
   details:{
 indexSize:2.56 MB,

 indexPath:D:\\usr\\local\\phx-unlimited\\jboss\\solr_cache\\test_pto_
 node1_solr\\contacts\\data\\index/,
 commits:[[
 indexVersion,1370612995391,
 generation,53,
 filelist,[_1r.fdt,
   _1r.fdx,
   _1r.fnm,
   _1r.nvd,
   _1r.nvm,
   _1r.si,
   _1r_Lucene41_0.doc,
   _1r_Lucene41_0.pos,
   _1r_Lucene41_0.tim,
   _1r_Lucene41_0.tip,
   segments_1h]]],
 isMaster:true,
 isSlave:false,
 indexVersion:1370612995391,
 generation:53,
 master:{
   confFiles:stopwords.txt, solrconfig.xml,
   replicateAfter:[commit,
 startup],
   replicationEnabled:true,
   replicableVersion:1370612995391,
   replicableGeneration:53}},
   WARNING:This response format is experimental.  It is likely to
 change in the future.}





 Any idea how we could go on?



 Regards

 Thomas






-- 
-
Noble Paul

Re: Replication not working

[Noble Paul നോബിള്‍ नोब्ळ्]

You said polling is happening and nothing is replicated

What do the logs say on slave (Set level to INFO) ?




On Tue, Jun 11, 2013 at 4:54 PM, thomas.poroc...@der.net wrote:

 Calling indexversion on master gives:
  response
   lst name=responseHeader
  int name=status0/intint name=QTime0/int
   /lst
   long name=indexversion1370612995391/long
   long name=generation53/long
  /response

 On Slave:
  response
lst name=responseHeaderint name=status0/int
int name=QTime0/int/lst
long name=indexversion0/long
long name=generation1/long
  /response

  pollInterval is set to 2 minutes. It is usually long

 I know ;-)


 -Ursprüngliche Nachricht-
 Von: Noble Paul നോബിള്‍ नोब्ळ् [mailto:noble.p...@gmail.com]
 Gesendet: Dienstag, 11. Juni 2013 13:16
 An: solr-user@lucene.apache.org
 Betreff: Re: Replication not working

 can you check with the indexversion command on both mater and slave?

 pollInterval is set to 2 minutes. It is usually long . So you may need to
 wait for 2 mins for the replication to kick in


 On Tue, Jun 11, 2013 at 3:21 PM, thomas.poroc...@der.net wrote:

  Hi all,
 
 
 
  we have a setup with multiple cores, loaded via DataImportHandlers.
 
  Works fine so far.
 
  Now we are trying to get the replication working (for one core so far).
  But the automated replication is never happening.
 
  Manually triggered replication works!
 
 
 
  Environment:
 
  Solr 4.1 (also tried with 4.3)
 
  App-Server JBoss 4.3.
 
  Java 1.6
 
 
 
  There are two JBoss instances running on different ports on the same box
  with their own solr.home directories.
 
 
 
  Configuration is done like described in the documentation:
 
 
 
  requestHandler name=/replication class=solr.ReplicationHandler 
 
   lst name=master
 
str name=enable${de.der.pu.solr.master.enable:false}/str
 
str name=replicateAfterstartup/str
 
str name=replicateAftercommit/str
 
str name=replicateAfteroptimize/str
 
str name=confFilesstopwords.txt, solrconfig.xml/str
 
   /lst
 
   lst name=slave
 
   str name=enable${de.der.pu.solr.slave.enable:false}/str
 
str
  name=masterUrlhttp://localhost:30006/solr/${solr.core.name}/str
 
str name=pollInterfall00:02:00/str
 
   /lst
 
/requestHandler
 
 
 
  Basically it looks all fine from the admin-pages.
 
 
 
  The polling from the slave is going on but nothing happens.
 
  We have tried to delete slave index completely and restart both servers.
  Reimportet the master data several times and so on..
 
 
 
  On the masters replication page I see:
 
  - replication enable: true
 
  - replicateAfter: commit, startup
 
  - confFiles: stopwords.txt, solrconfig.xml
 
 
 
  On slave side I see:
 
  -masters version 1370612995391  53   2.56 MB
 
  -master url:  http://localhost:30006/solr/contacts
 
  -poling enable: true
 
 
 
  And master settings like on master side...
 
 
 
  When I enter
  http://localhost:30006/solr/contacts/replication?command=detailswt=json
  indent=true in the browser the response seems ok:
 
  {
responseHeader:{
  status:0,
  QTime:0},
details:{
  indexSize:2.56 MB,
 
  indexPath:D:\\usr\\local\\phx-unlimited\\jboss\\solr_cache\\test_pto_
  node1_solr\\contacts\\data\\index/,
  commits:[[
  indexVersion,1370612995391,
  generation,53,
  filelist,[_1r.fdt,
_1r.fdx,
_1r.fnm,
_1r.nvd,
_1r.nvm,
_1r.si,
_1r_Lucene41_0.doc,
_1r_Lucene41_0.pos,
_1r_Lucene41_0.tim,
_1r_Lucene41_0.tip,
segments_1h]]],
  isMaster:true,
  isSlave:false,
  indexVersion:1370612995391,
  generation:53,
  master:{
confFiles:stopwords.txt, solrconfig.xml,
replicateAfter:[commit,
  startup],
replicationEnabled:true,
replicableVersion:1370612995391,
replicableGeneration:53}},
WARNING:This response format is experimental.  It is likely to
  change in the future.}
 
 
 
 
 
  Any idea how we could go on?
 
 
 
  Regards
 
  Thomas
 
 
 
 


 --
 -
 Noble Paul




-- 
-
Noble Paul

Re: Replication not working

[Noble Paul നോബിള്‍ नोब्ळ्]

I mean , the log when polling happens when from slave. Not when you issue a
command.


On Tue, Jun 11, 2013 at 5:28 PM, thomas.poroc...@der.net wrote:

 Log on slave:

 2013-06-11 13:19:08,477 8385607 INFO  [org.apache.solr.core.SolrCore]
 (http-0.0.0.0-31006-1:) [contacts] webapp=/solr path=/replication
 params={indent=truecommand=indexversionswt=json+} status=0 QTime=0
 2013-06-11 13:19:08,477 8385607 DEBUG
 [org.apache.solr.servlet.SolrDispatchFilter] (http-0.0.0.0-31006-1:)
 Closing out SolrRequest: {indent=truecommand=indexversionswt=json+}
 2013-06-11 13:22:27,017 8584147 INFO  [org.apache.solr.core.SolrCore]
 (http-0.0.0.0-31006-1:) [contacts] webapp=/solr path=/replication
 params={command=indexversion} status=0 QTime=0
 2013-06-11 13:22:27,017 8584147 DEBUG
 [org.apache.solr.servlet.SolrDispatchFilter] (http-0.0.0.0-31006-1:)
 Closing out SolrRequest: {command=indexversion}

 -Ursprüngliche Nachricht-
 Von: Noble Paul നോബിള്‍ नोब्ळ् [mailto:noble.p...@gmail.com]
 Gesendet: Dienstag, 11. Juni 2013 13:41
 An: solr-user@lucene.apache.org
 Betreff: Re: Replication not working

 You said polling is happening and nothing is replicated

 What do the logs say on slave (Set level to INFO) ?




 On Tue, Jun 11, 2013 at 4:54 PM, thomas.poroc...@der.net wrote:

  Calling indexversion on master gives:
   response
lst name=responseHeader
   int name=status0/intint name=QTime0/int
/lst
long name=indexversion1370612995391/long
long name=generation53/long
   /response
 
  On Slave:
   response
 lst name=responseHeaderint name=status0/int
 int name=QTime0/int/lst
 long name=indexversion0/long
 long name=generation1/long
   /response
 
   pollInterval is set to 2 minutes. It is usually long
 
  I know ;-)
 
 
  -Ursprüngliche Nachricht-
  Von: Noble Paul നോബിള്‍ नोब्ळ् [mailto:noble.p...@gmail.com]
  Gesendet: Dienstag, 11. Juni 2013 13:16
  An: solr-user@lucene.apache.org
  Betreff: Re: Replication not working
 
  can you check with the indexversion command on both mater and slave?
 
  pollInterval is set to 2 minutes. It is usually long . So you may need to
  wait for 2 mins for the replication to kick in
 
 
  On Tue, Jun 11, 2013 at 3:21 PM, thomas.poroc...@der.net wrote:
 
   Hi all,
  
  
  
   we have a setup with multiple cores, loaded via DataImportHandlers.
  
   Works fine so far.
  
   Now we are trying to get the replication working (for one core so far).
   But the automated replication is never happening.
  
   Manually triggered replication works!
  
  
  
   Environment:
  
   Solr 4.1 (also tried with 4.3)
  
   App-Server JBoss 4.3.
  
   Java 1.6
  
  
  
   There are two JBoss instances running on different ports on the same
 box
   with their own solr.home directories.
  
  
  
   Configuration is done like described in the documentation:
  
  
  
   requestHandler name=/replication class=solr.ReplicationHandler 
  
lst name=master
  
 str
 name=enable${de.der.pu.solr.master.enable:false}/str
  
 str name=replicateAfterstartup/str
  
 str name=replicateAftercommit/str
  
 str name=replicateAfteroptimize/str
  
 str name=confFilesstopwords.txt, solrconfig.xml/str
  
/lst
  
lst name=slave
  
str name=enable${de.der.pu.solr.slave.enable:false}/str
  
 str
   name=masterUrlhttp://localhost:30006/solr/${solr.core.name}/str
  
 str name=pollInterfall00:02:00/str
  
/lst
  
 /requestHandler
  
  
  
   Basically it looks all fine from the admin-pages.
  
  
  
   The polling from the slave is going on but nothing happens.
  
   We have tried to delete slave index completely and restart both
 servers.
   Reimportet the master data several times and so on..
  
  
  
   On the masters replication page I see:
  
   - replication enable: true
  
   - replicateAfter: commit, startup
  
   - confFiles: stopwords.txt, solrconfig.xml
  
  
  
   On slave side I see:
  
   -masters version 1370612995391  53   2.56 MB
  
   -master url:  http://localhost:30006/solr/contacts
  
   -poling enable: true
  
  
  
   And master settings like on master side...
  
  
  
   When I enter
  
 http://localhost:30006/solr/contacts/replication?command=detailswt=json
   indent=true in the browser the response seems ok:
  
   {
 responseHeader:{
   status:0,
   QTime:0},
 details:{
   indexSize:2.56 MB,
  
  
 indexPath:D:\\usr\\local\\phx-unlimited\\jboss\\solr_cache\\test_pto_
   node1_solr\\contacts\\data\\index/,
   commits:[[
   indexVersion,1370612995391,
   generation,53,
   filelist,[_1r.fdt,
 _1r.fdx,
 _1r.fnm,
 _1r.nvd,
 _1r.nvm,
 _1r.si,
 _1r_Lucene41_0.doc,
 _1r_Lucene41_0.pos,
 _1r_Lucene41_0.tim,
 _1r_Lucene41_0.tip

Re: LotsOfCores feature

[Noble Paul നോബിള്‍ नोब्ळ्]

Aleksey,

It was a less than ideal situation. because we did not have a choice. We
had external systems/scripts to manage this. A new custom implementation is
being built on SolrCloud which would have taken care of most of hose
 issues.

SolrReplication is a hidden once you move to cloud. But it will continue in
the same way if you have a stand-lone deployment.


On Mon, Jun 10, 2013 at 1:20 AM, Aleksey bitterc...@gmail.com wrote:

 Thanks Paul. Just a little clarification:

 You mention that you migrate data using built-in replication, but if
 you map and route users yourself, doesn't that mean that you also need
 to manage replication yourself? Your routing logic needs to be aware
 of how to map both replicas for each user, and if one hosts goes down,
 then it needs to distribute traffic that it was receiving over other
 hosts. Same thing for adding more hosts.
 I did a couple of quick searches and found mostly older wikis that say
 solr replication will change in the future. Would you be able to point
 me to the right one?


 -

 On Fri, Jun 7, 2013 at 8:34 PM, Noble Paul നോബിള്‍  नोब्ळ्
 noble.p...@gmail.com wrote:
  We set it up like this
  + individual solr instances are setup
  + external mapping/routing to allocate users to instances. This
 information
  can be stored in an external data store
  + all cores are created as transient and loadonstart as false
  + cores come online on demand
  + as and when users data get bigger (or hosts are hot)they are migrated
  between less hit hosts using in built replication
 
  Keep in mind we had the schema for all users. Currently there is no way
 to
  upload a new schema to solr.
  On Jun 8, 2013 1:15 AM, Aleksey bitterc...@gmail.com wrote:
 
   Aleksey: What would you say is the average core size for your use
 case -
   thousands or millions of rows? And how sharded would each of your
   collections be, if at all?
 
  Average core/collection size wouldn't even be thousands, hundreds more
  like. And the largest would be half a million or so but that's a
  pathological case. I don't need sharding and queries than fan out to
  different machines. If fact I'd like to avoid that so I don't have to
  collate the results.
 
 
   The Wiki page was built not for Cloud Solr.
  
   We have done such a deployment where less than a tenth of cores were
  active
   at any given point in time. though there were tens of million indices
  they
   were split among a large no:of hosts.
  
   If you don't insist of Cloud deployment it is possible. I'm not sure
 if
  it
   is possible with cloud
 
  By Cloud you mean specifically SolrCloud? I don't have to have it if I
  can do without it. Bottom line is I want a bunch of small cores to be
  distributed over a fleet, each core completely fitting on one server.
  Would you be willing to provide a little more details on your setup?
  In particular, how are you managing the cores?
  How do you route requests to proper server?
  If you scale the fleet up and down, does reshuffling of the cores
  happen automatically or is it an involved manual process?
 
  Thanks,
 
  Aleksey
 




-- 
-
Noble Paul

Re: LotsOfCores feature

[Noble Paul നോബിള്‍ नोब्ळ्]

The Wiki page was built not for Cloud Solr.

We have done such a deployment where less than a tenth of cores were active
at any given point in time. though there were tens of million indices they
were split among a large no:of hosts.


If you don't insist of Cloud deployment it is possible. I'm not sure if it
is possible with cloud


On Fri, Jun 7, 2013 at 12:38 AM, Aleksey bitterc...@gmail.com wrote:

 I was looking at this wiki and linked issues:
 http://wiki.apache.org/solr/LotsOfCores

 they talk about a limit being 100K cores. Is that per server or per
 entire fleet because zookeeper needs to manage that?

 I was considering a use case where I have tens of millions of indices
 but less that a million needs to be active at any time, so they need
 to be loaded on demand and evicted when not used for a while.
 Also since number one requirement is efficient loading of course I
 assume I will store a prebuilt index somewhere so Solr will just
 download it and strap it in, right?

 The root issue is marked as won;t fix but some other important
 subissues are marked as resolved. What's the overall status of the
 effort?

 Thank you in advance,

 Aleksey




-- 
-
Noble Paul

Re: SOLR CSV output in custom order

[Noble Paul നോബിള്‍ नोब्ळ्]

Have you tried explicitly giving the field names (fl) as parameter
 http://wiki.apache.org/solr/CommonQueryParameters#fl


On Thu, Jun 6, 2013 at 12:41 PM, anurag.jain anurag.k...@gmail.com wrote:

 I want output of csv file in proper order.  when I use wt=csv  it gives
 output in random order. Is there any way to get output in proper format.

 Thanks



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/SOLR-CSV-output-in-custom-order-tp4068527.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
-
Noble Paul

Re: LotsOfCores feature

[Noble Paul നോബിള്‍ नोब्ळ्]

We set it up like this
+ individual solr instances are setup
+ external mapping/routing to allocate users to instances. This information
can be stored in an external data store
+ all cores are created as transient and loadonstart as false
+ cores come online on demand
+ as and when users data get bigger (or hosts are hot)they are migrated
between less hit hosts using in built replication

Keep in mind we had the schema for all users. Currently there is no way to
upload a new schema to solr.
On Jun 8, 2013 1:15 AM, Aleksey bitterc...@gmail.com wrote:

  Aleksey: What would you say is the average core size for your use case -
  thousands or millions of rows? And how sharded would each of your
  collections be, if at all?

 Average core/collection size wouldn't even be thousands, hundreds more
 like. And the largest would be half a million or so but that's a
 pathological case. I don't need sharding and queries than fan out to
 different machines. If fact I'd like to avoid that so I don't have to
 collate the results.


  The Wiki page was built not for Cloud Solr.
 
  We have done such a deployment where less than a tenth of cores were
 active
  at any given point in time. though there were tens of million indices
 they
  were split among a large no:of hosts.
 
  If you don't insist of Cloud deployment it is possible. I'm not sure if
 it
  is possible with cloud

 By Cloud you mean specifically SolrCloud? I don't have to have it if I
 can do without it. Bottom line is I want a bunch of small cores to be
 distributed over a fleet, each core completely fitting on one server.
 Would you be willing to provide a little more details on your setup?
 In particular, how are you managing the cores?
 How do you route requests to proper server?
 If you scale the fleet up and down, does reshuffling of the cores
 happen automatically or is it an involved manual process?

 Thanks,

 Aleksey

Re: Upgrading from SOLR 3.5 to 4.2.1 Results.

[Noble Paul നോബിള്‍ नोब्ळ्]

Actually  , It's pretty high end for most of the users. Rishi, u can post
the real h/w details and our typical deployment .
No :of cpus per node
No :of disks per host
Vms per host
Gc params
No :of cores per instance

Noble Paul
Sent from phone
On 21 May 2013 01:47, Rishi Easwaran rishi.easwa...@aol.com wrote:

 No, we just upgraded to 4.2.1.
 With the size of our complex and effort required apply our patches and
 rollout, our upgrades are not that often.






 -Original Message-
 From: Noureddine Bouhlel nouredd...@ecotour.com
 To: solr-user solr-user@lucene.apache.org
 Sent: Mon, May 20, 2013 3:36 pm
 Subject: Re: Upgrading from SOLR 3.5 to 4.2.1 Results.


 Hi Rishi,

 Have you done any tests with Solr 4.3 ?

 Regards,


 Cordialement,

 BOUHLEL Noureddine



 On 17 May 2013 21:29, Rishi Easwaran rishi.easwa...@aol.com wrote:

 
 
  Hi All,
 
  Its Friday 3:00pm, warm  sunny outside and it was a good week. Figured
  I'd share some good news.
  I work for AOL mail team and we use SOLR for our mail search backend.
  We have been using it since pre-SOLR 1.4 and strong supporters of SOLR
  community.
  We deal with millions indexes and billions of requests a day across our
  complex.
  We finished full rollout of SOLR 4.2.1 into our production last week.
 
  Some key highlights:
  - ~75% Reduction in Search response times
  - ~50% Reduction in SOLR Disk busy , which in turn helped with ~90%
  Reduction in errors
  - Garbage collection total stop reduction by over 50% moving application
  throughput into the 99.8% - 99.9% range
  - ~15% reduction in CPU usage
 
  We did not tune our application moving from 3.5 to 4.2.1 nor update java.
  For the most part it was a binary upgrade, with patches for our special
  use case.
 
  Now going forward we are looking at prototyping SOLR Cloud for our search
  system, upgrade java and tomcat, tune our application further. Lots of
 fun
  stuff :)
 
  Have a great weekend everyone.
  Thanks,
 
  Rishi.
 
 
 
 
 

Re: javabin binary format specification

[Noble Paul നോബിള്‍ नोब्ळ्]

There is no spec documented anywhere . It is all in this single file
https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java

On Wed, Jul 25, 2012 at 6:47 PM, Ahmet Arslan iori...@yahoo.com wrote:

  Sorry, but I could not find any spec on the binary format
  SolrJ is
  using. Can you point me to an URL if any?

 may be this?
 https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/core/src/java/org/apache/solr/response/BinaryResponseWriter.java




-- 
-
Noble Paul

Re: @field for child object

[Noble Paul നോബിള്‍ नोब्ळ्]

no

On Mon, Jul 4, 2011 at 3:34 PM, Kiwi de coder kiwio...@gmail.com wrote:
 hi,

 i wondering solrj @Field annotation support embedded child object ? e.g.

 class A {

  @field
  string somefield;

  @emebedded
  B b;

 }

 regards,
 kiwi




-- 
-
Noble Paul