daily SolrCloud collection wipes

[Werner Detter]

Hi,
 
I've got a SolrCloud instance with two collections running (Solr 7.7.2) on 
Debian Stretch VMs. Every morning round about 03:3* am the collection gets 
reset by $something and I have no clue what causes this and how to prevent 
it as there areeven no log entries in SolrCloud (even with increased log level)

It seems like it's some internal trigger. For the sake of testings I've
completely recreated[1]the collections yesterday from scratch but they've
been reset again.

Maybe anybody else experienced something similar and can give a hint how to
track down the source of the collection resets

Thanks,
Werner


[1]
curl 
"http://localhost:8983/solr/admin/collections?action=DELETE=$SOLR_COLLECTION;
/bin/su - solr -c "/opt/solr/bin/solr create -c $SOLR_COLLECTION -s 1 -rf 2"
/bin/bash /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost 
localhost:2181 -cmd putfile /configs/$SOLR_COLLECTION/schema.xml 
/root/$SOLR_COLLECTION/conf/schema.xml
/bin/bash /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost 
localhost:2181 -cmd putfile /configs/$SOLR_COLLECTION/solrconfig.xml 
/root/$SOLR_COLLECTION/conf/solrconfig.xml
/usr/bin/curl 
"http://localhost:8983/solr/admin/collections?action=MODIFYCOLLECTION=$SOLR_COLLECTION=1;

/usr/share/zookeeper/bin/zkCli.sh
rmr /configs/$SOLR_COLLECTION/managed-schema

Re: Anyway to encrypt admin user plain text password in Solr

[Jason Gerlowski]

Hi,

To clarify, Solr credentials are stored and shown in a few different
places.  In some situations the password might live in your
"solr.in.sh" file.  It also might live in a separate basicAuth.conf
file.  If you're using SolrCloud, the password might appear in Solr's
Admin UI (depending on your version of Solr).  The password is also
stored in ZooKeeper.

Some of these locations already store the credentials in an encrypted
form.  Other locations are only problematic if attackers have access
to the disk that Solr is running on, at which point you have much
bigger problems.

If you can be more specific about the exposure you're concerned about,
we can discuss whether there's an actual security concern there and
how to work around it.

Best,

Jason

On Wed, Nov 13, 2019 at 11:22 AM Kommu, Vinodh K.  wrote:
>
> Does anyone have an any idea on this? If so, please help.
>
> Thanks
> From: Kommu, Vinodh K.
> Sent: Monday, November 11, 2019 4:11 PM
> To: solr-user@lucene.apache.org
> Subject: Anyway to encrypt admin user plain text password in Solr
>
> Hi,
>
> After creating admin user in Solr when security is enabled, we have to store 
> the admin user's credentials in plain text format. Is there any option or a 
> way to encrypt the plain text password?
>
> Thanks,
> Vinodh
> DTCC DISCLAIMER: This email and any files transmitted with it are 
> confidential and intended solely for the use of the individual or entity to 
> whom they are addressed. If you have received this email in error, please 
> notify us immediately and delete the email and any attachments from your 
> system. The recipient should check this email and any attachments for the 
> presence of viruses. The company accepts no liability for any damage caused 
> by any virus transmitted by this email.

Re: Does Solr replicate data securely

[Tomás Fernández Löbbe]

Yes, if you are using TLS for running Solr, the replication will happen
using TLS

On Wed, Nov 13, 2019 at 2:45 PM Pushkar Raste 
wrote:

> Hi,
> Can some help me with my question.
>
> On Tue, Nov 12, 2019 at 10:20 AM Pushkar Raste 
> wrote:
>
> > Hi,
> > How about in the master/slave set up. If I enable ssl in master/slave
> > setup would the segment and config files be copied using TLS.
> >
> > On Sat, Nov 9, 2019 at 3:31 PM Jan Høydahl 
> wrote:
> >
> >> You choose. If you use solr cloud and have enabled ssl in your cluster,
> >> then all requests including replication will be secure (https). This it
> is
> >> still tcp but using TLS :)
> >>
> >> Jan Høydahl
> >>
> >> > 6. nov. 2019 kl. 00:03 skrev Pushkar Raste :
> >> >
> >> > Hi,
> >> > When slaves/pull replicas copy index files from master is done using
> an
> >> > secure protocol or just over tcp?
> >> > --
> >> > — Pushkar Raste
> >>
> > --
> — Pushkar Raste
>

Re: Does Solr replicate data securely

[Pushkar Raste]

Hi,
Can some help me with my question.

On Tue, Nov 12, 2019 at 10:20 AM Pushkar Raste 
wrote:

> Hi,
> How about in the master/slave set up. If I enable ssl in master/slave
> setup would the segment and config files be copied using TLS.
>
> On Sat, Nov 9, 2019 at 3:31 PM Jan Høydahl  wrote:
>
>> You choose. If you use solr cloud and have enabled ssl in your cluster,
>> then all requests including replication will be secure (https). This it is
>> still tcp but using TLS :)
>>
>> Jan Høydahl
>>
>> > 6. nov. 2019 kl. 00:03 skrev Pushkar Raste :
>> >
>> > Hi,
>> > When slaves/pull replicas copy index files from master is done using an
>> > secure protocol or just over tcp?
>> > --
>> > — Pushkar Raste
>>
> --
— Pushkar Raste

Re: Query More Than One Core

[rhys J]

On Wed, Nov 13, 2019 at 3:16 PM Jörn Franke  wrote:

> You can use nested indexing and Index both types of documents in one core.
>
> https://lucene.apache.org/solr/guide/8_1/indexing-nested-documents.html


I had read that, but it doesn't really fit our needs right now.

I figured out how to do a join like so:

http://localhost8983/solr/debt/select?indent=on=100=id
asc=(debt_id:570856 OR reference_no: *570856*)={!join from=debtor_id
to=debt_id fromIndex=dbtr}ssn1:12


However, what is the use case for Solr if you have already a database?
>

The use case is that we have an old search tool that uses the db, but it's
painfully slow, and it doesn't do fuzzy searches very well, or handle
things like searching for phone numbers without it relying on a lot of
regular expressions. A search engine speeds things up, and gets more
precise results.

Thanks,

Rhys

Re: Query More Than One Core

[Jörn Franke]

You can use nested indexing and Index both types of documents in one core.

https://lucene.apache.org/solr/guide/8_1/indexing-nested-documents.html

However, what is the use case for Solr if you have already a database?

> Am 13.11.2019 um 20:50 schrieb rhys J :
> 
> I have more than one core. Each core represents one database table.
> 
> They are coordinated by debt_id/debtor_id, so we can do join statements on
> them with Sybase/SQL.
> 
> Is there a way to query more than one core at a time, or do I need to do
> separate queries per core, and then somehow with perl aggregate them into
> one list?
> 
> Thanks,
> 
> Rhys

Re: Need some help on solr versions (LTS vs stable)

[Adam Walz]

The LTS idea I believe comes from the solr downloads page where 7.7.x is
designated as LTS. https://lucene.apache.org/solr/downloads.html

On Wed, Nov 13, 2019 at 9:41 AM Shawn Heisey  wrote:

> On 11/6/2019 9:58 AM, suyog joshi wrote:
> > So we can say its better to go with latest stable version (8.x) instead
> of
> > 7.x, which is LTS right now, but can soon become EOL post launching of
> 9.x
> > sometime early next year.
>
> I don't know where you got the idea that 7.x is LTS ... but I do not
> think that is correct.  I don't think we have a version that could be
> called LTS, at least not the way I have seen the term used.
>
> It's true that 7.x currently is in a state where it is unlikely to have
> its feature list changed, which could be seen as stability.  But chances
> are that if you DO run into a bug with a 7.x version, the fix for that
> problem will probably only make it into the current stable branch, so
> you'd be upgrading to at least an 8.x version in order to obtain the fix.
>
> Changing to an LTS model would mean changes to the way development is
> done on the project.  Change is always scary.  I've asked on the dev
> list about this.
>
> Thanks,
> Shawn
>


-- 
Adam Walz

Query More Than One Core

[rhys J]

I have more than one core. Each core represents one database table.

They are coordinated by debt_id/debtor_id, so we can do join statements on
them with Sybase/SQL.

Is there a way to query more than one core at a time, or do I need to do
separate queries per core, and then somehow with perl aggregate them into
one list?

Thanks,

Rhys

Re: Full-text search for Solr manual

[Alexandre Rafalovitch]

Try: site:lucene.apache.org inurl:8_2 luceneMatchVersion
(8.3 does not work, seems to be not fully? indexed by google yet)

https://github.com/apache/lucene-solr/search?l=AsciiDoc=luceneMatchVersion
(latest development version only).

You can read the rendered documents (without extra processing we do),
right on GitHub:
https://github.com/apache/lucene-solr/blob/branch_8_3/solr/solr-ref-guide/src/blockjoin-faceting.adoc

Regards,
  Alex.

On Wed, 13 Nov 2019 at 17:23, Luke Miller  wrote:
>
> Thanks Alex,
>
>
>
> For your response.
>
>
>
> Unfortunately the Solr source does not ship with the source of the manual.
> (Directory /docs only contains a link to the online manual.)
>
>
>
> Google search with domain limitation does not give any results, as mentioned
> in my initial post. Any other limitation does not filter for a specific
> version. E.g. https://www.google.de/search?q=
>  //lucene.apache.org/%20luceneMatchVersion>
> "Solr%20Ref%20Guide%208.3"%20site:https://lucene.apache.org/%20luceneMatchVe
> rsion
>
>
>
> I ended up downloading the whole documentation manually:
>
> wget --timeout=1 --tries=5 --cut-dirs=3 -mkpnp -nH -P solr-8.3 -e robots=off
> https://lucene.apache.org/solr/guide/8_3/
>
>
>
> And then I have to grep. A plain PDF file would be so much more convenient!
> Of course a Solr-enabled search for the online manual would work as well.
>
>
>
> Thanks,
>
> Julian
>
>
>
>
>
> >Grep on the source of the manual (which ships with Solr source).
>
> >
>
> >Google search with domain or keywords limitations.
>
> >
>
> >Online copy searching is not powered by Solr yet. Yes, we are aware of the
>
> >irony and are discussing it.
>
> >
>
> >Regards,
>
> >Aled
>
> >
>
> >On Tue, Nov 12, 2019, 1:25 AM Luke Miller wrote:
>
> >
>
> >> Hi,
>
> >>
>
> >>
>
> >>
>
> >> I just noticed that since Solr 8.2 the Apache Solr Reference Guide is not
>
> >> available anymore as PDF.
>
> >>
>
> >>
>
> >>
>
> >> Is there a way to perform a full-text search using the HTML manual? E.g.
>
> >> I'd
>
> >> like to find every hit for "luceneMatchVersion".
>
> >>
>
> >>
>
> >>
>
> >> *   Using the integrated "Page title lookup." does not find anything
> (
>
> >> -
>
> >> sure, it only looks up page titles. )
>
> >> *   Google does not return anything either searching for:
>
> >> site:https://lucene.apache.org/solr/guide/8_3/ luceneMatchVersion
>
> >>
>
> >>
>
> >>
>
> >> Is there another search method I missed?
>
> >>
>
> >>
>
> >>
>
> >> Thanks.
>
> >>
>
> >>
>
>
>

Re: EmbeddedSolrServer thread-safety

[Kaminski, Adi]

Hi Shawn,
Thanks a lot for the clarification !!


Regards,
Adi

Sent from Workspace ONE Boxer

On Nov 13, 2019 19:09, Shawn Heisey  wrote:
On 11/13/2019 9:45 AM, Kaminski, Adi wrote:
> Can someone confirm that EmbeddedSolrServer calss is thread-safe (meaning 
> from API perspective of add/delete/query operations by different threads)?
> It's not written in any documentation, and we have some different answers 
> from different forums.
>
> The question is regarding Solr 7.6 binaries (in case was changed fixed in 
> later versions).

EmbeddedSolrServer is Solr, without all the http-related stuff normally
provided by Jetty.  Solr is thread-safe, and since the embedded server
runs exactly the same core code to provide an index as Solr does, I
would expect it to be thread-safe as well.

Note that we strongly recommend NOT using the embedded server in
production.  Achieving redundancy with the embedded server would require
significant extra work, mostly to rebuild the remote capability that
Jetty gives to a full Solr install.

Thanks,
Shawn


This electronic message may contain proprietary and confidential information of 
Verint Systems Inc., its affiliates and/or subsidiaries. The information is 
intended to be for the use of the individual(s) or entity(ies) named above. If 
you are not the intended recipient (or authorized to receive this e-mail for 
the intended recipient), you may not use, copy, disclose or distribute to 
anyone this message or any information contained in this message. If you have 
received this electronic message in error, please notify us by replying to this 
e-mail.

Re: date fields and invalid date string errors

[rhys J]

> You could do it that way ... but instead, I'd create a new fieldType,
> not change an existing one.  The existing name is "pdate" which implies
> "point date".  I would probably go with "daterange" or "rdate" as the
> name, but that is completely up to you.
>
>
I did that, deleted docs, stopped, started solr, and then re-indexed. And
it's working like I expect it to.

Thanks for the help.

Rhys

Re: date fields and invalid date string errors

[Shawn Heisey]

On 11/13/2019 10:45 AM, rhys J wrote:

If you use DateRangeField instead of DatePointField for your field's
class, then you can indeed use partial timestamps for both indexing and
querying.  This only works with DateRangeField.



I don't see that as an option in the API? Do I need to change what pdate's
type is in the managed-schema for it to take effect?

As in:

  

to

  


You could do it that way ... but instead, I'd create a new fieldType, 
not change an existing one.  The existing name is "pdate" which implies 
"point date".  I would probably go with "daterange" or "rdate" as the 
name, but that is completely up to you.


Note that trying to use an existing index with a new schema is not going 
to work.  You're going to need to delete the old index and build it 
again after you change the schema.


Thanks,
Shawn

Re: date fields and invalid date string errors

[rhys J]

> If you use DateRangeField instead of DatePointField for your field's
> class, then you can indeed use partial timestamps for both indexing and
> querying.  This only works with DateRangeField.
>
>
I don't see that as an option in the API? Do I need to change what pdate's
type is in the managed-schema for it to take effect?

As in:

 

to

 

Thanks,

Rhys

Re: Need some help on solr versions (LTS vs stable)

[Shawn Heisey]

On 11/6/2019 9:58 AM, suyog joshi wrote:

So we can say its better to go with latest stable version (8.x) instead of
7.x, which is LTS right now, but can soon become EOL post launching of 9.x
sometime early next year.


I don't know where you got the idea that 7.x is LTS ... but I do not 
think that is correct.  I don't think we have a version that could be 
called LTS, at least not the way I have seen the term used.


It's true that 7.x currently is in a state where it is unlikely to have 
its feature list changed, which could be seen as stability.  But chances 
are that if you DO run into a bug with a 7.x version, the fix for that 
problem will probably only make it into the current stable branch, so 
you'd be upgrading to at least an 8.x version in order to obtain the fix.


Changing to an LTS model would mean changes to the way development is 
done on the project.  Change is always scary.  I've asked on the dev 
list about this.


Thanks,
Shawn

Re: date fields and invalid date string errors

[Shawn Heisey]

On 11/13/2019 10:07 AM, rhys J wrote:

I have date fields in my documents that are just -MM-DD.


and



When I use the API to do a search and try:

2018-01-01
[2018-01-01 TO NOW]

I get 'Invalid Date String'.

Did I type my data wrong in the schema? Is there something I'm missing from
the field itself?

According to this page, I should be able to query on just  or -MM
or -MM-DD.


With DatePointField, you must use full date strings, that include time 
down to the second.  Here's an example.


2018-01-01T00:00:00

If you use DateRangeField instead of DatePointField for your field's 
class, then you can indeed use partial timestamps for both indexing and 
querying.  This only works with DateRangeField.


Thanks,
Shawn

Re: Question about startup memory usage

[Shawn Heisey]

On 11/13/2019 2:03 AM, Hongxu Ma wrote:

I have a solr-cloud cluster with a big collection, after startup (no any 
search/index operations), its jvm memory usage is 9GB (via top: RES).

Cluster and collection info:
each host: total 64G mem, two solr nodes with -xmx=15G
collection: total 9B billion docs (but each doc is very small: only some 
bytes), total size 3TB.

My question is:
Is the 9G mem usage after startup normal? If so, I am worried that the follow 
up index/search operations will cause an OOM error.
And how can I reduce the memory usage? Maybe I should introduce more host with 
nodes, but besides this, is there any other solution?


With the "-Xmx=15G" option, you've told Java that it can use up to 15GB 
for heap.  It's total resident memory usage is eventually going to reach 
a little over 15GB and probably never go down.  This is how Java works.


The amount of memory that Java allocates immediately on program startup 
is related to the -Xms setting.  Normally Solr uses the same number for 
both -Xms and -Xmx, but that can be changed if you desire.  We recommend 
using the same number.  If -Xms is smaller than -Xmx, Java may allocate 
less memory as soon as it starts, then Solr is going to run through its 
startup procedure.  We will not know exactly how much memory allocation 
is going to occur when that happens ... but with billions of documents, 
it's not going to be small.


Thanks,
Shawn

Re: EmbeddedSolrServer thread-safety

[Shawn Heisey]

On 11/13/2019 9:45 AM, Kaminski, Adi wrote:

Can someone confirm that EmbeddedSolrServer calss is thread-safe (meaning from 
API perspective of add/delete/query operations by different threads)?
It's not written in any documentation, and we have some different answers from 
different forums.

The question is regarding Solr 7.6 binaries (in case was changed fixed in later 
versions).


EmbeddedSolrServer is Solr, without all the http-related stuff normally 
provided by Jetty.  Solr is thread-safe, and since the embedded server 
runs exactly the same core code to provide an index as Solr does, I 
would expect it to be thread-safe as well.


Note that we strongly recommend NOT using the embedded server in 
production.  Achieving redundancy with the embedded server would require 
significant extra work, mostly to rebuild the remote capability that 
Jetty gives to a full Solr install.


Thanks,
Shawn

date fields and invalid date string errors

[rhys J]

I have date fields in my documents that are just -MM-DD.

I set them as a pdate field in the schema as such:



and



When I use the API to do a search and try:

2018-01-01
[2018-01-01 TO NOW]

I get 'Invalid Date String'.

Did I type my data wrong in the schema? Is there something I'm missing from
the field itself?

According to this page, I should be able to query on just  or -MM
or -MM-DD.

https://lucene.apache.org/solr/guide/6_6/working-with-dates.html

Thanks,

Rhys

Re: Full-text search for Solr manual

[Shawn Heisey]

On 11/12/2019 11:15 PM, Luke Miller wrote:

Unfortunately the Solr source does not ship with the source of the manual.
(Directory /docs only contains a link to the online manual.)


The docs directory is not where the manual lives.  In the source code 
download, the solr/docs directory has an HTML version of the CHANGES.txt 
file.  A clone of the git repository will not contain a docs directory 
at all.  If you see a /docs directory at the top level of the download, 
especially if it contains a link the the online guide, then you do not 
have the source code, you have a binary release of Solr.


If you either download the source tarball from the Solr website or pull 
the source down from the git repository, you will find the manual in 
solr/solr-ref-guide.  With a git repository, you can use "git grep XXX" 
and with either method you can use "grep -r XXX solr/solr-ref-guide".


We'd love to use Solr to power the online ref guide search, but there 
are some hurdles to doing that.  The infrastructure that Apache has 
provided to its projects is extensive, but doesn't currently accommodate 
that kind of setup.


Thanks,
Shawn

EmbeddedSolrServer thread-safety

[Kaminski, Adi]

Hi,
Can someone confirm that EmbeddedSolrServer calss is thread-safe (meaning from 
API perspective of add/delete/query operations by different threads)?
It's not written in any documentation, and we have some different answers from 
different forums.

The question is regarding Solr 7.6 binaries (in case was changed fixed in later 
versions).

Regards,
Adi




This electronic message may contain proprietary and confidential information of 
Verint Systems Inc., its affiliates and/or subsidiaries. The information is 
intended to be for the use of the individual(s) or entity(ies) named above. If 
you are not the intended recipient (or authorized to receive this e-mail for 
the intended recipient), you may not use, copy, disclose or distribute to 
anyone this message or any information contained in this message. If you have 
received this electronic message in error, please notify us by replying to this 
e-mail.

RE: Anyway to encrypt admin user plain text password in Solr

[Kommu, Vinodh K.]

Does anyone have an any idea on this? If so, please help.

Thanks
From: Kommu, Vinodh K.
Sent: Monday, November 11, 2019 4:11 PM
To: solr-user@lucene.apache.org
Subject: Anyway to encrypt admin user plain text password in Solr

Hi,

After creating admin user in Solr when security is enabled, we have to store 
the admin user's credentials in plain text format. Is there any option or a way 
to encrypt the plain text password?

Thanks,
Vinodh
DTCC DISCLAIMER: This email and any files transmitted with it are confidential 
and intended solely for the use of the individual or entity to whom they are 
addressed. If you have received this email in error, please notify us 
immediately and delete the email and any attachments from your system. The 
recipient should check this email and any attachments for the presence of 
viruses. The company accepts no liability for any damage caused by any virus 
transmitted by this email.

Re: Full-text search for Solr manual

[Erick Erickson]

The source tarball is what you need.

Go to the download page, click the “Apache Archive” link, thence to the version 
you care about and download the source. All the docs have an “.adoc” extension. 
The Atom editor will render those pages for you, but they’re not complicated 
markup.

Best,
Erick

> On Nov 13, 2019, at 1:15 AM, Luke Miller  wrote:
> 
> Thanks Alex, 
> 
> 
> 
> For your response.
> 
> 
> 
> Unfortunately the Solr source does not ship with the source of the manual.
> (Directory /docs only contains a link to the online manual.)
> 
> 
> 
> Google search with domain limitation does not give any results, as mentioned
> in my initial post. Any other limitation does not filter for a specific
> version. E.g. https://www.google.de/search?q=
>  //lucene.apache.org/%20luceneMatchVersion>
> "Solr%20Ref%20Guide%208.3"%20site:https://lucene.apache.org/%20luceneMatchVe
> rsion
> 
> 
> 
> I ended up downloading the whole documentation manually:
> 
> wget --timeout=1 --tries=5 --cut-dirs=3 -mkpnp -nH -P solr-8.3 -e robots=off
> https://lucene.apache.org/solr/guide/8_3/
> 
> 
> 
> And then I have to grep. A plain PDF file would be so much more convenient!
> Of course a Solr-enabled search for the online manual would work as well.
> 
> 
> 
> Thanks,
> 
> Julian
> 
> 
> 
> 
> 
>> Grep on the source of the manual (which ships with Solr source).
> 
>> 
> 
>> Google search with domain or keywords limitations.
> 
>> 
> 
>> Online copy searching is not powered by Solr yet. Yes, we are aware of the
> 
>> irony and are discussing it.
> 
>> 
> 
>> Regards,
> 
>>   Aled
> 
>> 
> 
>> On Tue, Nov 12, 2019, 1:25 AM Luke Miller wrote:
> 
>> 
> 
>>> Hi,
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> I just noticed that since Solr 8.2 the Apache Solr Reference Guide is not
> 
>>> available anymore as PDF.
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> Is there a way to perform a full-text search using the HTML manual? E.g.
> 
>>> I'd
> 
>>> like to find every hit for "luceneMatchVersion".
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> *   Using the integrated "Page title lookup." does not find anything
> (
> 
>>> -
> 
>>> sure, it only looks up page titles. )
> 
>>> *   Google does not return anything either searching for:
> 
>>> site:https://lucene.apache.org/solr/guide/8_3/ luceneMatchVersion
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> Is there another search method I missed?
> 
>>> 
> 
>>> 
> 
>>> 
> 
>>> Thanks.
> 
>>> 
> 
>>> 
> 
> 
> 

Question about startup memory usage

[Hongxu Ma]

Hi
I have a solr-cloud cluster with a big collection, after startup (no any 
search/index operations), its jvm memory usage is 9GB (via top: RES).

Cluster and collection info:
each host: total 64G mem, two solr nodes with -xmx=15G
collection: total 9B billion docs (but each doc is very small: only some 
bytes), total size 3TB.

My question is:
Is the 9G mem usage after startup normal? If so, I am worried that the follow 
up index/search operations will cause an OOM error.
And how can I reduce the memory usage? Maybe I should introduce more host with 
nodes, but besides this, is there any other solution?

Thanks.