Re: Indexing with customized parameters

2019-12-12 Thread Anuj Bhargava 
Emir Thanks,

Perfect

On Thu, 12 Dec 2019 at 13:40, Emir Arnautović 
wrote:

> Hi Anuj,
> Maybe I am missing something but this is more question for some SQL group
> than for Solr group. I am surprised that you get any records. You can
> consult your DB documentation for some more elegant solution, but a
> brute-force solution, if your column is string, could be:
> WHERE sector = 27 OR sector LIKE ’27,%’ OR sector LIKE ‘%,27,%’ OR sector
> LIKE ‘%,27’ OR sector = 2701…
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 12 Dec 2019, at 08:38, Anuj Bhargava  wrote:
> >
> > Any suggestions?
> >
> > Regards,
> >
> > Anuj
> >
> > On Tue, 10 Dec 2019 at 20:52, Anuj Bhargava  wrote:
> >
> >> I am trying to index where the *sector field* has the values 27 and/or
> >> 2701 and/or 2702 using the following -
> >>
> >>  >>  query="SELECT * FROM country WHERE sector = 27 OR sector = 2701 OR
> >> sector = 2702"
> >>  deltaImportQuery="SELECT * FROM country
> >>WHERE posting_id = '${dataimporter.delta.posting_id}' AND sector = 27
> >> OR sector = 2701 OR sector = 2702"
> >>  deltaQuery="SELECT posting_id FROM country
> >>WHERE last_modified > '${dataimporter.last_index_time}' AND sector =
> >> 27 OR sector = 2701 OR sector = 2702">
> >> 
> >>
> >> The sector field has comma separated multiple values like -
> >> 27,19,527
> >> 38,27,62701
> >> 2701,49
> >> 55,2702,327
> >>
> >> The issue is when I run the above, it indexes the fields containing data
> >> 27,19,527 and 2701,49 and ignores the other data. It indexes if the
> data in
> >> the sector fields starts with either 27 or 2701 or 2702. It doesn't
> index
> >> if the values 27 or 2701 or 2702 are placed 2nd or 3rd in the sector
> data
> >> field
> >>
>
>

Sometimes searching slow in Solr 6.1.0

2019-12-12 Thread vishal patel 
We have 2 shards and 2 replicas in our live environment. Total of 26 
collections. we give 64GB RAM for a single Solr instance.
I have faced a slow searching issue in our live environment. In our scenario, 
there are many update requests come within minutes like 50,000. At that time 
searching becomes slow.
The query is normal but taking a 4to5 seconds. When the same query will execute 
after sometimes it will not take time.
Our solr config details:
* autoCommit is 6 and autoSoftCommit is 100.
* No caching add in config.

Can we handle the update thread priority? why searching is slow at that time? 
can we monitor updates and search requests for solr cloud performance?


Regards,
Vishal

Re: Atomic solrj update

2019-12-12 Thread Shawn Heisey 

On 12/12/2019 10:00 PM, Prem wrote:

I am trying to partially update of 50M data in a collection from CSV using
Atomic script(solrj).But it is taking 2 hrs for 1M records.is there anyway i
can speed up my update.


How many documents are you sending in one request?


Using HTTPClient to establish connection and also i am validating whether
the particular document is available in collection or not and after that
updating the document.


I thought you were using SolrJ ... but here you say you're using HTTPClient.

Can you share your code?  What Solr server version? If you're using 
SolrJ, what version of that?


If your program checks whether every single document already exists 
before sending an update, that is going to be quite slow.


Thanks,
Shawn

Re: Atomic solrj update

2019-12-12 Thread Jörn Franke 
One needs to see the code or get more insights on your design. Do you reuse the 
HTTPClient or do you create for every request a new one?
How often do you commit?
Do you do parallel updates from the client (multiple threads?).

> Am 13.12.2019 um 06:56 schrieb Prem :
> 
> I am trying to partially update of 50M data in a collection from CSV using
> Atomic script(solrj).But it is taking 2 hrs for 1M records.is there anyway i
> can speed up my update.
> Using HTTPClient to establish connection and also i am validating whether
> the particular document is available in collection or not and after that
> updating the document.
> 
> 
> 
> 
> --
> Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Atomic solrj update

2019-12-12 Thread Prem 
I am trying to partially update of 50M data in a collection from CSV using
Atomic script(solrj).But it is taking 2 hrs for 1M records.is there anyway i
can speed up my update.
Using HTTPClient to establish connection and also i am validating whether
the particular document is available in collection or not and after that
updating the document.




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: A question of solr recovery

2019-12-12 Thread Hongxu Ma 
Thank you @Erick Erickson for your explanation! (although I don't fully 
understand all details  ).

I am using Solr 6.6, so I think there is only NRT replica in this version, and 
I understand the whole recovery process now.

Maybe I will upgrade to Solr 7+ in future, and try the new TLOG/PULL replica.
Thanks.



From: Erick Erickson 
Sent: Thursday, December 12, 2019 22:49
To: Hongxu Ma 
Subject: Re: A question of solr recovery

If you’re using TLOG/PULL replica types, then only changed segments
are downloaded. That replication pattern has a very different
algorithm. The problem with NRT replicas is that segments on
different replicas may not contain the same documents (in fact,
almost all the time won’t). This is because the wall-clock time
that the autocommit interval expires at, which closes segments,
will be different due to network delays and the like. This was
a deliberate design choice to make indexing as fast as possible
in distributed situations. If the leader coordinated all the commits,
it’d introduce a delay, potentially quite long if, say, the leader
needed to wait for a timeout.

Even if commits were exactly synchronous over all replicas in a shard,
the leader indexes a document and forwards it to the replica. The
commit could expire on both while the doc was in-flight.

Best,
Erick

On Dec 12, 2019, at 5:37 AM, Hongxu Ma  wrote:

Thank you very much @Erick Erickson
It's very clear.

And I found my "full sync" log:
"IndexFetcher Total time taken for download 
(fullCopy=true,bytesDownloaded=178161685180) : 4377 secs (40704063 bytes/sec) 
to NIOFSDirectory@..."

A more question:
Form the log, looks it downloaded all segment files (178GB), it's very big and 
took a long time.
Is it possible only download the segment file which contains the missing part? 
No need all files, maybe it can save time?

For example, there is my fabricated algorithm (like database does):
• recovery form local tlog as much as possible
• calculate the latest version
• only download the segment file which contains data > this version
Thanks.

From: Erick Erickson 
Sent: Wednesday, December 11, 2019 20:56
To: solr-user@lucene.apache.org 
Subject: Re: A question of solr recovery

Updates in this context are individual documents, either new ones
or a new version of an existing document. Long recoveries are
quite unlikely to be replaying a few documents from the tlog.

My bet is that you had to do a “full sync” (there should be messages
to that effect in the Solr log). This means that the replica had to
copy the entire index from the leader, and that varies with the size
of the index, network speed and contention, etc.

And to make it more complicated, and despite the comment about 100
docs and the tlog…. while that copy is going on, _new_ updates are
written to the tlog of the recovering replica and after the index
has been copied, those new updates are replayed locally. The 100
doc limit does _not_ apply in this case. So say the recovery starts
at time T and lasts for 60 seconds. All updates sent to the shard
leader over that 60 seconds are put in the local tlog and after the
copy is done, they’re replayed. And then, you guessed it, any
updates received by the leader over that 60 second period are written
to the recovering replica’s tlog and replayed… Under heavy
indexing loads, this can go no for quite a long time. Not certain
that’s what’s happening, but something to be aware of.

Best,
Erick

On Dec 10, 2019, at 10:39 PM, Hongxu Ma  wrote:

Hi all
In my cluster, Solr node turned into long time recovery sometimes.
So I want to know more about recovery and have read a good blog:
https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

It mentioned in the recovery section:
"Replays the documents from its own tlog if < 100 new updates have been 
received by the leader. "

My question: what's the meaning of "updates"? commits? or documents?
I refered solr code but still not sure about it.

Hope you can help, thanks.

Re: Solr8 changes how security.json restricts access to GUI

2019-12-12 Thread Jan Høydahl 
Attachments are stripped from list, can you post a link to the screenshot of 
the UI when you first visit?

Jan

> 12. des. 2019 kl. 17:27 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
> :
> 
> Below is the security.json (with password hashes redacted): in Solr7.4 it 
> prompts for a password and (if you get it right) lets you into the whole GUI; 
> But in Solr8.1.1 and in Solr 8.3, it does not prompt for a password before 
> letting you into a crippled version of the GUI (as depicted in the attachment)
> 
> {
>  "authentication":{
>"class":"solr.BasicAuthPlugin",
>"credentials":{
>  "solradmin":"[redacted]",
>  "pysolrmon":"[redacted]",
>  "solrtrg":"[redacted]"},
>"":{"v":2}},
>  "authorization":{
>"class":"solr.RuleBasedAuthorizationPlugin",
>"user-role":{
>  "solradmin":[
>"admin",
>"allgen",
>"trgadmin",
>"genadmin"],
>  "solrtrg":[
>"trgadmin",
>"allgen"],
>  "pysolrmon":["clustatus_role"]},
>"permissions":[
>  {
>"name":"gen_admin",
>"collection":"NULL",
>"path":"/admin/cores",
>"params":{"action":[
>"REGEX:(?i)CREATE",
>"REGEX:(?i)RENAME",
>"REGEX:(?i)SWAP",
>"REGEX:(?i)UNLOAD",
>"REGEX:(?i)SPLIT"]},
>"role":"genadmin"},
>  {
>"name":"col_admin",
>"collection":null,
>"path":"/admin/collections",
>"params":{"action":[
>"REGEX:(?i)CREATE",
>"REGEX:(?i)MODIFYCOLLECTION",
>"REGEX:(?i)SPLITSHARD",
>"REGEX:(?i)CREATESHARD",
>"REGEX:(?i)DELETESHARD",
>"REGEX:(?i)CREATEALIAS",
>"REGEX:(?i)DELETEALIAS",
>"REGEX:(?i)DELETE",
>"REGEX:(?i)DELETEREPLICA",
>"REGEX:(?i)ADDREPLICA",
>"REGEX:(?i)CLUSTERPROP",
>"REGEX:(?i)MIGRATE",
>"REGEX:(?i)ADDROLE",
>"REGEX:(?i)REMOVEROLE",
>"REGEX:(?i)ADDREPLICAPROP",
>"REGEX:(?i)DELETEREPLICAPROP",
>"REGEX:(?i)BALANCESHARDUNIQUE",
>"REGEX:(?i)REBALANCELEADERS",
>"REGEX:(?i)FORCELEADER",
>"REGEX:(?i)MIGRATESTATEFORMAT"]},
>"role":"genadmin"},
>  {
>"name":"security-edit",
>"role":"admin"},
>  {
>"name":"clustatus",
>"path":"/admin/collections",
>"params":{"action":["REGEX:(?i)CLUSTERSTATUS"]},
>"role":[
>  "clustatus_role",
>  "allgen"],
>"collection":null},
>  {
>"name":"corestatus",
>"path":"/admin/cores",
>"params":{"action":["REGEX:(?i)STATUS"]},
>"role":[
>  "allgen",
>  "clustatus_role"],
>"collection":null},
>  {
>"name":"trgadmin",
>"collection":"trg_col",
>"path":"/admin/*",
>"role":"trgadmin"},
>  {
>"name":"open_select",
>"path":"/select/*",
>"role":null},
>  {
>"name":"open_search",
>"path":"/search/*",
>"role":null},
>  {
>"name":"catch-all-nocollection",
>"collection":null,
>"path":"/*",
>"role":"allgen"},
>  {
>"name":"catch-all-collection",
>"path":"/*",
>"role":"allgen"},
>  {
>"name":"all-admincol",
>"collection":null,
>"path":"/admin/collections",
>"role":"allgen"},
>  {
>"name":"all-admincores",
>"collection":null,
>"path":"/admin/cores",
>"role":"allgen"}],
>"":{"v":5}}}
> 
> -Original Message-
> From: Jan Høydahl  
> Sent: Wednesday, December 11, 2019 7:35 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr8 changes how security.json restricts access to GUI
> 
> Please show your complete Security.json so we know how auth is configured. 
> Which 8.x version are you trying? There should be a login screen shown in 
> admin UI now.
> 
> Jan Høydahl
> 
>> 11. des. 2019 kl. 22:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
>> :
>> 
>> In Solr 7, we had clauses in our security.json saying
>> 
>> {
>>   "name":"all-admin",
>>   "collection":null,
>>   "path":"/*",
>>   "role":"allgen",
>>   "index":15},
>> {
>>   "name":"all-core-handlers",
>>   "path":"/*",
>>   "role":"allgen",
>>   "index":16},
>> 
>> We granted the role allgen to all users; but this kept our security folk 
>> happy in that no one could even get to the top level of the Solr GUI without 
>> a password.
>> 
>> Now under Solr 8, the GUI does not prompt for a password. It just brings you 
>> into the GUI (albeit a stripped down version, saying such things as "No 
>> cores available"). By what means can we require a password to get this far? 
>> And by what means can we prompt for a password in order to get further?

Re: Facing jwt authentication problem using solr 8.1.1

2019-12-12 Thread Jan Høydahl 
Try something like this 
https://gist.github.com/b330e1bea7842bcdc1e5fa3940b4a4f7 


The trick is to «whitelist» certain paths that will not require auth, but then 
further down add rules to block all other paths either as admin role or with 
special role *»* which means «any authenticated user».

Jan

> 12. des. 2019 kl. 07:47 skrev Lakhan Gupta 
> :
> 
> Hi,
> 
> Using solr 8.1.1 version and facing problem while enabling jwt authentication 
> in solr. Jwt authentication is working fine after configuring security.json 
> file. Below is the configuration I am using for enabling jwt authentication.
> 
> Security.json
> 
> {
>  "authentication":{
>   "blockUnknown": false,
>"class":"solr.JWTAuthPlugin",
>   "jwk":{
>  "kty":"oct",
>  "use":"sig",
>  "kid":"k1",
>  
> "k":"7A02618BE6943C22FD81CAB9F6FCF063B6E1732C3614BC3ACA6032B6B3215CAF0D28A34FD423423CA3AC34BEA27D3F79",
>  "alg":"HS256"},
>"aud":"solr"},
>   "authorization":{
>  "class":"solr.RuleBasedAuthorizationPlugin",
>  "permissions":[
>  {
>"name":"all",
> "path":"/*",
>"role":"admin"
> }
>  ],
>  "user-role":{
> "solr":"admin"
>  }
>   }
> }
> 
> Using secret key
> 7A02618BE6943C22FD81CAB9F6FCF063B6E1732C3614BC3ACA6032B6B3215CAF0D28A34FD423423CA3AC34BEA27D3F79
> 
> Jwt token is generated:
> eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhZCIsImF1ZCI6InNvbHIiLCJleHAiOjk5MTYyMzkwMjJ9.M4PksJTJ9gFjOlvvFmG1eDSyXDtKIRSGIYicIW9hwT4
> 
> Below header and payload I'm using for generate jwt token :
> 
> The header is
> {
>  "alg": "HS256",
>  "typ": "JWT"
> }
> 
> and the payload is
> 
> {
>  "sub": "admin",
>  "aud": "Solr",
>  "exp": 9916239022
> }
> 
> With above configuration my jwt authentication is working fine. But there is 
> a problem when request is sent without authentication in header the api still 
> retrieving data. I want to prevent it when request come without 
> authentication header.
> 
> For that, I've enabled blockUnknown parameter in security.json file. That 
> works fine and authentication request is required. But, after enabling 
> blockunknown  parameter I am facing below exception while starting solr using 
> solr start command.
> 
> 
> ERROR: Solr requires authentication for 
> http://localhost:8983/solr/admin/info/system. Please supply valid 
> credentials. HTTP code=401
> 
> I've googled a lot and find out
> 
> solr/admin/info/system endpoint required authentication.
> 
> How to authenticate solr/admin/info/system endpoint while startup solr?
> 
> Need urgent help. I'd appreciate if someone can help me.
> 
> Thanks
> Lakhan Gupta
> 
> 
> 
> The information in this email is confidential and may be legally privileged. 
> It is intended solely for the addressee and access to it by anyone else is 
> unauthorized. If you are not the intended recipient, any disclosure, copying, 
> distribution or any action taken or omitted to be taken based on it, is 
> strictly prohibited and may be unlawful.

RE: Solr8 changes how security.json restricts access to GUI

2019-12-12 Thread Oakley, Craig (NIH/NLM/NCBI) [C] 
Below is the security.json (with password hashes redacted): in Solr7.4 it 
prompts for a password and (if you get it right) lets you into the whole GUI; 
But in Solr8.1.1 and in Solr 8.3, it does not prompt for a password before 
letting you into a crippled version of the GUI (as depicted in the attachment)

{
  "authentication":{
"class":"solr.BasicAuthPlugin",
"credentials":{
  "solradmin":"[redacted]",
  "pysolrmon":"[redacted]",
  "solrtrg":"[redacted]"},
"":{"v":2}},
  "authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"user-role":{
  "solradmin":[
"admin",
"allgen",
"trgadmin",
"genadmin"],
  "solrtrg":[
"trgadmin",
"allgen"],
  "pysolrmon":["clustatus_role"]},
"permissions":[
  {
"name":"gen_admin",
"collection":"NULL",
"path":"/admin/cores",
"params":{"action":[
"REGEX:(?i)CREATE",
"REGEX:(?i)RENAME",
"REGEX:(?i)SWAP",
"REGEX:(?i)UNLOAD",
"REGEX:(?i)SPLIT"]},
"role":"genadmin"},
  {
"name":"col_admin",
"collection":null,
"path":"/admin/collections",
"params":{"action":[
"REGEX:(?i)CREATE",
"REGEX:(?i)MODIFYCOLLECTION",
"REGEX:(?i)SPLITSHARD",
"REGEX:(?i)CREATESHARD",
"REGEX:(?i)DELETESHARD",
"REGEX:(?i)CREATEALIAS",
"REGEX:(?i)DELETEALIAS",
"REGEX:(?i)DELETE",
"REGEX:(?i)DELETEREPLICA",
"REGEX:(?i)ADDREPLICA",
"REGEX:(?i)CLUSTERPROP",
"REGEX:(?i)MIGRATE",
"REGEX:(?i)ADDROLE",
"REGEX:(?i)REMOVEROLE",
"REGEX:(?i)ADDREPLICAPROP",
"REGEX:(?i)DELETEREPLICAPROP",
"REGEX:(?i)BALANCESHARDUNIQUE",
"REGEX:(?i)REBALANCELEADERS",
"REGEX:(?i)FORCELEADER",
"REGEX:(?i)MIGRATESTATEFORMAT"]},
"role":"genadmin"},
  {
"name":"security-edit",
"role":"admin"},
  {
"name":"clustatus",
"path":"/admin/collections",
"params":{"action":["REGEX:(?i)CLUSTERSTATUS"]},
"role":[
  "clustatus_role",
  "allgen"],
"collection":null},
  {
"name":"corestatus",
"path":"/admin/cores",
"params":{"action":["REGEX:(?i)STATUS"]},
"role":[
  "allgen",
  "clustatus_role"],
"collection":null},
  {
"name":"trgadmin",
"collection":"trg_col",
"path":"/admin/*",
"role":"trgadmin"},
  {
"name":"open_select",
"path":"/select/*",
"role":null},
  {
"name":"open_search",
"path":"/search/*",
"role":null},
  {
"name":"catch-all-nocollection",
"collection":null,
"path":"/*",
"role":"allgen"},
  {
"name":"catch-all-collection",
"path":"/*",
"role":"allgen"},
  {
"name":"all-admincol",
"collection":null,
"path":"/admin/collections",
"role":"allgen"},
  {
"name":"all-admincores",
"collection":null,
"path":"/admin/cores",
"role":"allgen"}],
"":{"v":5}}}

-Original Message-
From: Jan Høydahl  
Sent: Wednesday, December 11, 2019 7:35 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr8 changes how security.json restricts access to GUI

Please show your complete Security.json so we know how auth is configured. 
Which 8.x version are you trying? There should be a login screen shown in admin 
UI now.

Jan Høydahl

> 11. des. 2019 kl. 22:40 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
> :
> 
> In Solr 7, we had clauses in our security.json saying
> 
>  {
>"name":"all-admin",
>"collection":null,
>"path":"/*",
>"role":"allgen",
>"index":15},
>  {
>"name":"all-core-handlers",
>"path":"/*",
>"role":"allgen",
>"index":16},
> 
> We granted the role allgen to all users; but this kept our security folk 
> happy in that no one could even get to the top level of the Solr GUI without 
> a password.
> 
> Now under Solr 8, the GUI does not prompt for a password. It just brings you 
> into the GUI (albeit a stripped down version, saying such things as "No cores 
> available"). By what means can we require a password to get this far? And by 
> what means can we prompt for a password in order to get further?

Re: backing up and restoring

2019-12-12 Thread rhys J 
I was able to successfully restore a backup by specifying name and location
in the restore command.

But now when i try to run:

sudo -u solr curl http://localhost:8983/solr/debt/update -H "Content-type:
text/xml" --data-binary '*:*'

I get the following error:

 no segments* file found in
LockValidatingDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/var/solr/data/debt/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@4746f577;
maxCacheMB=48.0 maxMergeSizeMB=4.0)): files: [write.lock]
  org.apache.lucene.index.IndexNotFoundException: no
segments* file found in
LockValidatingDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/var/solr/data/debt/data/index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@4746f577;
maxCacheMB=48.0 maxMergeSizeMB=4.0)): files: [write.lock]

I am just copying the top portion of the error, as it is very long.

What did I do wrong?

Thanks,

Rhys

Re: A question of solr recovery

2019-12-12 Thread Shawn Heisey 

On 12/12/2019 8:53 AM, Shawn Heisey wrote:
I do not think the replication handler deals with tlog files at all. The 
transaction log capability did not exist when the replication handler 
was built.


I may have mixed up your message with a different one.  Looking back 
over this, I don't see any mention of the replication handler ... 
that's in another thread about making backups.  Apologies.


Thanks,
Shawn

Re: A question of solr recovery

2019-12-12 Thread Shawn Heisey 

On 12/12/2019 3:37 AM, Hongxu Ma wrote:

And I found my "full sync" log:
"IndexFetcher Total time taken for download 
(fullCopy=true,bytesDownloaded=178161685180) : 4377 secs (40704063 bytes/sec) to 
NIOFSDirectory@..."

A more question:
Form the log, looks it downloaded all segment files (178GB), it's very big and 
took a long time.
Is it possible only download the segment file which contains the missing part? 
No need all files, maybe it can save time?


I'm finding the code in the replication handler to be difficult to 
follow ... but I do not think there is any way to have it do a partial 
or catchup backup copy.  I believe it must copy the entire index every time.


To achieve what you're after, using an alternate backup strategy would 
be advisable.  There are two good ways:


1) Assuming you're on an OS that supports them, make a hardlink copy of 
the data directory to an alternate location on the same filesystem. 
This should happen almost instantly.  Then at your leisure, copy from 
the hardlink copy to your final destination.  When finished, delete the 
hardlink copy.


2) Use rsync (probably with "-avH --delete" options) to copy from the 
current index directory to another location.  This will skip any 
segments that already exist in the destination.


You could potentially even combine those two options in your final solution.

I do not think the replication handler deals with tlog files at all. 
The transaction log capability did not exist when the replication 
handler was built.


Thanks,
Shawn

Re: backing up and restoring

2019-12-12 Thread rhys J 
This page seems to indicate that I should copy the files from the backup
directory back into the index?

Is this accurate?

https://codice.atlassian.net/wiki/spaces/DDF22/pages/2785407/Solr+Standalone+Server+Backup

Thanks,

Rhys

Re: backing up and restoring

2019-12-12 Thread rhys J 
On Thu, Dec 12, 2019 at 3:49 AM sudhir kumar 
wrote:


> once you backup index with some location, you have to specify the same
> location to restore.
>
> ie in your case /tmp/solr is the location indexed was backed up , use same
> location for restore.
>
> you did not provide name so latest snapshot will be restored.
>
> curl '
>
> http://localhost:8983/solr/debt/replication?command=backup=/tmp/solr
> '
>
> snapshot is created at /tmp/solr/snapshot.2019xx
>
> execute below command, latest snapshot will be restored
> curl '
>
> http://localhost:8983/solr/debt/replication?command=restore=/tmp/solr
> <
> http://localhost:8983/solr/debt/replication?command=backup=/tmp/solr
> >
> '
>
>
I figured this out, but even when I specify location, and even name, I get
an OK from the status, but the index remains empty?

Commands used:

sudo -u solr curl '
http://localhost:8983/solr/debt/replication?command=backup

=/tmp/solrbackups/debt/'

sudo -u solr curl
'http://localhost:8983/solr/debt/replication?command=restore=/tmp/solrbackups/debt/'



Thanks,

Rhys

Re: JSON Facet doesn't allow date range facets

2019-12-12 Thread Joel Bernstein 
Searching on a single point in time would be extremely limiting because it
has millisecond precision. So range queries would be the only real way to
search the DatePointField.

I've used this construct many times on the DatePointField:

[2000-05-01T00:00:01Z TO 2019-06-02T00:00:01Z]


Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Dec 12, 2019 at 7:19 AM Mel Mason 
wrote:

> I need to store date ranges in the index. While DatePointField can be
> queried using date ranges, the actual value can only be a single date -
> it can't represent a range of dates as far as I know.
>
> On 12/12/2019 12:11, Joel Bernstein wrote:
> > So something like this should work:
> >
> > [2000-05-01T00:00:01Z TO 2019-06-02T00:00:01Z]
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Thu, Dec 12, 2019 at 7:08 AM Joel Bernstein 
> wrote:
> >
> >> With the DatePointField you can still do the range query, but I believe
> >> you'll need to specify the full ISO date string: 2000-05-01T01:01:01Z
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
> >> On Thu, Dec 12, 2019 at 6:46 AM Mel Mason 
> >> wrote:
> >>
> >>> Unfortunately I need a date range field, e.g [2000-05-01 TO
> 2019-06-02].
> >>> DatePointFields can't represent that as far as I know.
> >>>
> >>> On 12/12/2019 11:40, Joel Bernstein wrote:
>  There is a field type in the schema called pdate:
> 
>  
> 
>  This should work for you.
> 
>  The timeseries Streaming Expression uses the JSON facet API for range
>  faceting and works really well.
> 
> 
> 
> 
>  Joel Bernstein
>  http://joelsolr.blogspot.com/
> 
> 
>  On Thu, Dec 12, 2019 at 6:28 AM Mel Mason <
> mel.ma...@bodleian.ox.ac.uk>
>  wrote:
> 
> > Hi,
> >
> > I'm trying to have a range facet on a field of type
> >>> solr.DateRangeField.
> > As far as I can tell, this isn't possible with JSONFacet, only with
> the
> > old facet system - a quick google turns up several other people with
> >>> the
> > same problem. When using JSONFacet I get problems with this line of
> > code:
> >
> >
> >>>
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/facet/FacetRange.java#L255
> > It looks like during the change to JSONFacet the range facets have
> been
> > restricted to only allow Trie or PointField fields. Is this likely to
> >>> be
> > fixed in future updates, or were there problems with using
> > DateRangeFields? I could use the old parameter facet system, but
> there
> > are features in JSONFacet I need.
> >
> > For completeness, the error I get:
> >
> > org.apache.solr.common.SolrException: Unable to range facet on
> > field:date_dtr
> >   at
> >
> >>>
> org.apache.solr.search.facet.FacetRangeProcessor.getCalcForField(FacetRange.java:238)
> >   at
> >
> >>>
> org.apache.solr.search.facet.FacetRangeProcessor.(FacetRange.java:122)
> >   at
> >
> >>>
> org.apache.solr.search.facet.FacetRange.createFacetProcessor(FacetRange.java:65)
> >   at
> >
> >>>
> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:397)
> >   at
> >
> >>>
> org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)
> >   at
> >
> >>>
> org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:432)
> >   at
> >
> >>>
> org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:64)
> >   at
> >
> >>>
> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:401)
> >   at
> >
> org.apache.solr.search.facet.FacetModule.process(FacetModule.java:146)
> >   at
> >
> >>>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
> >   at
> >
> >>>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> >   at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2566)
> >   at
> > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:756)
> >   at
> >>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:542)
> >   at
> >
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
> >   at
> >
> >>>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
> >   at
> >
> >>>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> >   at
> >
> >>>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> >   at
> >
> >>>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> >   at
> >
> >>>
>

Re: JSON Facet doesn't allow date range facets

2019-12-12 Thread Mel Mason 
I need to store date ranges in the index. While DatePointField can be 
queried using date ranges, the actual value can only be a single date - 
it can't represent a range of dates as far as I know.


On 12/12/2019 12:11, Joel Bernstein wrote:

So something like this should work:

[2000-05-01T00:00:01Z TO 2019-06-02T00:00:01Z]

Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Dec 12, 2019 at 7:08 AM Joel Bernstein  wrote:


With the DatePointField you can still do the range query, but I believe
you'll need to specify the full ISO date string: 2000-05-01T01:01:01Z

Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Dec 12, 2019 at 6:46 AM Mel Mason 
wrote:


Unfortunately I need a date range field, e.g [2000-05-01 TO 2019-06-02].
DatePointFields can't represent that as far as I know.

On 12/12/2019 11:40, Joel Bernstein wrote:

There is a field type in the schema called pdate:



This should work for you.

The timeseries Streaming Expression uses the JSON facet API for range
faceting and works really well.




Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Dec 12, 2019 at 6:28 AM Mel Mason 
wrote:


Hi,

I'm trying to have a range facet on a field of type

solr.DateRangeField.

As far as I can tell, this isn't possible with JSONFacet, only with the
old facet system - a quick google turns up several other people with

the

same problem. When using JSONFacet I get problems with this line of
code:



https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/facet/FacetRange.java#L255

It looks like during the change to JSONFacet the range facets have been
restricted to only allow Trie or PointField fields. Is this likely to

be

fixed in future updates, or were there problems with using
DateRangeFields? I could use the old parameter facet system, but there
are features in JSONFacet I need.

For completeness, the error I get:

org.apache.solr.common.SolrException: Unable to range facet on
field:date_dtr
  at


org.apache.solr.search.facet.FacetRangeProcessor.getCalcForField(FacetRange.java:238)

  at


org.apache.solr.search.facet.FacetRangeProcessor.(FacetRange.java:122)

  at


org.apache.solr.search.facet.FacetRange.createFacetProcessor(FacetRange.java:65)

  at


org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:397)

  at


org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)

  at


org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:432)

  at


org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:64)

  at


org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:401)

  at
org.apache.solr.search.facet.FacetModule.process(FacetModule.java:146)
  at


org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)

  at


org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)

  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2566)
  at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:756)
  at

org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:542)

  at


org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)

  at


org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)

  at


org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)

  at


org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)

  at


org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)

  at


org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

  at


org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)

  at


org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)

  at


org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)

  at


org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)

  at


org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)

  at


org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)

  at


org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)

  at


org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)

  at


org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)

  at


org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)

  at


org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)

  at

Re: JSON Facet doesn't allow date range facets

2019-12-12 Thread Joel Bernstein 
So something like this should work:

[2000-05-01T00:00:01Z TO 2019-06-02T00:00:01Z]

Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Dec 12, 2019 at 7:08 AM Joel Bernstein  wrote:

> With the DatePointField you can still do the range query, but I believe
> you'll need to specify the full ISO date string: 2000-05-01T01:01:01Z
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Thu, Dec 12, 2019 at 6:46 AM Mel Mason 
> wrote:
>
>> Unfortunately I need a date range field, e.g [2000-05-01 TO 2019-06-02].
>> DatePointFields can't represent that as far as I know.
>>
>> On 12/12/2019 11:40, Joel Bernstein wrote:
>> > There is a field type in the schema called pdate:
>> >
>> > 
>> >
>> > This should work for you.
>> >
>> > The timeseries Streaming Expression uses the JSON facet API for range
>> > faceting and works really well.
>> >
>> >
>> >
>> >
>> > Joel Bernstein
>> > http://joelsolr.blogspot.com/
>> >
>> >
>> > On Thu, Dec 12, 2019 at 6:28 AM Mel Mason 
>> > wrote:
>> >
>> >> Hi,
>> >>
>> >> I'm trying to have a range facet on a field of type
>> solr.DateRangeField.
>> >> As far as I can tell, this isn't possible with JSONFacet, only with the
>> >> old facet system - a quick google turns up several other people with
>> the
>> >> same problem. When using JSONFacet I get problems with this line of
>> >> code:
>> >>
>> >>
>> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/facet/FacetRange.java#L255
>> >>
>> >> It looks like during the change to JSONFacet the range facets have been
>> >> restricted to only allow Trie or PointField fields. Is this likely to
>> be
>> >> fixed in future updates, or were there problems with using
>> >> DateRangeFields? I could use the old parameter facet system, but there
>> >> are features in JSONFacet I need.
>> >>
>> >> For completeness, the error I get:
>> >>
>> >> org.apache.solr.common.SolrException: Unable to range facet on
>> >> field:date_dtr
>> >>  at
>> >>
>> org.apache.solr.search.facet.FacetRangeProcessor.getCalcForField(FacetRange.java:238)
>> >>  at
>> >>
>> org.apache.solr.search.facet.FacetRangeProcessor.(FacetRange.java:122)
>> >>  at
>> >>
>> org.apache.solr.search.facet.FacetRange.createFacetProcessor(FacetRange.java:65)
>> >>  at
>> >>
>> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:397)
>> >>  at
>> >>
>> org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)
>> >>  at
>> >>
>> org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:432)
>> >>  at
>> >>
>> org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:64)
>> >>  at
>> >>
>> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:401)
>> >>  at
>> >> org.apache.solr.search.facet.FacetModule.process(FacetModule.java:146)
>> >>  at
>> >>
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
>> >>  at
>> >>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>> >>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2566)
>> >>  at
>> >> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:756)
>> >>  at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:542)
>> >>  at
>> >>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
>> >>  at
>> >>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
>> >>  at
>> >>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
>> >>  at
>> >>
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
>> >>  at
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
>> >>  at
>> >>
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>> >>  at
>> >>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>> >>  at
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
>> >>  at
>> >>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
>> >>  at
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
>> >>  at
>> >>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
>> >>  at
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
>> >>  at
>> >>
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
>> >>  at
>> >>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
>> >>  at
>> >>
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
>>

Re: JSON Facet doesn't allow date range facets

2019-12-12 Thread Joel Bernstein 
With the DatePointField you can still do the range query, but I believe
you'll need to specify the full ISO date string: 2000-05-01T01:01:01Z

Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Dec 12, 2019 at 6:46 AM Mel Mason 
wrote:

> Unfortunately I need a date range field, e.g [2000-05-01 TO 2019-06-02].
> DatePointFields can't represent that as far as I know.
>
> On 12/12/2019 11:40, Joel Bernstein wrote:
> > There is a field type in the schema called pdate:
> >
> > 
> >
> > This should work for you.
> >
> > The timeseries Streaming Expression uses the JSON facet API for range
> > faceting and works really well.
> >
> >
> >
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Thu, Dec 12, 2019 at 6:28 AM Mel Mason 
> > wrote:
> >
> >> Hi,
> >>
> >> I'm trying to have a range facet on a field of type solr.DateRangeField.
> >> As far as I can tell, this isn't possible with JSONFacet, only with the
> >> old facet system - a quick google turns up several other people with the
> >> same problem. When using JSONFacet I get problems with this line of
> >> code:
> >>
> >>
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/facet/FacetRange.java#L255
> >>
> >> It looks like during the change to JSONFacet the range facets have been
> >> restricted to only allow Trie or PointField fields. Is this likely to be
> >> fixed in future updates, or were there problems with using
> >> DateRangeFields? I could use the old parameter facet system, but there
> >> are features in JSONFacet I need.
> >>
> >> For completeness, the error I get:
> >>
> >> org.apache.solr.common.SolrException: Unable to range facet on
> >> field:date_dtr
> >>  at
> >>
> org.apache.solr.search.facet.FacetRangeProcessor.getCalcForField(FacetRange.java:238)
> >>  at
> >>
> org.apache.solr.search.facet.FacetRangeProcessor.(FacetRange.java:122)
> >>  at
> >>
> org.apache.solr.search.facet.FacetRange.createFacetProcessor(FacetRange.java:65)
> >>  at
> >> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:397)
> >>  at
> >>
> org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)
> >>  at
> >>
> org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:432)
> >>  at
> >>
> org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:64)
> >>  at
> >> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:401)
> >>  at
> >> org.apache.solr.search.facet.FacetModule.process(FacetModule.java:146)
> >>  at
> >>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
> >>  at
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> >>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:2566)
> >>  at
> >> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:756)
> >>  at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:542)
> >>  at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
> >>  at
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
> >>  at
> >>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> >>  at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> >>  at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> >>  at
> >>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> >>  at
> >>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> >>  at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> >>  at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
> >>  at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> >>  at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> >>  at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> >>  at
> >>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> >>  at
> >>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
> >>  at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> >>  at
> >>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> >>  at
> >>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> >>  at
> >>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> >>

Re: JSON Facet doesn't allow date range facets

2019-12-12 Thread Mel Mason 
Unfortunately I need a date range field, e.g [2000-05-01 TO 2019-06-02]. 
DatePointFields can't represent that as far as I know.


On 12/12/2019 11:40, Joel Bernstein wrote:

There is a field type in the schema called pdate:



This should work for you.

The timeseries Streaming Expression uses the JSON facet API for range
faceting and works really well.




Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Dec 12, 2019 at 6:28 AM Mel Mason 
wrote:


Hi,

I'm trying to have a range facet on a field of type solr.DateRangeField.
As far as I can tell, this isn't possible with JSONFacet, only with the
old facet system - a quick google turns up several other people with the
same problem. When using JSONFacet I get problems with this line of
code:

https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/facet/FacetRange.java#L255

It looks like during the change to JSONFacet the range facets have been
restricted to only allow Trie or PointField fields. Is this likely to be
fixed in future updates, or were there problems with using
DateRangeFields? I could use the old parameter facet system, but there
are features in JSONFacet I need.

For completeness, the error I get:

org.apache.solr.common.SolrException: Unable to range facet on
field:date_dtr
 at
org.apache.solr.search.facet.FacetRangeProcessor.getCalcForField(FacetRange.java:238)
 at
org.apache.solr.search.facet.FacetRangeProcessor.(FacetRange.java:122)
 at
org.apache.solr.search.facet.FacetRange.createFacetProcessor(FacetRange.java:65)
 at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:397)
 at
org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)
 at
org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:432)
 at
org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:64)
 at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:401)
 at
org.apache.solr.search.facet.FacetModule.process(FacetModule.java:146)
 at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
 at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:2566)
 at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:756)
 at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:542)
 at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
 at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
 at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
 at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
 at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
 at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
 at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
 at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
 at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
 at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
 at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
 at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
 at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
 at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
 at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
 at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
 at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
 at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
 at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
 at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
 at org.eclipse.jetty.server.Server.handle(Server.java:502)
 at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
 at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
 at org.eclipse.jetty.io
.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
 at org.eclipse.jetty.io

Re: JSON Facet doesn't allow date range facets

2019-12-12 Thread Joel Bernstein 
There is a field type in the schema called pdate:



This should work for you.

The timeseries Streaming Expression uses the JSON facet API for range
faceting and works really well.




Joel Bernstein
http://joelsolr.blogspot.com/


On Thu, Dec 12, 2019 at 6:28 AM Mel Mason 
wrote:

> Hi,
>
> I'm trying to have a range facet on a field of type solr.DateRangeField.
> As far as I can tell, this isn't possible with JSONFacet, only with the
> old facet system - a quick google turns up several other people with the
> same problem. When using JSONFacet I get problems with this line of
> code:
>
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/facet/FacetRange.java#L255
>
> It looks like during the change to JSONFacet the range facets have been
> restricted to only allow Trie or PointField fields. Is this likely to be
> fixed in future updates, or were there problems with using
> DateRangeFields? I could use the old parameter facet system, but there
> are features in JSONFacet I need.
>
> For completeness, the error I get:
>
> org.apache.solr.common.SolrException: Unable to range facet on
> field:date_dtr
> at
> org.apache.solr.search.facet.FacetRangeProcessor.getCalcForField(FacetRange.java:238)
> at
> org.apache.solr.search.facet.FacetRangeProcessor.(FacetRange.java:122)
> at
> org.apache.solr.search.facet.FacetRange.createFacetProcessor(FacetRange.java:65)
> at
> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:397)
> at
> org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)
> at
> org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:432)
> at
> org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:64)
> at
> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:401)
> at
> org.apache.solr.search.facet.FacetModule.process(FacetModule.java:146)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2566)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:756)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:542)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:502)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
> at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:103)
> at

Re: JSON Facet doesn't allow date range facets

2019-12-12 Thread Munendra S N 
Currently, it is not supported yet. Related JIRA -
https://issues.apache.org/jira/browse/SOLR-10567

Regards,
Munendra S N



On Thu, Dec 12, 2019 at 4:58 PM Mel Mason 
wrote:

> Hi,
>
> I'm trying to have a range facet on a field of type solr.DateRangeField.
> As far as I can tell, this isn't possible with JSONFacet, only with the
> old facet system - a quick google turns up several other people with the
> same problem. When using JSONFacet I get problems with this line of
> code:
>
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/facet/FacetRange.java#L255
>
> It looks like during the change to JSONFacet the range facets have been
> restricted to only allow Trie or PointField fields. Is this likely to be
> fixed in future updates, or were there problems with using
> DateRangeFields? I could use the old parameter facet system, but there
> are features in JSONFacet I need.
>
> For completeness, the error I get:
>
> org.apache.solr.common.SolrException: Unable to range facet on
> field:date_dtr
> at
> org.apache.solr.search.facet.FacetRangeProcessor.getCalcForField(FacetRange.java:238)
> at
> org.apache.solr.search.facet.FacetRangeProcessor.(FacetRange.java:122)
> at
> org.apache.solr.search.facet.FacetRange.createFacetProcessor(FacetRange.java:65)
> at
> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:397)
> at
> org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)
> at
> org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:432)
> at
> org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:64)
> at
> org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:401)
> at
> org.apache.solr.search.facet.FacetModule.process(FacetModule.java:146)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2566)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:756)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:542)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> at org.eclipse.jetty.server.Server.handle(Server.java:502)
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
> at org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
> at org.eclipse.jetty.io
> .FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io
> .ChannelEndPoint$2.run(ChannelEndPoint.java:118)
> at
>

JSON Facet doesn't allow date range facets

2019-12-12 Thread Mel Mason 

Hi,

I'm trying to have a range facet on a field of type solr.DateRangeField. 
As far as I can tell, this isn't possible with JSONFacet, only with the 
old facet system - a quick google turns up several other people with the 
same problem. When using JSONFacet I get problems with this line of 
code: 
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/search/facet/FacetRange.java#L255


It looks like during the change to JSONFacet the range facets have been 
restricted to only allow Trie or PointField fields. Is this likely to be 
fixed in future updates, or were there problems with using 
DateRangeFields? I could use the old parameter facet system, but there 
are features in JSONFacet I need.


For completeness, the error I get:

org.apache.solr.common.SolrException: Unable to range facet on field:date_dtr
at 
org.apache.solr.search.facet.FacetRangeProcessor.getCalcForField(FacetRange.java:238)
at 
org.apache.solr.search.facet.FacetRangeProcessor.(FacetRange.java:122)
at 
org.apache.solr.search.facet.FacetRange.createFacetProcessor(FacetRange.java:65)
at 
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:397)
at 
org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:475)
at 
org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:432)
at 
org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:64)
at 
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:401)
at 
org.apache.solr.search.facet.FacetModule.process(FacetModule.java:146)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2566)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:756)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:542)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:343)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at

Re: A question of solr recovery

2019-12-12 Thread Hongxu Ma 
Thank you very much @Erick Erickson
It's very clear.

And I found my "full sync" log:
"IndexFetcher Total time taken for download 
(fullCopy=true,bytesDownloaded=178161685180) : 4377 secs (40704063 bytes/sec) 
to NIOFSDirectory@..."

A more question:
Form the log, looks it downloaded all segment files (178GB), it's very big and 
took a long time.
Is it possible only download the segment file which contains the missing part? 
No need all files, maybe it can save time?

For example, there is my fabricated algorithm (like database does):

  *   recovery form local tlog as much as possible
  *   calculate the latest version
  *   only download the segment file which contains data > this version

Thanks.


From: Erick Erickson 
Sent: Wednesday, December 11, 2019 20:56
To: solr-user@lucene.apache.org 
Subject: Re: A question of solr recovery

Updates in this context are individual documents, either new ones
or a new version of an existing document. Long recoveries are
quite unlikely to be replaying a few documents from the tlog.

My bet is that you had to do a “full sync” (there should be messages
to that effect in the Solr log). This means that the replica had to
copy the entire index from the leader, and that varies with the size
of the index, network speed and contention, etc.

And to make it more complicated, and despite the comment about 100
docs and the tlog…. while that copy is going on, _new_ updates are
written to the tlog of the recovering replica and after the index
has been copied, those new updates are replayed locally. The 100
doc limit does _not_ apply in this case. So say the recovery starts
at time T and lasts for 60 seconds. All updates sent to the shard
leader over that 60 seconds are put in the local tlog and after the
copy is done, they’re replayed. And then, you guessed it, any
updates received by the leader over that 60 second period are written
to the recovering replica’s tlog and replayed… Under heavy
indexing loads, this can go no for quite a long time. Not certain
that’s what’s happening, but something to be aware of.

Best,
Erick

> On Dec 10, 2019, at 10:39 PM, Hongxu Ma  wrote:
>
> Hi all
> In my cluster, Solr node turned into long time recovery sometimes.
> So I want to know more about recovery and have read a good blog:
> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> It mentioned in the recovery section:
> "Replays the documents from its own tlog if < 100 new updates have been 
> received by the leader. "
>
> My question: what's the meaning of "updates"? commits? or documents?
> I refered solr code but still not sure about it.
>
> Hope you can help, thanks.
>

Re: backing up and restoring

2019-12-12 Thread sudhir kumar 
Hi Rhyhs,

once you backup index with some location, you have to specify the same
location to restore.

ie in your case /tmp/solr is the location indexed was backed up , use same
location for restore.

you did not provide name so latest snapshot will be restored.

curl '
http://localhost:8983/solr/debt/replication?command=backup=/tmp/solr
'

snapshot is created at /tmp/solr/snapshot.2019xx

execute below command, latest snapshot will be restored
curl '
http://localhost:8983/solr/debt/replication?command=restore=/tmp/solr

'

below command can be used for details:
curl 'http://localhost:8983/solr/debt/replication?command=details

'

for details please refer:
https://lucene.apache.org/solr/guide/6_6/making-and-restoring-backups.html


Thanks,
Sudhir .



On Wed, Dec 11, 2019 at 11:36 PM rhys J  wrote:

> I made backups with the following command:
>
> sudo -u solr curl '
>
> http://localhost:8983/solr/debt/replication?command=backup=/tmp/solr
> <
> http://localhost:8983/solr/gettingstarted/replication?command=backup=/tmp/solr
> >
> backups/debt/'
>
> I double checked that I had made the backup, and I had a backup.
>
> To test the restore function, I deleted documents with the following
> command:
>
> sudo -u solr curl http://localhost:8983/solr/debt/update -H "Content-type:
> text/xml" --data-binary '*:*
>
> I stopped and started the service to verify that there are 0 documents in
> the debt core.
>
> Then I ran the following command:
>
> sudo -u solr curl
> '
> http://localhost:8983/solr/debt/replication?command=restore=/tmp/solrbackups/debt/snapshot.20191211175715254
> '
>
> response:
>
>  "responseHeader":{
> "status":0,
> "QTime":4},
>   "status":"OK"}
>
> Then I went to the web API to check the amount of documents I had on the
> core. It still says 0.
>
> I stopped and started the service to just be sure that that wasn't the
> problem, but it still says there are 0 documents.
>
> Am I missing a step in how i restore a backup?
>
> Thanks,
>
> Rhys
>

Re: Indexing with customized parameters

2019-12-12 Thread Emir Arnautović 
Hi Anuj,
Maybe I am missing something but this is more question for some SQL group than 
for Solr group. I am surprised that you get any records. You can consult your 
DB documentation for some more elegant solution, but a brute-force solution, if 
your column is string, could be:
WHERE sector = 27 OR sector LIKE ’27,%’ OR sector LIKE ‘%,27,%’ OR sector LIKE 
‘%,27’ OR sector = 2701…

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 12 Dec 2019, at 08:38, Anuj Bhargava  wrote:
> 
> Any suggestions?
> 
> Regards,
> 
> Anuj
> 
> On Tue, 10 Dec 2019 at 20:52, Anuj Bhargava  wrote:
> 
>> I am trying to index where the *sector field* has the values 27 and/or
>> 2701 and/or 2702 using the following -
>> 
>> >  query="SELECT * FROM country WHERE sector = 27 OR sector = 2701 OR
>> sector = 2702"
>>  deltaImportQuery="SELECT * FROM country
>>WHERE posting_id = '${dataimporter.delta.posting_id}' AND sector = 27
>> OR sector = 2701 OR sector = 2702"
>>  deltaQuery="SELECT posting_id FROM country
>>WHERE last_modified > '${dataimporter.last_index_time}' AND sector =
>> 27 OR sector = 2701 OR sector = 2702">
>> 
>> 
>> The sector field has comma separated multiple values like -
>> 27,19,527
>> 38,27,62701
>> 2701,49
>> 55,2702,327
>> 
>> The issue is when I run the above, it indexes the fields containing data
>> 27,19,527 and 2701,49 and ignores the other data. It indexes if the data in
>> the sector fields starts with either 27 or 2701 or 2702. It doesn't index
>> if the values 27 or 2701 or 2702 are placed 2nd or 3rd in the sector data
>> field
>>

Facing jwt authentication problem using solr 8.1.1

2019-12-12 Thread Lakhan Gupta 
Hi,

Using solr 8.1.1 version and facing problem while enabling jwt authentication 
in solr. Jwt authentication is working fine after configuring security.json 
file. Below is the configuration I am using for enabling jwt authentication.

Security.json

{
  "authentication":{
   "blockUnknown": false,
"class":"solr.JWTAuthPlugin",
   "jwk":{
  "kty":"oct",
  "use":"sig",
  "kid":"k1",
  
"k":"7A02618BE6943C22FD81CAB9F6FCF063B6E1732C3614BC3ACA6032B6B3215CAF0D28A34FD423423CA3AC34BEA27D3F79",
  "alg":"HS256"},
"aud":"solr"},
   "authorization":{
  "class":"solr.RuleBasedAuthorizationPlugin",
  "permissions":[
  {
"name":"all",
 "path":"/*",
"role":"admin"
 }
  ],
  "user-role":{
 "solr":"admin"
  }
   }
}

Using secret key
7A02618BE6943C22FD81CAB9F6FCF063B6E1732C3614BC3ACA6032B6B3215CAF0D28A34FD423423CA3AC34BEA27D3F79

Jwt token is generated:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJhZCIsImF1ZCI6InNvbHIiLCJleHAiOjk5MTYyMzkwMjJ9.M4PksJTJ9gFjOlvvFmG1eDSyXDtKIRSGIYicIW9hwT4

Below header and payload I'm using for generate jwt token :

The header is
{
  "alg": "HS256",
  "typ": "JWT"
}

and the payload is

{
  "sub": "admin",
  "aud": "Solr",
  "exp": 9916239022
}

With above configuration my jwt authentication is working fine. But there is a 
problem when request is sent without authentication in header the api still 
retrieving data. I want to prevent it when request come without authentication 
header.

For that, I've enabled blockUnknown parameter in security.json file. That works 
fine and authentication request is required. But, after enabling blockunknown  
parameter I am facing below exception while starting solr using solr start 
command.


ERROR: Solr requires authentication for 
http://localhost:8983/solr/admin/info/system. Please supply valid credentials. 
HTTP code=401

I've googled a lot and find out

solr/admin/info/system endpoint required authentication.

How to authenticate solr/admin/info/system endpoint while startup solr?

Need urgent help. I'd appreciate if someone can help me.

Thanks
Lakhan Gupta



The information in this email is confidential and may be legally privileged. It 
is intended solely for the addressee and access to it by anyone else is 
unauthorized. If you are not the intended recipient, any disclosure, copying, 
distribution or any action taken or omitted to be taken based on it, is 
strictly prohibited and may be unlawful.