RE: adding documents to a secured solr server.
Yes, that worked. -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Thursday, 2 November 2017 6:14 p.m. To: solr-user@lucene.apache.org Subject: Re: adding documents to a secured solr server. On 11/1/2017 10:04 PM, Phil Scadden wrote: > For testing, I changed to HttpSolrClient and specifying the core on process > and commit instead of opening it as server/core. This time worked... sort of. > Despite deleting the entire index with deletebyquery and seeing that it was > empty in the coreAdmin, I get : > > possible analysis error: cannot change DocValues type from SORTED_SET to > NUMERIC for field "access" > > I tried deleting the field in the admin interface and then adding it back in > again in that admin interface. But, no. Still comes up with that error. I > know deleting the index files on disk works but I don’t have access to the > server. This is a frustrating problem. Variations of this error happen when settings on a field with docValues="true" are changed, and the index already has documents added with the previous settings. Each Lucene segment stores information about what kind of docValues are present for each field that has docValues, and if you change an aspect of the field (multivalued, field class, etc) and try to add a new document with that different information, Lucene will complain. The reason that deleting all documents didn't work is that when you delete documents, they are only MARKED as deleted, the segments (and deleted docs) remain on the disk. The only SURE way to fix it is to completely delete the index directory (or directories), reload the core/collection (or restart Solr), and reindex from scratch. One thing you *might* be able to do if you don't have access to the server is delete all documents and then optimize the index, which should delete all segments and effectively leave you with a brand new empty index. I'm not 100% sure that this would take care of it, but I *think* it would. Thanks, Shawn Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
Re: adding documents to a secured solr server.
On 11/1/2017 10:04 PM, Phil Scadden wrote: For testing, I changed to HttpSolrClient and specifying the core on process and commit instead of opening it as server/core. This time worked... sort of. Despite deleting the entire index with deletebyquery and seeing that it was empty in the coreAdmin, I get : possible analysis error: cannot change DocValues type from SORTED_SET to NUMERIC for field "access" I tried deleting the field in the admin interface and then adding it back in again in that admin interface. But, no. Still comes up with that error. I know deleting the index files on disk works but I don’t have access to the server. This is a frustrating problem. Variations of this error happen when settings on a field with docValues="true" are changed, and the index already has documents added with the previous settings. Each Lucene segment stores information about what kind of docValues are present for each field that has docValues, and if you change an aspect of the field (multivalued, field class, etc) and try to add a new document with that different information, Lucene will complain. The reason that deleting all documents didn't work is that when you delete documents, they are only MARKED as deleted, the segments (and deleted docs) remain on the disk. The only SURE way to fix it is to completely delete the index directory (or directories), reload the core/collection (or restart Solr), and reindex from scratch. One thing you *might* be able to do if you don't have access to the server is delete all documents and then optimize the index, which should delete all segments and effectively leave you with a brand new empty index. I'm not 100% sure that this would take care of it, but I *think* it would. Thanks, Shawn
RE: adding documents to a secured solr server.
Requested reload and now it indexes with secure server using HttpSolrClietn. Phew. I now look to see if I can optimize and get concurrentupdate server to work. At least I can get the index back now. -Original Message- From: Phil Scadden [mailto:p.scad...@gns.cri.nz] Sent: Thursday, 2 November 2017 5:04 p.m. To: solr-user@lucene.apache.org Subject: RE: adding documents to a secured solr server. For testing, I changed to HttpSolrClient and specifying the core on process and commit instead of opening it as server/core. This time worked... sort of. Despite deleting the entire index with deletebyquery and seeing that it was empty in the coreAdmin, I get : possible analysis error: cannot change DocValues type from SORTED_SET to NUMERIC for field "access" I tried deleting the field in the admin interface and then adding it back in again in that admin interface. But, no. Still comes up with that error. I know deleting the index files on disk works but I don’t have access to the server. This is a frustrating problem. -Original Message- From: Shawn Heisey [mailto:elyog...@elyograg.org] Sent: Thursday, 2 November 2017 3:55 p.m. To: solr-user@lucene.apache.org Subject: Re: adding documents to a secured solr server. On 11/1/2017 8:13 PM, Phil Scadden wrote: > 14:52:45,962 DEBUG ConcurrentUpdateSolrClient:177 - starting runner: > org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner@6e > eba4a > 14:52:46,224 WARN ConcurrentUpdateSolrClient:343 - Failed to parse > error response from http://online-dev.gns.cri.nz:8983/solr/prindex due > to: java.lang.RuntimeException: Invalid version (expected 2, but 60) > or the data in not in 'javabin' format > Even more puzzling. Authentication is set. What is the invalid version bit?? > I think my solrj is 6.4.1; the server is 6.6.2. Do these have to match > exactly?? The only time I would be worried about different SolrJ and Solr versions is when using the CloudSolrClient object. For the other client types, you can usually have a VERY wide version spread without problems. For the cloud object, you *might* have problems with different versions, or it might work fine. If the SolrJ version is higher than the Solr version, the cloud client tends to work. I would always recommend that the client version be the same or higher than the server version... but with non-cloud clients, it won't matter very much. I would not expect problems with the two versions you have, as long as you don't try to use the cloud client. This error is different. It's happening because SolrJ is expecting a Javabin response, but it is getting an HTML error response instead, with the "require authentication" error. This logging message will happen anytime SolrJ gets an error response instead of a "real" response. What this error says is technically correct, but very confusing to novice users. The specific numbers in the message are a result of the first character of the response. With javabin, the first character would be 0x02 to indicate the javabin version of 2, but with HTML, the first character is the opening angle bracket, or character number 60 (0x3C). This is where those two numbers come from. Thanks, Shawn Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents. Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
RE: adding documents to a secured solr server.
For testing, I changed to HttpSolrClient and specifying the core on process and commit instead of opening it as server/core. This time worked... sort of. Despite deleting the entire index with deletebyquery and seeing that it was empty in the coreAdmin, I get : possible analysis error: cannot change DocValues type from SORTED_SET to NUMERIC for field "access" I tried deleting the field in the admin interface and then adding it back in again in that admin interface. But, no. Still comes up with that error. I know deleting the index files on disk works but I don’t have access to the server. This is a frustrating problem. -Original Message- From: Shawn Heisey [mailto:elyog...@elyograg.org] Sent: Thursday, 2 November 2017 3:55 p.m. To: solr-user@lucene.apache.org Subject: Re: adding documents to a secured solr server. On 11/1/2017 8:13 PM, Phil Scadden wrote: > 14:52:45,962 DEBUG ConcurrentUpdateSolrClient:177 - starting runner: > org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner@6e > eba4a > 14:52:46,224 WARN ConcurrentUpdateSolrClient:343 - Failed to parse > error response from http://online-dev.gns.cri.nz:8983/solr/prindex due > to: java.lang.RuntimeException: Invalid version (expected 2, but 60) > or the data in not in 'javabin' format > Even more puzzling. Authentication is set. What is the invalid version bit?? > I think my solrj is 6.4.1; the server is 6.6.2. Do these have to match > exactly?? The only time I would be worried about different SolrJ and Solr versions is when using the CloudSolrClient object. For the other client types, you can usually have a VERY wide version spread without problems. For the cloud object, you *might* have problems with different versions, or it might work fine. If the SolrJ version is higher than the Solr version, the cloud client tends to work. I would always recommend that the client version be the same or higher than the server version... but with non-cloud clients, it won't matter very much. I would not expect problems with the two versions you have, as long as you don't try to use the cloud client. This error is different. It's happening because SolrJ is expecting a Javabin response, but it is getting an HTML error response instead, with the "require authentication" error. This logging message will happen anytime SolrJ gets an error response instead of a "real" response. What this error says is technically correct, but very confusing to novice users. The specific numbers in the message are a result of the first character of the response. With javabin, the first character would be 0x02 to indicate the javabin version of 2, but with HTML, the first character is the opening angle bracket, or character number 60 (0x3C). This is where those two numbers come from. Thanks, Shawn Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
RE: adding documents to a secured solr server.
So the real error is authentication, (the version is spurious) but why that when authentication is being set on the updateRequest? -Original Message- From: Shawn Heisey [mailto:elyog...@elyograg.org] Sent: Thursday, 2 November 2017 3:55 p.m. To: solr-user@lucene.apache.org Subject: Re: adding documents to a secured solr server. On 11/1/2017 8:13 PM, Phil Scadden wrote: > 14:52:45,962 DEBUG ConcurrentUpdateSolrClient:177 - starting runner: > org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner@6e > eba4a > 14:52:46,224 WARN ConcurrentUpdateSolrClient:343 - Failed to parse > error response from http://online-dev.gns.cri.nz:8983/solr/prindex due > to: java.lang.RuntimeException: Invalid version (expected 2, but 60) > or the data in not in 'javabin' format > Even more puzzling. Authentication is set. What is the invalid version bit?? > I think my solrj is 6.4.1; the server is 6.6.2. Do these have to match > exactly?? The only time I would be worried about different SolrJ and Solr versions is when using the CloudSolrClient object. For the other client types, you can usually have a VERY wide version spread without problems. For the cloud object, you *might* have problems with different versions, or it might work fine. If the SolrJ version is higher than the Solr version, the cloud client tends to work. I would always recommend that the client version be the same or higher than the server version... but with non-cloud clients, it won't matter very much. I would not expect problems with the two versions you have, as long as you don't try to use the cloud client. This error is different. It's happening because SolrJ is expecting a Javabin response, but it is getting an HTML error response instead, with the "require authentication" error. This logging message will happen anytime SolrJ gets an error response instead of a "real" response. What this error says is technically correct, but very confusing to novice users. The specific numbers in the message are a result of the first character of the response. With javabin, the first character would be 0x02 to indicate the javabin version of 2, but with HTML, the first character is the opening angle bracket, or character number 60 (0x3C). This is where those two numbers come from. Thanks, Shawn Notice: This email and any attachments are confidential and may not be used, published or redistributed without the prior written consent of the Institute of Geological and Nuclear Sciences Limited (GNS Science). If received in error please destroy and immediately notify GNS Science. Do not copy or disclose the contents.
Re: adding documents to a secured solr server.
On 11/1/2017 8:13 PM, Phil Scadden wrote: 14:52:45,962 DEBUG ConcurrentUpdateSolrClient:177 - starting runner: org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner@6eeba4a 14:52:46,224 WARN ConcurrentUpdateSolrClient:343 - Failed to parse error response from http://online-dev.gns.cri.nz:8983/solr/prindex due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format Even more puzzling. Authentication is set. What is the invalid version bit?? I think my solrj is 6.4.1; the server is 6.6.2. Do these have to match exactly?? The only time I would be worried about different SolrJ and Solr versions is when using the CloudSolrClient object. For the other client types, you can usually have a VERY wide version spread without problems. For the cloud object, you *might* have problems with different versions, or it might work fine. If the SolrJ version is higher than the Solr version, the cloud client tends to work. I would always recommend that the client version be the same or higher than the server version... but with non-cloud clients, it won't matter very much. I would not expect problems with the two versions you have, as long as you don't try to use the cloud client. This error is different. It's happening because SolrJ is expecting a Javabin response, but it is getting an HTML error response instead, with the "require authentication" error. This logging message will happen anytime SolrJ gets an error response instead of a "real" response. What this error says is technically correct, but very confusing to novice users. The specific numbers in the message are a result of the first character of the response. With javabin, the first character would be 0x02 to indicate the javabin version of 2, but with HTML, the first character is the opening angle bracket, or character number 60 (0x3C). This is where those two numbers come from. Thanks, Shawn
Re: adding documents to a secured solr server.
On 11/1/2017 7:59 PM, Phil Scadden wrote:
After some digging, I tried this approach...
solr = new ConcurrentUpdateSolrClient.Builder(solrUrl)
.withQueueSize(20)
.build();
SolrInputDocument up = new SolrInputDocument();
up.addField("id",f.getCanonicalPath());
up.addField("title",title);
up.addField("author",author);
String content = textHandler.toString();
up.addField("_text_",content);
UpdateRequest req = new UpdateRequest();
req.setCommitWithin(1000);
req.add(up);
req.setBasicAuthCredentials("solrAdmin", password);
UpdateResponse ur = req.process(solr);
You need to create an UpdateRequest object and set the auth parameters
on that object, rather than using the sugar methods on the client to
have it add the docs directly.
See this:
https://stackoverflow.com/a/44637540
An alternate idea would be to create a custom HttpClient object (using
their Builder methods) that has the authentication credentials baked
into it, and build the solr client using that object. If you do that,
then you won't need to add authentication to any request object.
Side note about custom HttpClient objects: If you intend to use your
solr client object with multiple threads, you will need to create a
custom HttpClient object anyway. This is because the default thread
limit on the HttpClient that is created in the background is two
threads. This limit is not in Solr code, it's in HttpClient. To allow
more, the HttpClient object must be custom-built. I suspect that the
reason you chose ConcurrentUpdateSolrClient was for automatic handling
of several threads (you set the queue size to 20) ... but with a default
object, that won't be what you actually get. I have filed the following
issue to try and improve the default situation:
https://issues.apache.org/jira/browse/SOLR-11596
Something else to add as a strong caution: ConcurrentUpdateSolrClient
swallows all indexing errors. If the Solr server were completely down,
you would not see any exceptions on "add" calls, even though the
requests all would fail ... the program would only get an error on the
"commit" call, and it is fairly common for developers to leave the
commit out, letting Solr handle all commits. If you want your program
to be aware of all indexing errors, you will need to use HttpSolrClient
or CloudSolrClient and handle multiple threads in your own code.
Thanks,
Shawn
RE: adding documents to a secured solr server.
And my security.json looks like:
{
"authentication":{
"class":"solr.BasicAuthPlugin",
"blockUnknown":true,
"credentials":{
"solrAdmin":" a hash ",
"solrGuest":"another hash"},
"":{"v":0}},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"permissions":[
{
"name":"all",
"role":"admin"},
{
"name":"read",
"role":"guest"}],
"user-role":{"solrAdmin":["admin","guest"],"solrGuest":"guest"}}}
It looks like I should be able to add.
this one worked to delete the entire index:
UpdateRequest up = new UpdateRequest();
up.setBasicAuthCredentials("solrAdmin",password);
up.deleteByQuery("*:*");
up.setCommitWithin(1000);
up.process(solr);
-Original Message-
From: Phil Scadden [mailto:p.scad...@gns.cri.nz]
Sent: Thursday, 2 November 2017 2:59 p.m.
To: solr-user@lucene.apache.org
Subject: RE: adding documents to a secured solr server.
After some digging, I tried this approach...
solr = new ConcurrentUpdateSolrClient.Builder(solrUrl)
.withQueueSize(20)
.build();
SolrInputDocument up = new SolrInputDocument();
up.addField("id",f.getCanonicalPath());
up.addField("title",title);
up.addField("author",author);
String content = textHandler.toString();
up.addField("_text_",content);
UpdateRequest req = new UpdateRequest();
req.setCommitWithin(1000);
req.add(up);
req.setBasicAuthCredentials("solrAdmin", password);
UpdateResponse ur = req.process(solr);
However, I get error back of:
14:52:45,962 DEBUG ConcurrentUpdateSolrClient:177 - starting runner:
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner@6eeba4a
14:52:46,224 WARN ConcurrentUpdateSolrClient:343 - Failed to parse error
response from http://online-dev.gns.cri.nz:8983/solr/prindex due to:
java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in
not in 'javabin' format
14:52:46,224 ERROR ConcurrentUpdateSolrClient:540 - error
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://online-dev.gns.cri.nz:8983/solr/prindex: require
authentication
request:
http://online-dev.gns.cri.nz:8983/solr/prindex/update?wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:345)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
14:52:46,224 DEBUG ConcurrentUpdateSolrClient:210 - finished:
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner@6eeba4a
Even more puzzling. Authentication is set. What is the invalid version bit?? I
think my solrj is 6.4.1; the server is 6.6.2. Do these have to match exactly??
-Original Message-
From: Phil Scadden [mailto:p.scad...@gns.cri.nz]
Sent: Thursday, 2 November 2017 11:28 a.m.
To: solr-user@lucene.apache.org
Subject: adding documents to a secured solr server.
Solrj QueryRequest object has a method to set basic authorization
username/password but what is the equivalent way to pass authorization when you
are adding new documents to an index?
ConcurrentUpdateSolrClient solr = new
ConcurrentUpdateSolrClient(solrProperties.getServer(),10,2);
...
up.addField("id","myid");
up.addField("title",title);
up.addField("author",author);
String content = textHandler.toString();
up.addField("_text_",content);
solr.add(up);
solr.commit();
I cant see where authorization occurs?
Notice: This email and any attachments are confidential and may not be used,
published or redistributed without the prior written consent of the Institute
of Geological and Nuclear Sciences Limited (GNS Science). If received in error
please destroy and immediately notify GNS Science. Do not copy or disclose the
contents.
Notice: This email and any attachments are confidential and may not be used,
published or redistributed without the prior written c
RE: adding documents to a secured solr server.
After some digging, I tried this approach...
solr = new ConcurrentUpdateSolrClient.Builder(solrUrl)
.withQueueSize(20)
.build();
SolrInputDocument up = new SolrInputDocument();
up.addField("id",f.getCanonicalPath());
up.addField("title",title);
up.addField("author",author);
String content = textHandler.toString();
up.addField("_text_",content);
UpdateRequest req = new UpdateRequest();
req.setCommitWithin(1000);
req.add(up);
req.setBasicAuthCredentials("solrAdmin", password);
UpdateResponse ur = req.process(solr);
However, I get error back of:
14:52:45,962 DEBUG ConcurrentUpdateSolrClient:177 - starting runner:
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner@6eeba4a
14:52:46,224 WARN ConcurrentUpdateSolrClient:343 - Failed to parse error
response from http://online-dev.gns.cri.nz:8983/solr/prindex due to:
java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in
not in 'javabin' format
14:52:46,224 ERROR ConcurrentUpdateSolrClient:540 - error
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://online-dev.gns.cri.nz:8983/solr/prindex: require
authentication
request:
http://online-dev.gns.cri.nz:8983/solr/prindex/update?wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:345)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
14:52:46,224 DEBUG ConcurrentUpdateSolrClient:210 - finished:
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner@6eeba4a
Even more puzzling. Authentication is set. What is the invalid version bit?? I
think my solrj is 6.4.1; the server is 6.6.2. Do these have to match exactly??
-Original Message-
From: Phil Scadden [mailto:p.scad...@gns.cri.nz]
Sent: Thursday, 2 November 2017 11:28 a.m.
To: solr-user@lucene.apache.org
Subject: adding documents to a secured solr server.
Solrj QueryRequest object has a method to set basic authorization
username/password but what is the equivalent way to pass authorization when you
are adding new documents to an index?
ConcurrentUpdateSolrClient solr = new
ConcurrentUpdateSolrClient(solrProperties.getServer(),10,2);
...
up.addField("id","myid");
up.addField("title",title);
up.addField("author",author);
String content = textHandler.toString();
up.addField("_text_",content);
solr.add(up);
solr.commit();
I cant see where authorization occurs?
Notice: This email and any attachments are confidential and may not be used,
published or redistributed without the prior written consent of the Institute
of Geological and Nuclear Sciences Limited (GNS Science). If received in error
please destroy and immediately notify GNS Science. Do not copy or disclose the
contents.
Notice: This email and any attachments are confidential and may not be used,
published or redistributed without the prior written consent of the Institute
of Geological and Nuclear Sciences Limited (GNS Science). If received in error
please destroy and immediately notify GNS Science. Do not copy or disclose the
contents.
adding documents to a secured solr server.
Solrj QueryRequest object has a method to set basic authorization
username/password but what is the equivalent way to pass authorization when you
are adding new documents to an index?
ConcurrentUpdateSolrClient solr = new
ConcurrentUpdateSolrClient(solrProperties.getServer(),10,2);
...
up.addField("id","myid");
up.addField("title",title);
up.addField("author",author);
String content = textHandler.toString();
up.addField("_text_",content);
solr.add(up);
solr.commit();
I cant see where authorization occurs?
Notice: This email and any attachments are confidential and may not be used,
published or redistributed without the prior written consent of the Institute
of Geological and Nuclear Sciences Limited (GNS Science). If received in error
please destroy and immediately notify GNS Science. Do not copy or disclose the
contents.
