Re: AJAX access to Solr Server

[Doug Turnbull]

We do this all the time, whitelisting only the readonly search end points
we want to support and disallowing excessively large paging.

Here is a template for an nginx solr proxy. The read me describes more of
our philosophy

https://github.com/o19s/solr_nginx

On Friday, December 25, 2015, Eric Dain  wrote:

> Hi all,
>
> Does allowing javascript direct access to SolrCloud raise security concern?
> should I build a REST service in between?
>
> I need to provide async search capability to web pages. the pages will be
> public with no authentication.
>
> Happy searching,
> Eric
>


-- 
*Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections
, LLC | 240.476.9983
Author: Relevant Search 
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.

AJAX access to Solr Server

[Eric Dain]

Hi all,

Does allowing javascript direct access to SolrCloud raise security concern?
should I build a REST service in between?

I need to provide async search capability to web pages. the pages will be
public with no authentication.

Happy searching,
Eric

( no servers hosting shard ) very strange

[elvis鱼人]

my environment is solrcloud 5.2.0, 3 shards, every shard has 2 replica,one 
leader,one replica


[11-18 06:58:42.515][ERROR]
org.apache.solr.common.SolrException.log(SolrException.java:139) -
org.apache.solr.common.SolrException: no servers hosting shard: 
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:211)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

[11-18 06:58:42.515][INFO]
org.apache.solr.core.SolrCore.execute(SolrCore.java:2069) - [feed]
webapp=/solr path=/select params={q=uid:2647309=javabin=2}
status=503 QTime=4 

query to this node,not to solrcloud



--
View this message in context: 
http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: ( no servers hosting shard ) very strange

[Binoy Dalal]

Hi Elvis,
We're having a similar problem with one of our cloud setups.
I'm currently running Solr 4.10.4 with 2 shards and no replicas. One of our
shards also keeps dropping off with the same error message. This happens
because the shard executes too many full GCs which completely freezes the
machine and causes zookeeper to think that it has dropped off. So I would
suggest that you analyze your gc logs and see if you're facing a similar
problem.
Moreover, I'm currently running the zookeeper that is embedded with solr.
Are you doing the same or do you have an external instance/ensemble setup?
So far, I've got no solutions but I'll update this thread if I find
anything.

On Fri, Dec 25, 2015 at 5:01 PM elvis鱼人  wrote:

> my environment is solrcloud 5.2.0, 3 shards, every shard has 2 replica,one
> leader,one replica
>
>
> [11-18 06:58:42.515][ERROR]
> org.apache.solr.common.SolrException.log(SolrException.java:139) -
> org.apache.solr.common.SolrException: no servers hosting shard:
> at
>
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:211)
> at
>
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
>
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> [11-18 06:58:42.515][INFO]
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2069) - [feed]
> webapp=/solr path=/select params={q=uid:2647309=javabin=2}
> status=503 QTime=4
>
> query to this node,not to solrcloud
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/no-servers-hosting-shard-very-strange-tp4247349.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
-- 
Regards,
Binoy Dalal

Re: Data import issue

[Alexandre Rafalovitch]

Do you have a full stack trace? A bit hard to help without that.
On 24 Dec 2015 2:54 pm, "Midas A"  wrote:

> Hi ,
>
>
> Please provide the steps to resolve the issue.
>
>
> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
> Communications link failure during rollback(). Transaction resolution
> unknown.
>

Re: AJAX access to Solr Server

[Shawn Heisey]

On 12/25/2015 12:17 PM, Eric Dain wrote:
> Does allowing javascript direct access to SolrCloud raise security concern?
> should I build a REST service in between?
> 
> I need to provide async search capability to web pages. the pages will be
> public with no authentication.

End users should never have access to Solr.  Access to Solr from the
end-user machine is required if you want to accept Solr responses directly.

In one of the other replies that you received, Doug has given you an
nginx config for proxying access to Solr -- indirect access.  This can
protect against *changes* to the index, and it has protection against
high start/rows values, but there are many other ways that an attacker
can construct denial of service queries, which this proxy config will
not prevent.

I think that indirect access (through a proxy) should not be allowed
either, unless you can trust all the people that will have access.

If Solr is open to a sufficiently wide audience (especially the
Internet), someone will find a way to abuse the service even with a
proxy, either to cause harm or to learn things they shouldn't know.

The most secure option is to only allow the webservers and trusted
administrators to access Solr.  All end user (Internet) access to Solr
should be handled through a custom web application.  This might be
something that you find and install (such as wordpress, drupal, etc), or
one that you write yourself.

You can still do AJAX while maintaining security.  You'll need to write
something in a server-side web programming language like PHP, Java, etc.
 This code will need to accept the AJAX requests from your client-side
javascript code, validate the request parameters to make sure they're
sane, get a response from Solr, and return relevant data.  If the
parameters don't validate, return an error, and handle that error
appropriately in the javascript code.

Thanks,
Shawn

Re: AJAX access to Solr Server

[GW]

If you are using Linux a simple one liner in IP tables

iptables -I INPUT \! --src www.yourwebserver.com -m tcp -p tcp --dport
8983 -j DROP


If windows, you can do something similar

otherwise it is very easy for anyone to delete all your documents with

http://yoursolrserver.com:8983/solr/your-core/update?stream.body=
*:*=true




On 25 December 2015 at 20:42, Doug Turnbull <
dturnb...@opensourceconnections.com> wrote:

> Hi Shawn
>
> Maybe I should have qualified the parameters of scenarios this make me
> comfortable just proxying Solr directly w/o an API
>
> These situations include:
>
> 1. I've got no qualms about giving the whole world access to every document
> in the index. There's nothing protected about anything.
> 2. The content can be easily rebuilt , it's not very large. (I can easily
> push a button and make a new one)
>
> Sure you can denial of service Solr, and I might lose my search index. But
> you can denial of service anything. This includes just about anything you
> put in front of Solr. Moreover, the added complexity of a
> Drupal/Wordpress/your API might only add to the security problems with
> their own security issues. I'd rather keep it simple and have fewer moving
> parts.
>
> Cases where I would want an API in front of Solr (these are just the
> security ones):
> - I want to protect the content (ie based on some notion of a "user" or
> other permissions)
> - Rebuilding the content would be very hard and time consuming
>
> I would also say to expose Solr directly to everyone you probably should
> know about Solr's bugaboos:
> - the lovely qt parameter and the request dispatcher (the nginx proxy below
> disallows qt)
> - deep paging (prevented by the nginx proxy)
> - how to lock down a request handler fairly robustly, how to use invariants
> - mitigating intentionally malicious queries (such as the lovely "sleep"
> function query).
>
> I'm also curious to hear what the websolr people do, or anyone else that
> hosts Solr for the JavaScript app development crowd.
>
> Cheers
> -Doug
>
>
> On Friday, December 25, 2015, Shawn Heisey  wrote:
>
> > On 12/25/2015 12:17 PM, Eric Dain wrote:
> > > Does allowing javascript direct access to SolrCloud raise security
> > concern?
> > > should I build a REST service in between?
> > >
> > > I need to provide async search capability to web pages. the pages will
> be
> > > public with no authentication.
> >
> > End users should never have access to Solr.  Access to Solr from the
> > end-user machine is required if you want to accept Solr responses
> directly.
> >
> > In one of the other replies that you received, Doug has given you an
> > nginx config for proxying access to Solr -- indirect access.  This can
> > protect against *changes* to the index, and it has protection against
> > high start/rows values, but there are many other ways that an attacker
> > can construct denial of service queries, which this proxy config will
> > not prevent.
> >
> > I think that indirect access (through a proxy) should not be allowed
> > either, unless you can trust all the people that will have access.
> >
> > If Solr is open to a sufficiently wide audience (especially the
> > Internet), someone will find a way to abuse the service even with a
> > proxy, either to cause harm or to learn things they shouldn't know.
> >
> > The most secure option is to only allow the webservers and trusted
> > administrators to access Solr.  All end user (Internet) access to Solr
> > should be handled through a custom web application.  This might be
> > something that you find and install (such as wordpress, drupal, etc), or
> > one that you write yourself.
> >
> > You can still do AJAX while maintaining security.  You'll need to write
> > something in a server-side web programming language like PHP, Java, etc.
> >  This code will need to accept the AJAX requests from your client-side
> > javascript code, validate the request parameters to make sure they're
> > sane, get a response from Solr, and return relevant data.  If the
> > parameters don't validate, return an error, and handle that error
> > appropriately in the javascript code.
> >
> > Thanks,
> > Shawn
> >
> >
>
> --
> *Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections
> , LLC | 240.476.9983
> Author: Relevant Search 
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>

Re: AJAX access to Solr Server

[GW]

I would put in a basic iptables statement to allow only your webserver to
prevent

http://172.16.0.22:8983/solr/products/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E=true

On 25 December 2015 at 14:58, Eric Dain  wrote:

> Thanks, that is very helpful.
>
> Have you tried denying access to some fields in the documents?
>
> On Fri, Dec 25, 2015 at 11:31 AM, Doug Turnbull <
> dturnb...@opensourceconnections.com> wrote:
>
> > We do this all the time, whitelisting only the readonly search end points
> > we want to support and disallowing excessively large paging.
> >
> > Here is a template for an nginx solr proxy. The read me describes more of
> > our philosophy
> >
> > https://github.com/o19s/solr_nginx
> >
> > On Friday, December 25, 2015, Eric Dain  wrote:
> >
> > > Hi all,
> > >
> > > Does allowing javascript direct access to SolrCloud raise security
> > concern?
> > > should I build a REST service in between?
> > >
> > > I need to provide async search capability to web pages. the pages will
> be
> > > public with no authentication.
> > >
> > > Happy searching,
> > > Eric
> > >
> >
> >
> > --
> > *Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections
> > , LLC | 240.476.9983
> > Author: Relevant Search 
> > This e-mail and all contents, including attachments, is considered to be
> > Company Confidential unless explicitly stated otherwise, regardless
> > of whether attachments are marked as such.
> >
>

Re: AJAX access to Solr Server

[Eric Dain]

Thanks, that is very helpful.

Have you tried denying access to some fields in the documents?

On Fri, Dec 25, 2015 at 11:31 AM, Doug Turnbull <
dturnb...@opensourceconnections.com> wrote:

> We do this all the time, whitelisting only the readonly search end points
> we want to support and disallowing excessively large paging.
>
> Here is a template for an nginx solr proxy. The read me describes more of
> our philosophy
>
> https://github.com/o19s/solr_nginx
>
> On Friday, December 25, 2015, Eric Dain  wrote:
>
> > Hi all,
> >
> > Does allowing javascript direct access to SolrCloud raise security
> concern?
> > should I build a REST service in between?
> >
> > I need to provide async search capability to web pages. the pages will be
> > public with no authentication.
> >
> > Happy searching,
> > Eric
> >
>
>
> --
> *Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections
> , LLC | 240.476.9983
> Author: Relevant Search 
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>

Re: AJAX access to Solr Server

[Doug Turnbull]

Hi Shawn

Maybe I should have qualified the parameters of scenarios this make me
comfortable just proxying Solr directly w/o an API

These situations include:

1. I've got no qualms about giving the whole world access to every document
in the index. There's nothing protected about anything.
2. The content can be easily rebuilt , it's not very large. (I can easily
push a button and make a new one)

Sure you can denial of service Solr, and I might lose my search index. But
you can denial of service anything. This includes just about anything you
put in front of Solr. Moreover, the added complexity of a
Drupal/Wordpress/your API might only add to the security problems with
their own security issues. I'd rather keep it simple and have fewer moving
parts.

Cases where I would want an API in front of Solr (these are just the
security ones):
- I want to protect the content (ie based on some notion of a "user" or
other permissions)
- Rebuilding the content would be very hard and time consuming

I would also say to expose Solr directly to everyone you probably should
know about Solr's bugaboos:
- the lovely qt parameter and the request dispatcher (the nginx proxy below
disallows qt)
- deep paging (prevented by the nginx proxy)
- how to lock down a request handler fairly robustly, how to use invariants
- mitigating intentionally malicious queries (such as the lovely "sleep"
function query).

I'm also curious to hear what the websolr people do, or anyone else that
hosts Solr for the JavaScript app development crowd.

Cheers
-Doug


On Friday, December 25, 2015, Shawn Heisey  wrote:

> On 12/25/2015 12:17 PM, Eric Dain wrote:
> > Does allowing javascript direct access to SolrCloud raise security
> concern?
> > should I build a REST service in between?
> >
> > I need to provide async search capability to web pages. the pages will be
> > public with no authentication.
>
> End users should never have access to Solr.  Access to Solr from the
> end-user machine is required if you want to accept Solr responses directly.
>
> In one of the other replies that you received, Doug has given you an
> nginx config for proxying access to Solr -- indirect access.  This can
> protect against *changes* to the index, and it has protection against
> high start/rows values, but there are many other ways that an attacker
> can construct denial of service queries, which this proxy config will
> not prevent.
>
> I think that indirect access (through a proxy) should not be allowed
> either, unless you can trust all the people that will have access.
>
> If Solr is open to a sufficiently wide audience (especially the
> Internet), someone will find a way to abuse the service even with a
> proxy, either to cause harm or to learn things they shouldn't know.
>
> The most secure option is to only allow the webservers and trusted
> administrators to access Solr.  All end user (Internet) access to Solr
> should be handled through a custom web application.  This might be
> something that you find and install (such as wordpress, drupal, etc), or
> one that you write yourself.
>
> You can still do AJAX while maintaining security.  You'll need to write
> something in a server-side web programming language like PHP, Java, etc.
>  This code will need to accept the AJAX requests from your client-side
> javascript code, validate the request parameters to make sure they're
> sane, get a response from Solr, and return relevant data.  If the
> parameters don't validate, return an error, and handle that error
> appropriately in the javascript code.
>
> Thanks,
> Shawn
>
>

-- 
*Doug Turnbull **| *Search Relevance Consultant | OpenSource Connections
, LLC | 240.476.9983
Author: Relevant Search 
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.

Re: AJAX access to Solr Server

[Doug Turnbull]

Yeah I prefer a whitelist of locked down query request handlers via a
proxy that are reasonably well protected. I would never expose update to
the web or allow any updating over a public interface.

If you want an example, you can checkout

http://solr.quepid.com/solr/statedecoded/select?q=*:*=update=
*:*=true

http://solr.quepid.com/solr/statedecoded/update?stream.body=
*:*=true

But still get search results back:
http://solr.quepid.com/solr/statedecoded/select?q=*:*

Click all those all day long. And do let me know if you find holes... I'm
sure there's room for improvement

Cheers,
-Doug

On Friday, December 25, 2015, GW  wrote:

> If you are using Linux a simple one liner in IP tables
>
> iptables -I INPUT \! --src www.yourwebserver.com -m tcp -p tcp --dport
> 8983 -j DROP
>
>
> If windows, you can do something similar
>
> otherwise it is very easy for anyone to delete all your documents with
>
> http://yoursolrserver.com:8983/solr/your-core/update?stream.body=
> *:*=true
>
>
>
>
> On 25 December 2015 at 20:42, Doug Turnbull <
> dturnb...@opensourceconnections.com > wrote:
>
> > Hi Shawn
> >
> > Maybe I should have qualified the parameters of scenarios this make me
> > comfortable just proxying Solr directly w/o an API
> >
> > These situations include:
> >
> > 1. I've got no qualms about giving the whole world access to every
> document
> > in the index. There's nothing protected about anything.
> > 2. The content can be easily rebuilt , it's not very large. (I can easily
> > push a button and make a new one)
> >
> > Sure you can denial of service Solr, and I might lose my search index.
> But
> > you can denial of service anything. This includes just about anything you
> > put in front of Solr. Moreover, the added complexity of a
> > Drupal/Wordpress/your API might only add to the security problems with
> > their own security issues. I'd rather keep it simple and have fewer
> moving
> > parts.
> >
> > Cases where I would want an API in front of Solr (these are just the
> > security ones):
> > - I want to protect the content (ie based on some notion of a "user" or
> > other permissions)
> > - Rebuilding the content would be very hard and time consuming
> >
> > I would also say to expose Solr directly to everyone you probably should
> > know about Solr's bugaboos:
> > - the lovely qt parameter and the request dispatcher (the nginx proxy
> below
> > disallows qt)
> > - deep paging (prevented by the nginx proxy)
> > - how to lock down a request handler fairly robustly, how to use
> invariants
> > - mitigating intentionally malicious queries (such as the lovely "sleep"
> > function query).
> >
> > I'm also curious to hear what the websolr people do, or anyone else that
> > hosts Solr for the JavaScript app development crowd.
> >
> > Cheers
> > -Doug
> >
> >
> > On Friday, December 25, 2015, Shawn Heisey  > wrote:
> >
> > > On 12/25/2015 12:17 PM, Eric Dain wrote:
> > > > Does allowing javascript direct access to SolrCloud raise security
> > > concern?
> > > > should I build a REST service in between?
> > > >
> > > > I need to provide async search capability to web pages. the pages
> will
> > be
> > > > public with no authentication.
> > >
> > > End users should never have access to Solr.  Access to Solr from the
> > > end-user machine is required if you want to accept Solr responses
> > directly.
> > >
> > > In one of the other replies that you received, Doug has given you an
> > > nginx config for proxying access to Solr -- indirect access.  This can
> > > protect against *changes* to the index, and it has protection against
> > > high start/rows values, but there are many other ways that an attacker
> > > can construct denial of service queries, which this proxy config will
> > > not prevent.
> > >
> > > I think that indirect access (through a proxy) should not be allowed
> > > either, unless you can trust all the people that will have access.
> > >
> > > If Solr is open to a sufficiently wide audience (especially the
> > > Internet), someone will find a way to abuse the service even with a
> > > proxy, either to cause harm or to learn things they shouldn't know.
> > >
> > > The most secure option is to only allow the webservers and trusted
> > > administrators to access Solr.  All end user (Internet) access to Solr
> > > should be handled through a custom web application.  This might be
> > > something that you find and install (such as wordpress, drupal, etc),
> or
> > > one that you write yourself.
> > >
> > > You can still do AJAX while maintaining security.  You'll need to write
> > > something in a server-side web programming language like PHP, Java,
> etc.
> > >  This code will need to accept the AJAX requests from your client-side
> > > javascript code, validate the request parameters to make sure they're
> > > sane, get a response from Solr, and return relevant data.  If the
> > > parameters don't validate, return an error,