Thanks Eric, fixed the issue. The IT team corrected the solrconfig.xml but
forgot to execute the zkcli.sh script on solr node. After I executed the script
its working now.
On 9/20/17, 10:20 AM, "Erick Erickson" <erickerick...@gmail.com> wrote:
WARNING - External email; exercise caution.
Your solrconfig.xml file is mal-formed. The smoking gun is:
Exception during parsing file: solrconfig.xml
Best,
Erick
On Tue, Sep 19, 2017 at 4:48 PM, Satyaprashant Bezwada
<satyaprashant.bezw...@nasdaq.com> wrote:
> Need some inputs or help in resolving replication across solr nodes. We
have installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes
configured. Enabled Solr replication in my Solrj client but the replication
fails and is unable to create a collection.
>
> The same code works in our different environment where in I have 1
zookeeper and 3 Solr nodes configured. Here is the exception I see on one of
the nodes of Solr when I try to create a collection in the environment where it
fails. I have compared the Solrconfig.xml on both the environments and didn’t
see any difference.
>
>
>
> 2017-09-19 22:09:35.471 ERROR
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may
have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ]
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer
Overseer (id=170740497916499012-sr01:8983_solr-n_0000000029) closing
> 2017-09-19 22:38:36.645 INFO
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ]
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
> Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
/var/solr/logs/solr.log
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-19 22:09:35.471 ERROR
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_0000000029)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-0000000410 doesn't exist. Requestor may
have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ]
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer
Overseer (id=170740497916499012-sr01:8983_solr-n_0000000029) closing
> 2017-09-19 22:38:36.645 INFO
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_0000000029) [ ]
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
/var/solr/logs/solr.log
> 2017-09-19 23:12:22.230 INFO (main) [ ] o.a.s.s.SolrDispatchFilter
Loading solr.xml from SolrHome (not found in ZooKeeper)
> 2017-09-19 23:12:22.233 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading
container configuration from /var/solr/data/Node122/solr.xml
> 2017-09-19 23:12:22.644 INFO (main) [ ] o.a.s.u.UpdateShardHandler
Creating UpdateShardHandler HTTP client with params:
socketTimeout=600000&connTimeout=60000&retry=true
> 2017-09-19 23:12:22.650 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper
client=zk01:2181,zk02:2181,zk03:2181
> 2017-09-19 23:12:22.731 INFO (main) [ ] o.a.s.c.c.ZkStateReader
Updated live nodes from ZooKeeper... (0) -> (1)
> 2017-09-19 23:12:22.762 INFO (main) [ ] o.a.s.c.Overseer Overseer
(id=null) closing
> 2017-09-19 23:12:22.780 INFO (main) [ ] o.a.s.c.ZkController Register
node as live in ZooKeeper:/live_nodes/sr01:8983_solr
> 2017-09-19 23:12:22.788 INFO
(zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
> 2017-09-19 23:12:23.241 INFO (main) [ ] o.a.s.c.CorePropertiesLocator
Found 0 core definitions underneath /var/solr/data/Node122
> 2017-09-19 23:12:23.374 INFO (main) [ ] o.e.j.s.Server Started @2345ms
> 2017-09-19 23:12:47.125 INFO (qtp401424608-13) [ ]
o.a.s.h.a.CoreAdminOperation core create command
qt=/admin/cores&collection.configName=meetx&newCollection=true&name=CIUZLW_shard1_replica1&action=CREATE&numShards=1&collection=CIUZLW&shard=shard1&wt=javabin&version=2
> 2017-09-19 23:12:47.256 INFO
(zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json]
for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
> 2017-09-19 23:12:48.148 INFO
(zkCallback-4-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged path:/collections/CIUZLW/state.json]
for collection [CIUZLW] has occurred - updating... (live nodes size: [2])
> 2017-09-19 23:12:48.175 ERROR (qtp401424608-13) [c:CIUZLW s:shard1
x:CIUZLW_shard1_replica1] o.a.s.c.Config Exception during parsing file:
solrconfig.xml:org.xml.sax.SAXParseException; systemId:
solrres:/solrconfig.xml; lineNumber: 135; columnNumber: 13; Open quote is
expected for attribute "name" associated with an element type "str".
> at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
> at
com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:177)
> at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
> at
com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
> at
com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1437)
> at
com.sun.org.apache.xerces.internal.impl.XMLScanner.scanAttributeValue(XMLScanner.java:830)
> at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanAttribute(XMLNSDocumentScannerImpl.java:440)
> at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.scanStartElement(XMLNSDocumentScannerImpl.java:256)
> at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)
> at
com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at
com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:118)
> at
com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at
com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at
com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
> at
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
> at org.apache.solr.core.Config.<init>(Config.java:143)
> at org.apache.solr.core.Config.<init>(Config.java:90)
> at
org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:203)
> at
org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:180)
> at
org.apache.solr.core.ConfigSetService.createSolrConfig(ConfigSetService.java:96)
> at
org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:77)
> at
org.apache.solr.core.CoreContainer.create(CoreContainer.java:915)
> at
org.apache.solr.core.CoreContainer.create(CoreContainer.java:854)
> at
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:88)
> at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:370)
> at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:388)
> at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
> at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:748)
> at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:729)
> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:510)
> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:745)
>
> ***********************************************************
> CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the
exclusive and confidential use of the intended recipient and may constitute
non-public information. If you received this e-mail in error, disclosing,
copying, distributing or taking any action in reliance of this e-mail is
strictly prohibited and may be unlawful. Instead, please notify us immediately
by return e-mail and promptly delete this message and its attachments from your
computer system. We do not waive any work product or other applicable legal
privilege(s) by the transmission of this message.
> ***********************************************************
***********************************************************
CONFIDENTIALITY NOTICE: This e-mail and any attachments are for the exclusive
and confidential use of the intended recipient and may constitute non-public
information. If you received this e-mail in error, disclosing, copying,
distributing or taking any action in reliance of this e-mail is strictly
prohibited and may be unlawful. Instead, please notify us immediately by return
e-mail and promptly delete this message and its attachments from your computer
system. We do not waive any work product or other applicable legal privilege(s)
by the transmission of this message.
***********************************************************
