We’re having quite a bit of trouble with ELBs in AWS. There is an ELB in
front of the service/app nodes, and another ELB between them and the three node
MarkLogic cluster.
We’ve had problems with the ELB in front of ML marking nodes down if they
are not responsive enough, which results in 504 responses on calls to ML after
that. We’ve had to crank up the timeouts to fairly high levels to avoid one
slow query from causing subsequent queries to fail.
That’s annoying but manageable. What’s really problematic is apparent data
corruption when using XCC, specifically CORB.
For most calls to MarkLogic, we’re using our bespoke REST interface using
URL rewriting on an HTTP appserver. We don’t use XCC normally. But we’re
doing some one time batch reprocessing with CORB. Pointing CORB at one of the
E-nodes works fine. But if we point it at the ELB (which works fine for the
REST calls) then we randomly get failures like the one below.
Since I originally wrote XCC, I know that this error message is the result
of the data stream being out of sync with what XCC is expecting to see. It’s
looking for a sub-header in the partitioned response that describes the XDM
type of the next result item. It’s seeing something else instead.
Are there known issues with ELBs and XCC/XDBC and/or CORB? Is there an ELB
setting get around this? I have a feeling that the ELB may be munging the
response in some way that is tripping up XCC.
Any help appreciated.
=======
INFO: monitoring 1,171,599 tasks
Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.Manager populateQueue
INFO: received first uri:
urn:cambridge.org:id:content-bundle:S0360966900011993:version:1
2016-06-30 15:35:23.454 SEVERE [1] (StreamingResultSequence.next): IOException
instantiating ResultItem 240: Nodes of type 'rsion' are not supported in XCC
result sequences
java.io.IOException: Nodes of type 'rsion' are not supported in XCC result
sequences
com.marklogic.xcc.impl.AbstractResultSequence.nodeFactory(AbstractResultSequence.java:248)
com.marklogic.xcc.impl.AbstractResultSequence.instantiateResultItem(AbstractResultSequence.java:202)
com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:147)
com.marklogic.xcc.impl.StreamingResultSequence.next(StreamingResultSequence.java:166)
com.marklogic.developer.corb.QueryUrisLoader.next(QueryUrisLoader.java:158)
com.marklogic.developer.corb.Manager.populateQueue(Manager.java:719)
com.marklogic.developer.corb.Manager.run(Manager.java:472)
com.marklogic.developer.corb.Manager.main(Manager.java:162)
Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.Manager stop
INFO: cleaning up
Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.QueryUrisLoader close
INFO: closing uris session
Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.Monitor run
SEVERE: interrupted: exiting
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at
java.util.concurrent.ExecutorCompletionService.poll(ExecutorCompletionService.java:202)
at com.marklogic.developer.corb.Monitor.monitorResults(Monitor.java:96)
at com.marklogic.developer.corb.Monitor.run(Monitor.java:75)
at java.lang.Thread.run(Thread.java:745)
2016-06-30 15:35:23.475 WARNING [10] (AbstractRequestController.runRequest):
Cannot obtain connection: null
java.nio.channels.ClosedByInterruptException
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
java.nio.channels.SocketChannel.open(SocketChannel.java:189)
com.marklogic.xcc.impl.SocketPoolProvider.obtainConnection(SocketPoolProvider.java:93)
com.marklogic.xcc.impl.handlers.AbstractRequestController.runRequest(AbstractRequestController.java:85)
com.marklogic.xcc.impl.SessionImpl.submitRequestInternal(SessionImpl.java:437)
com.marklogic.xcc.impl.SessionImpl.submitRequest(SessionImpl.java:432)
com.marklogic.developer.corb.AbstractTask.invokeModule(AbstractTask.java:233)
com.marklogic.developer.corb.AbstractTask.call(AbstractTask.java:148)
com.marklogic.developer.corb.AbstractTask.call(AbstractTask.java:56)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
Jun 30, 2016 3:35:23 PM com.marklogic.developer.corb.AbstractTask
handleRequestException
WARNING: Encountered ServerConnectionException from Marklogic Server. Retrying
attempt 1 after 60 seconds..: null at URI:
urn:cambridge.org:id:content-bundle:S0360966900011993:version:1
---
Ron Hitchens {[email protected]} +44 7879 358212
_______________________________________________
General mailing list
[email protected]
Manage your subscription at:
http://developer.marklogic.com/mailman/listinfo/general
