[
https://issues.apache.org/jira/browse/QPID-6710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rudyy updated QPID-6710:
-----------------------------
Attachment: 0001-QPID-6710-Handle-JE-exceptions-in-order-to-restart-e.patch
> NPE masks IOException on running out of disk space
> --------------------------------------------------
>
> Key: QPID-6710
> URL: https://issues.apache.org/jira/browse/QPID-6710
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Reporter: Lorenz Quack
> Attachments:
> 0001-QPID-6710-Handle-JE-exceptions-in-order-to-restart-e.patch,
> 0001-QPID-6710-Java-Broker-NPE-masks-IOException-on-runni.patch,
> delegate-exception-throwing-to-je-and-handle-je-exceptions-everywhere.diff
>
>
> When the broker is running out of disk space the following issue can occur.
> {noformat}
> ########################################################################
> #
> # Unhandled Exception org.apache.qpid.server.store.StoreException: Unexpected
> exception occurred on store operation in Thread virtualhost-default-iopool-55
> #
> # Exiting
> #
> ########################################################################
> org.apache.qpid.server.store.StoreException: Unexpected exception occurred on
> store operation
> at
> org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.handleDatabaseException(StandardEnvironmentFacade.java:287)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.removeMessage(AbstractBDBMessageStore.java:317)
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.remove(AbstractBDBMessageStore.java:1234)
> at
> org.apache.qpid.server.message.AbstractServerMessageImpl.decrementReference(AbstractServerMessageImpl.java:101)
> at
> org.apache.qpid.server.message.AbstractServerMessageImpl.access$500(AbstractServerMessageImpl.java:37)
> at
> org.apache.qpid.server.message.AbstractServerMessageImpl$Reference.release(AbstractServerMessageImpl.java:275)
> at
> org.apache.qpid.server.protocol.v0_8.AMQChannel.deliverCurrentMessageIfComplete(AMQChannel.java:526)
> at
> org.apache.qpid.server.protocol.v0_8.AMQChannel.publishContentBody(AMQChannel.java:655)
> at
> org.apache.qpid.server.protocol.v0_8.AMQChannel.receiveMessageContent(AMQChannel.java:2519)
> at org.apache.qpid.framing.ContentBody.process(ContentBody.java:105)
> at org.apache.qpid.codec.AMQDecoder.processFrame(AMQDecoder.java:394)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder.doProcessFrame(BrokerDecoder.java:114)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder.access$000(BrokerDecoder.java:37)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:80)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:76)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.qpid.server.protocol.v0_8.BrokerDecoder.processFrame(BrokerDecoder.java:75)
> at org.apache.qpid.codec.AMQDecoder.processInput(AMQDecoder.java:370)
> at org.apache.qpid.codec.AMQDecoder.decodeBuffer(AMQDecoder.java:259)
> at
> org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8$2.run(AMQPConnection_0_8.java:319)
> at
> org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8$2.run(AMQPConnection_0_8.java:299)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8.received(AMQPConnection_0_8.java:298)
> at
> org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:138)
> at
> org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:465)
> at
> org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:45)
> at
> org.apache.qpid.server.transport.NonBlockingConnection.processData(NonBlockingConnection.java:398)
> at
> org.apache.qpid.server.transport.NonBlockingConnection.readAndProcessData(NonBlockingConnection.java:349)
> at
> org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.doRead(NonBlockingConnectionPlainDelegate.java:39)
> at
> org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:337)
> at
> org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:229)
> at
> org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:119)
> at
> org.apache.qpid.server.transport.NetworkConnectionScheduler.access$000(NetworkConnectionScheduler.java:37)
> at
> org.apache.qpid.server.transport.NetworkConnectionScheduler$2.run(NetworkConnectionScheduler.java:100)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.removeMessage(AbstractBDBMessageStore.java:227)
> ... 37 more
> {noformat}
> Broker log file might have the issue reported in some cases as:
> {noformat}
> 2015-07-23 05:19:30,460 INFO [default:FileSystemSpaceChecker]
> (q.m.v.filesystem_full) - [default:FileSystemSpaceChecker]
> [vh(/default)/ms(BDBMessageStore)] VHT-1006 : Filesystem is over 90 per cent
> full, enforcing flow control.
> 2015-07-23 05:19:46,668 ERROR [IO-/169.124.113.161:58633]
> (o.a.q.s.u.ServerScopedRuntimeException) - Unexpected exception occurred on
> store operation
> com.sleepycat.je.LogWriteException: (JE 5.0.104) Environment must be closed,
> caused by: com.sleepycat.je.LogWriteException: Environment invalid because of
> previous exception: (JE 5.0.104)
> /apps/qpid/destruction-tests/qpid-broker/6.0.0-SNAPSHOT/work/default/messages
> java.io.IOException: No space left on device LOG_WRITE: IOExcep
> {noformat}
> However, in some other cases exception is not get stored in the logs leaving
> only NPE (reported into system output) to be the only evidence of an issue
> which might confuse the user about the nature of a problem
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
