Lorenz Quack created QPID-6710:
----------------------------------
Summary: NPE masks IOException on running out of disk space
Key: QPID-6710
URL: https://issues.apache.org/jira/browse/QPID-6710
Project: Qpid
Issue Type: Bug
Components: Java Broker
Reporter: Lorenz Quack
When the broker is running out of disk space the following issue can occur.
{noformat}
########################################################################
#
# Unhandled Exception org.apache.qpid.server.store.StoreException: Unexpected
exception occurred on store operation in Thread virtualhost-default-iopool-55
#
# Exiting
#
########################################################################
org.apache.qpid.server.store.StoreException: Unexpected exception occurred on
store operation
at
org.apache.qpid.server.store.berkeleydb.StandardEnvironmentFacade.handleDatabaseException(StandardEnvironmentFacade.java:287)
at
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.removeMessage(AbstractBDBMessageStore.java:317)
at
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.remove(AbstractBDBMessageStore.java:1234)
at
org.apache.qpid.server.message.AbstractServerMessageImpl.decrementReference(AbstractServerMessageImpl.java:101)
at
org.apache.qpid.server.message.AbstractServerMessageImpl.access$500(AbstractServerMessageImpl.java:37)
at
org.apache.qpid.server.message.AbstractServerMessageImpl$Reference.release(AbstractServerMessageImpl.java:275)
at
org.apache.qpid.server.protocol.v0_8.AMQChannel.deliverCurrentMessageIfComplete(AMQChannel.java:526)
at
org.apache.qpid.server.protocol.v0_8.AMQChannel.publishContentBody(AMQChannel.java:655)
at
org.apache.qpid.server.protocol.v0_8.AMQChannel.receiveMessageContent(AMQChannel.java:2519)
at org.apache.qpid.framing.ContentBody.process(ContentBody.java:105)
at org.apache.qpid.codec.AMQDecoder.processFrame(AMQDecoder.java:394)
at
org.apache.qpid.server.protocol.v0_8.BrokerDecoder.doProcessFrame(BrokerDecoder.java:114)
at
org.apache.qpid.server.protocol.v0_8.BrokerDecoder.access$000(BrokerDecoder.java:37)
at
org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:80)
at
org.apache.qpid.server.protocol.v0_8.BrokerDecoder$1.run(BrokerDecoder.java:76)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.qpid.server.protocol.v0_8.BrokerDecoder.processFrame(BrokerDecoder.java:75)
at org.apache.qpid.codec.AMQDecoder.processInput(AMQDecoder.java:370)
at org.apache.qpid.codec.AMQDecoder.decodeBuffer(AMQDecoder.java:259)
at
org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8$2.run(AMQPConnection_0_8.java:319)
at
org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8$2.run(AMQPConnection_0_8.java:299)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.qpid.server.protocol.v0_8.AMQPConnection_0_8.received(AMQPConnection_0_8.java:298)
at
org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:138)
at
org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:465)
at
org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:45)
at
org.apache.qpid.server.transport.NonBlockingConnection.processData(NonBlockingConnection.java:398)
at
org.apache.qpid.server.transport.NonBlockingConnection.readAndProcessData(NonBlockingConnection.java:349)
at
org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.doRead(NonBlockingConnectionPlainDelegate.java:39)
at
org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:337)
at
org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:229)
at
org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:119)
at
org.apache.qpid.server.transport.NetworkConnectionScheduler.access$000(NetworkConnectionScheduler.java:37)
at
org.apache.qpid.server.transport.NetworkConnectionScheduler$2.run(NetworkConnectionScheduler.java:100)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.removeMessage(AbstractBDBMessageStore.java:227)
... 37 more
{noformat}
Broker log file might have the issue reported in some cases as:
{noformat}
2015-07-23 05:19:30,460 INFO [default:FileSystemSpaceChecker]
(q.m.v.filesystem_full) - [default:FileSystemSpaceChecker]
[vh(/default)/ms(BDBMessageStore)] VHT-1006 : Filesystem is over 90 per cent
full, enforcing flow control.
2015-07-23 05:19:46,668 ERROR [IO-/169.124.113.161:58633]
(o.a.q.s.u.ServerScopedRuntimeException) - Unexpected exception occurred on
store operation
com.sleepycat.je.LogWriteException: (JE 5.0.104) Environment must be closed,
caused by: com.sleepycat.je.LogWriteException: Environment invalid because of
previous exception: (JE 5.0.104)
/apps/qpid/destruction-tests/qpid-broker/6.0.0-SNAPSHOT/work/default/messages
java.io.IOException: No space left on device LOG_WRITE: IOExcep
{noformat}
However, in some other cases exception is not get stored in the logs leaving
only NPE (reported into system output) to be the only evidence of an issue
which might confuse the user about the nature of a problem
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
