[
https://issues.apache.org/jira/browse/AMQ-7010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16559646#comment-16559646
]
Gary Tully commented on AMQ-7010:
---------------------------------
I agree. It is best to stop. The latest update to this feature introduced
"systemExitOnShutdown" to force a quick termination.
It turns out that is is not safe to continue to write on a "no space" error,
some smaller write may succeed and async writes will succeed and lead to
invalid locations in the index. It gets messy quickly.
The intent is that the appender stops, the IOExceptionHandler can suspend
connections till a checkpoint succeeds, then enable the appender again; that is
reasonably safe.
But simply ignoring all errors is no longer a flyer.
The safest approach from a consistency point of view is to shutdown and let
recovery deal with any partial writes.
>From a client perspective, shutdown is not unlike stopStartClientConnections,
>but it does require something to restart the broker.
This does point to the need to update the documentation to reflect how this has
evolved.
> No space left on device leading to broker shutdown even if ignored
> ------------------------------------------------------------------
>
> Key: AMQ-7010
> URL: https://issues.apache.org/jira/browse/AMQ-7010
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB
> Affects Versions: 5.15.0, 5.15.1, 5.15.2, 5.15.3
> Reporter: Laurent Chiarello
> Priority: Major
> Attachments: activemq_logs.txt
>
>
> Recently, we upgraded our ActiveMQ instance from {{5.13.2}} to {{5.15.3}}. We
> basically use an almost default broker setup with KahaDB. Due to our setup,
> we occasionally run into {{java.io.IOException: No space left on device}},
> which until now was fine since the broker would properly ignore them using
> the {{DefaultIOExceptionHandler}}, and resume its work once space is
> available again.
> After upgrading to {{5.15.3}}, however, we noticed that the broker would
> actually stop itself following a _no space left_ exception. Digging a little
> bit, it seems that the change in behavior can be linked to a fix introduced
> in the {{5.15.0}} release, see AMQ-6625. With this fix, KahaDB would
> sometimes throw the following exception: {{java.io.IOException: Async Writer
> Thread Shutdown}} from its {{DataFileAppender}}, which, unfortunately, is not
> handled by the {{DefaultIOExceptionHandler}} unless {{ignoreAllErrors}} is
> set to true.
> This would basically mean that the {{DefaultIOExceptionHandler}} has become
> useless in catching only _no space left_ exceptions, which I do not think is
> the intended behavior.
> h3. How to reproduce
> # Start a broker with default config on a small partition
> # Send a persistent message with web console to any queue
> # Fill the partition by any mean
> # Send more messages with web console, broker is eventually shut down
> See the attached broker logs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
