After adding support for the IBatch interface to IBX and then stress
testing the implementation, I found that IBatch, when used with a remote
server was silently losing data when the buffer size was exceeded. That
issue has now been fixed.
See - https://github.com/FirebirdSQL/firebird/issues/6900
However, I believe that more needs to be done to resolve the issue and
any resolution needs to have a wider discussion.
The current situation is IBatch::add returns an error when the internal
buffer overflows:
* when used with an embedded server, this appears always to happen when
IBatch::add is called for the message buffer that causes the overflow.
It is thus possible to recover by calling IBatch::execute to apply all
the previous message buffers and then starting a new batch and call
IBatch::add again for the same message buffer.
* when used with a remote server, my understanding is that the error is
generated on the server side and is always many messages behind.
IBatch::execute can still be called to apply the batch, but only by
looking at the completion data can you work out at which point the
overflow occurred, and hence from which message buffer you need to restart.
My problem is with the remote server case and that it seems to be
difficult to determine, in advance, how many message buffers to keep a
copy of in order to recover from the error.
With my test data, the buffer overflow occurs after several thousand
calls to IBatch::add. The number of message buffers lost is of the order
of hundreds. I don't want to have to keep a copy of every message buffer
as that could result in other out of memory issues.
My problem is that the number of message buffers that may be lost on
buffer overflow appears to be indeterminate. It would be more useful if
when IBatch is created, I could specify a message buffer limit and the
internal buffers were allocated in advance. Alternatively, I could also
work with an IBatch method that told me the minimum number of guaranteed
calls to IBatch::add (for a given message buffer size). Other
suggestions are welcome.
I believe that a solution is necessary because you cannot use IBatch for
any large scale use without knowing what you need to do to recover from
a buffer overflow error without losing user data.
Tony Whyman
MWA
Firebird-Devel mailing list, web interface at
https://lists.sourceforge.net/lists/listinfo/firebird-devel
