Re: Can you change the kahadb journal file size between broker starts

[jahlborn]

Thanks again for your help.  I created
https://issues.apache.org/jira/browse/AMQ-6572 .

I suspect that the second situation is probably an edge case which would be
rarely hit (well, both are probably edge cases).  I think the consequences
of the second may be "harmless"...?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Can-you-change-the-kahadb-journal-file-size-between-broker-starts-tp4721155p4721257.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Can you change the kahadb journal file size between broker starts

[Christopher Shannon]

Yes, I would expect maxFileLength should only be used on file
allocation.  My expectation is that the journal file length should be
able to be changed in between restarts and have everything work.  In
the past I have increased the size many times before without issues so
I'm not sure if the second use case is an issue (KahaDB might handle
it elsewhere) but the first portion of the startup code seems like it
might be a problem and need to be fixed.

Can you create a new Jira? I should have time to take a peak at this
next week to verify if a fix is needed.  It should be pretty
straightforward to write a quick unit test to check both scenarios
(length increasing and decreasing) to show if there is an issue and
validate any fix.

On Fri, Jan 20, 2017 at 9:58 AM, jahlborn  wrote:
> So, i was poking around to see the actual implications of changing the max
> journal file length, and i must say, the code doesn't seem to back up the
> claim that it is a safe thing to do.
>
> https://fisheye.apache.org/browse/activemq-6/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal/Journal.java?hb=true#to310
>
> This is the startup code.  it attempts to adjust the total length by shaving
> off an unused portion of the final data file.  if you have made the
> maxFileLength smaller since the last run, and you have actual data in the
> last journal file which is after the new maxFileLength, then this
> computation will incorrectly return a negative value.  my suspicion is that
> the length of the last data file should be used here instead of
> maxFileLength.
>
> https://fisheye.apache.org/browse/activemq-6/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal/Journal.java?hb=true#to867
>
> likewise, this code attempts to handle adjustments to maxFileLength since
> the journal file was created, but i think it again fails.  if the
> maxFileLength has been increased since the data file was created, this would
> seem to be setting an offset which is past the length of the current data
> file.  again, it seems like the length of the data file should be used
> directly.
>
> in general, in order for this to work correctly, the maxFileLength should
> only be used for code which is creating new files.  all the other code
> should be solely relying on the size of the existing file, right?
>
>
>
> --
> View this message in context: 
> http://activemq.2283324.n4.nabble.com/Can-you-change-the-kahadb-journal-file-size-between-broker-starts-tp4721155p4721252.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Can you change the kahadb journal file size between broker starts

[jahlborn]

So, i was poking around to see the actual implications of changing the max
journal file length, and i must say, the code doesn't seem to back up the
claim that it is a safe thing to do.

https://fisheye.apache.org/browse/activemq-6/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal/Journal.java?hb=true#to310

This is the startup code.  it attempts to adjust the total length by shaving
off an unused portion of the final data file.  if you have made the
maxFileLength smaller since the last run, and you have actual data in the
last journal file which is after the new maxFileLength, then this
computation will incorrectly return a negative value.  my suspicion is that
the length of the last data file should be used here instead of
maxFileLength.

https://fisheye.apache.org/browse/activemq-6/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/journal/Journal.java?hb=true#to867

likewise, this code attempts to handle adjustments to maxFileLength since
the journal file was created, but i think it again fails.  if the
maxFileLength has been increased since the data file was created, this would
seem to be setting an offset which is past the length of the current data
file.  again, it seems like the length of the data file should be used
directly.

in general, in order for this to work correctly, the maxFileLength should
only be used for code which is creating new files.  all the other code
should be solely relying on the size of the existing file, right?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Can-you-change-the-kahadb-journal-file-size-between-broker-starts-tp4721155p4721252.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Can you change the kahadb journal file size between broker starts

[Clebert Suconic]

That is the same with artemis actually.

On Wed, Jan 18, 2017 at 7:53 AM, Christopher Shannon
 wrote:
> Yes, that is safe to do.  The journalMaxFileLength value is only used
> for allocating new journal files and doesn't have any effect on
> previous files.  If you change the value the broker will continue to
> work fine and be able to read the old files but all new allocated
> files will be the new size.
>
> On Tue, Jan 17, 2017 at 4:28 PM, jahlborn  wrote:
>> I was wondering if it was safe to change the kahadb "journalMaxFileLength"
>> cofiguration value between restarts of a broker.  in other words, will the
>> broker function correctly with different journal files being different
>> sizes?
>>
>>
>>
>> --
>> View this message in context: 
>> http://activemq.2283324.n4.nabble.com/Can-you-change-the-kahadb-journal-file-size-between-broker-starts-tp4721155.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



-- 
Clebert Suconic

Re: Can you change the kahadb journal file size between broker starts

[jahlborn]

thanks!



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Can-you-change-the-kahadb-journal-file-size-between-broker-starts-tp4721155p4721162.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Can you change the kahadb journal file size between broker starts

[Christopher Shannon]

Yes, that is safe to do.  The journalMaxFileLength value is only used
for allocating new journal files and doesn't have any effect on
previous files.  If you change the value the broker will continue to
work fine and be able to read the old files but all new allocated
files will be the new size.

On Tue, Jan 17, 2017 at 4:28 PM, jahlborn  wrote:
> I was wondering if it was safe to change the kahadb "journalMaxFileLength"
> cofiguration value between restarts of a broker.  in other words, will the
> broker function correctly with different journal files being different
> sizes?
>
>
>
> --
> View this message in context: 
> http://activemq.2283324.n4.nabble.com/Can-you-change-the-kahadb-journal-file-size-between-broker-starts-tp4721155.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.