Attached patch increases the maximum possible log file number to 2^31
-1 and keeps the old limit of 2^22-1 on soft upgrade.
Earlier patch for this problem increased it to 2^33 -1 , but that fix
created upgrade problems because of the additional bits
that are used from the file position field, so the limit is being
reduced to 2^31-1.
Thanks
-suresht
Mike Matrigali wrote:
This seems like a reasonable 1st step, that would solve current
upgrade issues. I suggest you submit this patch, close the current
jira, and submit a new future improvement jira noting what would
have to be done to eek out the last few bits (or documenting it
somewhere appropriate in the code is also fine - just rather not see
you have learned lost). Given that this
soft upgrade stuff is new I would rather be safe this time.
/mikem
Suresh Thalamati wrote:
By looking at the code , it is kind of tricky to handle upgrade,
because users can crash in the middle of recovery
and reboot again N times , which can generate new log for uncommitted
transaction and also all the constant that manipulates LSN
are static now and are used by static routines in
org.apache.derby.impl.store.raw.log.LogCounter.java, like
:makeLogInstantAsLong().
To handle upgrade:
1) all static methods in LogCounter.java has to be replaced by
instance methods , so that
in the same JVM, one can run a database that is in softupgrade
mode and one that is new/upgraded..
2) modify the code to update the version number on disk(log control
file ) to only when the recovery is complete;
currently it is updated in the boot code itself.
3) perform a checkpoint after recovery on upgrade, so that new log
file limits can be uses, As it will make
sure that no old log instant format will be in use.
Instead of doing all of the above , I can reduce the limit to 2^31
-1 instead of 2^33-1 ;
this way I would not need to change LogCounter.FILE_NUMBER_SHIFT
from 32 (old value) to 30,
which causes all upgrade issues.
with 2^31 -1 it will take approximately 68 years (2147483647/(24 * 60
* 60 * 365))
to hit the log file number limit if a new log file is generated
every second., instead of 272 years
with 2^33-1.
If no one has objections , I would prefer to increase the log file
number limit to 2^31 -1 and
avoid major upgrade issues. If any one strongly believes it should
be 2^33 -1 , I will try
to do the necessary upgrade changes.
Any comments/suggestions ?
Thanks
-suresht
Suresh Thalamati wrote:
I just realized , this patch actaully break's hard upgrade also, if
log needs to be replayed.
Becuase I extract the log file number and log file postion from the
instant that is on disk ,
differently after my change.
If any one else on the list is testing upgrade is hitting this
problem, please let me know.
I can post a patch to backout the portion that cause upgrade issues.
Thanks
-suresht
Mike Matrigali wrote:
I am looking at committing this. The changes look good to me, but I
believe there are upgrade issues to handle.
For a hard upgrade either new or old databases are fine.
For a soft upgrade I think there is a problem if the db generates
enough
log files to start using the new bits, and then the software is
reverted
to before the fix.
Seems like using the bits needs to be somehow only enabled for hard
upgrade. It would be best if it was controlled just by hard upgrade,
but if that is not possible then just doing it for databases created
since this version would also work - but still leave problems with old
hard upgraded databases.
I have reviewed the code and am running tests. I plan on committing
this part of the fix and let you address upgrade issues with a
follow on
patch.
Suresh Thalamati wrote:
Index: java/engine/org/apache/derby/impl/store/raw/log/LogCounter.java
===================================================================
--- java/engine/org/apache/derby/impl/store/raw/log/LogCounter.java
(revision 171180)
+++ java/engine/org/apache/derby/impl/store/raw/log/LogCounter.java
(working copy)
@@ -60,18 +60,18 @@
/** A well defined value of an invalid log instant. */
public static final long INVALID_LOG_INSTANT = 0;
+
+ // max possible log file number in versions before 10.1 is 2^22 -1
+ public static final long DERBY_10_0_MAX_LOGFILE_NUMBER =
(long)0x003FFFFFL; // 4194303
+ // max possible log file number is 2^31 -1
+ public static final long MAX_LOGFILE_NUMBER = (long)0x7FFFFFFFL; //
2147483647
+ // lower end of 32 bits in long type are used to store the log file
position
+ private static final long FILE_NUMBER_SHIFT = 32;
-
- // max possible log file number is 2^33 -1 (8589934591)
- public static final long MAX_LOGFILE_NUMBER = (long)0x1FFFFFFFFL;
-
- // lower end of 30 bits in long type are used to store the log file
position
- private static final long FILE_NUMBER_SHIFT = 30;
-
- // reserve top 2 bits in log file size for future use
+ // reserve top 4 bits in log file size for future use
public static final long MAX_LOGFILE_SIZE =
(long)0x0FFFFFFFL; // 268435455
- // 30 bits are used to store the log file postion
- private static final long FILE_POSITION_MASK = (long)0x3FFFFFFFL;
+ // 32 bits are used to store the log file postion
+ private static final long FILE_POSITION_MASK = (long)0x7FFFFFFFL;
private long fileNumber;
private long filePosition;
Index: java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
===================================================================
--- java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
(revision 171180)
+++ java/engine/org/apache/derby/impl/store/raw/log/LogToFile.java
(working copy)
@@ -333,7 +333,7 @@
// must be
synchronized with this to access
// or change.
-
+ private long maxLogFileNumber =
LogCounter.MAX_LOGFILE_NUMBER;
private CheckpointOperation currentCheckpoint;
// last
checkpoint successfully taken
//
@@ -1831,11 +1831,11 @@
// used.
StorageFile newLogFile =
getLogFileName(logFileNumber+1);
- if (logFileNumber+1 >= LogCounter.MAX_LOGFILE_NUMBER)
+ if (logFileNumber+1 >= maxLogFileNumber)
{
throw StandardException.newException(
SQLState.LOG_EXCEED_MAX_LOG_FILE_NUMBER,
- new Long(LogCounter.MAX_LOGFILE_NUMBER));
+ new Long(maxLogFileNumber));
}
StorageRandomAccessFile newLog = null; // the new log
file
@@ -3059,6 +3059,21 @@
{
throw Monitor.exceptionStartingModule(ioe);
}
+
+ // Number of the log file that can be created in Derby is
increased from
+ // 2^22 -1 to 2^31 -1 in version 10.1. But if the database is
running on
+ // engines 10.1 or above on a softupgrade from versions 10.0
or
+ // before, the max log file number that can be created is
+ // still limited to 2^22 -1, because users can revert back to
older versions
+ // which does not have logic to handle a log file number
greater than
+ // 2^22-1.
+
+ // set max possible log file number to derby 10.0 limit, if the
database is not
+ // fully upgraded to or created in version 10.1 or above.
+ if (!checkVersion(RawStoreFactory.DERBY_STORE_MAJOR_VERSION_10,
+
RawStoreFactory.DERBY_STORE_MINOR_VERSION_1))
+ maxLogFileNumber =
LogCounter.DERBY_10_0_MAX_LOGFILE_NUMBER;
+
} // end of boot
private void getLogStorageFactory() throws StandardException
