AW: [firebird-support] NBackup Levels
Hi Is there a negative effect on too many levels? (Slowdown or database storage overhead?) No negative effect. The database does track the ID of the last backup at each level, so you would have 2 or 3 more entries that the typical usage, but the details are notional in size. But as far as I understand, the database has to track which page changes since which backup. So keeping that information for more bcakups will (in the worst case) require N*M space (with N being the number of levels and M being the number of pages). Where is the fault in that logic? Regards, Steffen [Non-text portions of this message have been removed]
RE: [firebird-support] NBackup Levels
But as far as I understand, the database has to track which page changes since which backup. Correct, but that information is actually stored as a version number in the header of each data page. When page data changes the version number is changed. The nbackup database structure only stores the version number for the backup level. NBackup physically reads each page header to determine if the page version has changed since the last backup. No list of the pages that have changed is stored within the database. Sean
AW: [firebird-support] NBackup Levels
Hi Correct, but that information is actually stored as a version number in the header of each data page. When page data changes the version number is changed. The nbackup database structure only stores the version number for the backup level. From a quick search, I suspect that will be the pag_scn 4 byte value. That would mean that I could relatively easily write my own sync code that could also read all those values and decide which page would need to be transferred while the file is locked. The advantage above nbackup would simply be that I could control the way the information is merged and stored on the server better and that I could use that information to update different clients at different times (with no common base). The disadvantage would be that I would need to keep close control of these files as nbackup could not come to help to prevent incorrect ordering while merging... NBackup physically reads each page header to determine if the page version has changed since the last backup. Does NBackup do any other magic than that? Regards, Steffen [Non-text portions of this message have been removed]
RE: [firebird-support] NBackup Levels
From a quick search, I suspect that will be the pag_scn 4 byte value. I don't recall exactly, if you want to pursue this you should ask on the Devel list for confirmation. That would mean that I could relatively easily write my own sync code that could also read all those values and decide which page would need to be transferred while the file is locked. I don't think that it will be advisable (see below), but why duplicate what nbackup is already doing? The disadvantage would be that I would need to keep close control of these files as nbackup could not come to help to prevent incorrect ordering while merging... There are many other factors to consider: - backup/restore of source database using gbak -- this would reset all page version values - correctly detecting the database page size value -- which can be changed via gbak backup/restore. - the ODS (on disk structure) for database pages can change from version to version*. You would need to factor these changes in your own code. Does NBackup do any other magic than that? It ensures that a backup file is being applied to the correct target database -- it does not blindly restores backup files on top of database. Sean * A change to the header layout would be part of a major ODS update. Such updates have (historically) only occurred with major Firebird releases (though this may change in the future)
AW: [firebird-support] NBackup Levels
Hi I could simply create backups with incrementing levels, move the backup to the other server(s) and apply them there (that database is offline). However, I suspect there is a limit for the number of backup levels a database can have. You don't need a lot of backup levels to do what you want. For example, Do a level 0 backup once a month/year/whatever, then do a level 1 backup every weekend, a level 2 backup every day, and if required a level 3 backup every hour/whatever. For another type of service I already have a backup script that creates 5 levels of backups: Level 0 at the first Sunday of every quarter. Level 1 at the first Sunday every month. Level 2 at every Sunday. Level 3 every day. Level 4 every hour. But for this project I want more than hourly consistency. I am targeting 5 minuites or even less. That could be done using: Level 5 every 5 minutes. However in this case there might be lots of days with nearly no difference and then there may be some days with gigabytes of changes. Using an approach as above would mean to copy all these changes up to 23 times... I would really like to prevent that kind of extra traffic AND more important that delay in synchronization. Moreover, the servers hard drives may be rather slow and the database may grow up to 200 GB. (During operation there are relatively few reads and only some writes, the database is idle 99% of the time, so for operation the slow io system is not a problem.) Regards, Steffen [Non-text portions of this message have been removed]
AW: [firebird-support] NBackup Levels
Hi There is no formal limit, ... Is there a negative effect on too many levels? (Slowdown or database storage overhead?) Regards, Steffen
Re: [firebird-support] NBackup Levels
Steffen Heil (Mailinglisten) skriver: Hi I could simply create backups with incrementing levels, move the backup to the other server(s) and apply them there (that database is offline). However, I suspect there is a limit for the number of backup levels a database can have. You don't need a lot of backup levels to do what you want. For example, Do a level 0 backup once a month/year/whatever, then do a level 1 backup every weekend, a level 2 backup every day, and if required a level 3 backup every hour/whatever. For another type of service I already have a backup script that creates 5 levels of backups: Level 0 at the first Sunday of every quarter. Level 1 at the first Sunday every month. Level 2 at every Sunday. Level 3 every day. Level 4 every hour. But for this project I want more than hourly consistency. I am targeting 5 minuites or even less. That could be done using: Level 5 every 5 minutes. However in this case there might be lots of days with nearly no difference and then there may be some days with gigabytes of changes. Using an approach as above would mean to copy all these changes up to 23 times... I would really like to prevent that kind of extra traffic AND more important that delay in synchronization. Moreover, the servers hard drives may be rather slow and the database may grow up to 200 GB. (During operation there are relatively few reads and only some writes, the database is idle 99% of the time, so for operation the slow io system is not a problem.) I understand your problem. Would it be possible to measure the size of each level 5 backup, and if it's over a certain threshold use a higher level for the rest of that day to keep from copying the same large set of data over and over? In essence, you would create a script that backs up every five minutes, and if the last backup size was below a certains threshold use the same level as before, otherwise bump it up. Reset to level 3 each night. The benefit is that you won't create a large number of levels unless it's actually needed to reduce backup traffic. Regards, Kjell ++ Visit http://www.firebirdsql.org and click the Resources item on the main (top) menu. Try Knowledgebase and FAQ links ! Also search the knowledgebases at http://www.ibphoenix.com ++ Yahoo Groups Links * To visit your group on the web, go to: http://groups.yahoo.com/group/firebird-support/ * Your email settings: Individual Email | Traditional * To change settings online go to: http://groups.yahoo.com/group/firebird-support/join (Yahoo! ID required) * To change settings via email: firebird-support-dig...@yahoogroups.com firebird-support-fullfeatu...@yahoogroups.com * To unsubscribe from this group, send an email to: firebird-support-unsubscr...@yahoogroups.com * Your use of Yahoo Groups is subject to: https://info.yahoo.com/legal/us/yahoo/utos/terms/
RE: [firebird-support] NBackup Levels
I understand your problem. Would it be possible to measure the size of each level 5 backup, and if it's over a certain threshold use a higher level for the rest of that day to keep from copying the same large set of data over and over? In essence, you would create a script that backs up every five minutes, and if the last backup size was below a certains threshold use the same level as before, otherwise bump it up. Reset to level 3 each night. The benefit is that you won't create a large number of levels unless it's actually needed to reduce backup traffic. I thought similar to Kjell, yet opposite: If the size is over the threshold, just do an extra backup at a lower level, e.g. make a level 2 backup if the level 5 backup is above 1 Gb and a level 3 backup if the level 5 backup is above 100Mb or similar. I would expect it to be possible to mix this with the backup scheme that you've already described, but I've absolutely no knowledge of NBackup and may be completely wrong. HTH, Set
RE: [firebird-support] NBackup Levels
There is no formal limit, ... Is there a negative effect on too many levels? (Slowdown or database storage overhead?) No negative effect. The database does track the ID of the last backup at each level, so you would have 2 or 3 more entries that the typical usage, but the details are notional in size. Sean
Re: [firebird-support] NBackup Levels
Every operating system has some limit on environment strings like command line, that can be a problem on restore where the every level's filename should be supplied. Not sure if it's a theoretical or practical problem in Your case. I use 3 levels (0 - friday night, 1 - every other weekday night, 2 - 90 minutes). Regards Zsazsi 2014-05-07 8:43 GMT+02:00 Steffen Heil (Mailinglisten) li...@steffen-heil.de: Hi There is no formal limit, ... Is there a negative effect on too many levels? (Slowdown or database storage overhead?) Regards, Steffen ++ Visit http://www.firebirdsql.org and click the Resources item on the main (top) menu. Try Knowledgebase and FAQ links ! Also search the knowledgebases at http://www.ibphoenix.com ++ Yahoo Groups Links
Re: [firebird-support] NBackup Levels
Den 2014-05-05 22:27 skrev Steffen Heil (Mailinglisten) såhär: I know that nbackup keeps track of which blocks in the database changed and which did not so that backups contain only the blocks that changes since the last backup of the next lower level. I want to use that feature to do syncs between servers. I could simply create backups with incrementing levels, move the backup to the other server(s) and apply them there (that database is offline). However, I suspect there is a limit for the number of backup levels a database can have. You don't need a lot of backup levels to do what you want. For example, Do a level 0 backup once a month/year/whatever, then do a level 1 backup every weekend, a level 2 backup every day, and if required a level 3 backup every hour/whatever. Copy the backup files to the remote server, and use nbackup there, to restore the most current backup chain. Granted, the restore operations on the other server(s) will take a bit of time, but if those servers are offline, that would not really be a problem. In fact, I see no real point in doing those restores until you actually need to put the backup online. Just save the most recent chain of backup files, and you're ready to do a restore when the need arises. Another strategy would be to lock the database, take a copy, unlock it and then use external software to do the diff. However that would mean that the database would have to be copied very often and that a lot of extra space was needed. That would mean a lot of disc io. That could be reduced, if I locked the database, sync it using some software as rsync and then unlock it again. I would not need extra copies any more, but a) the database would need to be offline much longer (remote sync needs more time) and b) it would have to be done for every replica again. This is the approach I use for my 100+ Gbyte database on modest hardware (Windows server with 7200 rpm scsi traditional non-ssd harddisks). I use FastCopy (http://ipmsg.org/tools/fastcopy.html.en) to copy the database files while it's locked and then rsync the copy to our backup server. The FastCopy takes about 45-60 minutes and the rsync usually takes a couple of hours. There is NO downtime! The application is online when the database is locked and of course also during rsync. So here are my questions: a) How many levels of backups can a database have (nbackup, not gbak)? Sorry, I don't know. But I don't think you need more than 3-5 levels. b) How bad is it (in terms of performance) to lock a database for a longer period of time? I've seen no problems whatsoever with the scheme I described, in which the database is locked for about an hour. I don't see how/why you would need any more than that. c) Is there any way to get the information used by nbackup without actually creating a new backup level? (Just use this information to use external block copying?) Sorry, I don't know. d) Is there any other good way to create a live exact replica? Shadow file feature? Third paerty replication tools? Regards, Kjell
[firebird-support] NBackup Levels
Hi I know that nbackup keeps track of which blocks in the database changed and which did not so that backups contain only the blocks that changes since the last backup of the next lower level. I want to use that feature to do syncs between servers. I could simply create backups with incrementing levels, move the backup to the other server(s) and apply them there (that database is offline). However, I suspect there is a limit for the number of backup levels a database can have. Another strategy would be to lock the database, take a copy, unlock it and then use external software to do the diff. However that would mean that the database would have to be copied very often and that a lot of extra space was needed. That would mean a lot of disc io. That could be reduced, if I locked the database, sync it using some software as rsync and then unlock it again. I would not need extra copies any more, but a) the database would need to be offline much longer (remote sync needs more time) and b) it would have to be done for every replica again. So here are my questions: a) How many levels of backups can a database have (nbackup, not gbak)? b) How bad is it (in terms of performance) to lock a database for a longer period of time? c) Is there any way to get the information used by nbackup without actually creating a new backup level? (Just use this information to use external block copying?) d) Is there any other good way to create a live exact replica? Best regards, Steffen
