Hi *, sorry to the mailinglist for the cryptic mangled patches.
In my intial posting about 'Feature Suggestions for Cross-Platform-Compatibility' I wrote about some feature enhacements (I think) would benefit rdiff-backup while keeping full backward compatibility: http://www.backupcentral.com/phpBB2/two-way-mirrors-of-external-mailing-lists-3/rdiff-backup-23/feature-suggestions-for-cross-platform-compatibility-93227/ Now this functionality is implemented and (well-)tested locally on Windows XP NTFS and Linux EXT2+EXT3. During learning Python and reviewing the rdiff-backup code I fixed two small bugs in unrelated modules and cleaned up/updated the man-page. BUGS Fixed: - module eas_acls.py, Line: 224: comment position was erroneously calculated before, resulting in a (potential) nasty bug, truncating AccessControlLists-files - module fs_abilities.py, get_ctq_from_fsas(): WAS pot. bug before, because unconditionally only ';' was quoted, even if overridden with another character Here are the patches, first the modifications to the man/help-page to explain most of the reasons and added functionality: [code] --- rdiff-backup/rdiff-backup.1 2008-08-20 02:37:41.000000000 +0000 +++ rdiff-backup-mycode/rdiff-backup.1 2008-11-09 11:40:58.000000000 +0000 @@ -1,4 +1,4 @@ -.TH RDIFF-BACKUP 1 "JULY 2007" "Version 1.1.13" "User Manuals" \" -*- nroff -*- +.TH RDIFF-BACKUP 1 "OCTOBER 2008" "Version 1.2.2" "User Manuals" \" -*- nroff -*- .SH NAME rdiff-backup \- local/remote mirror and incremental backup .SH SYNOPSIS @@ -344,7 +344,7 @@ rdiff-backup-data directory. rdiff-backup will run slightly quicker and take up a bit less space. .TP -.BI \-\-no-hard-links +.B \-\-no-hard-links Don't replicate hard links on destination side. If many hard-linked files are present, this option can drastically decrease memory usage. This option is enabled by default if the backup source or restore @@ -369,6 +369,15 @@ .B \-\-override-chars-to-quote If the filesystem to which we are backing up is not case-sensitive, automatic 'quoting' of characters occurs. For example, a file 'Developer.doc' will be converted into ';068eveloper.doc'. To override this behavior, you need to specify this option. .TP +.BI "\-\-override-quote-chars-and-fsabilities-from-file " filename +This option lets you override ANY filesystem attributes and characters to quote, read from a config file. +Similar to above \-\-override-chars-to-quote, but offers advanced functionality, you can specify INCLUDES, EXCLUDES, +each also as a range of values (e.g. \\0x00-\\0x30) and supports decimal/hexadecimal and octal values. +Use this option to create a single cross-platform-compatibility file, when you use several OS with different filesystems, +then all filenames are converted properly to be +.B completely cross-platform +compatible. See below for a file-format description and example. +.TP .B \-\-preserve-numerical-ids If set, rdiff-backup will preserve uids/gids instead of trying to preserve unames and gnames. See the @@ -485,6 +494,28 @@ in the following host::filename argument(s). The filename section will be ignored. .TP +.B \-\-use-compatible-timestamps +When \-\-use-compatible-timestamps is enabled, the above timestamp +is created as following: "2008-10-13T04-09-38-07-00" (instead +of default: "2008-10-13T04:09:38-07:00"). This format is allowed +also on Windows filesystems where colons (":") are disallowed in +filenames! If you want to locally backup a Unix path and later use +.BR rsync (1) +to backup to Microsoft Windows system, use this option! +This option is fail-safe, you can continue your 'traditional' timestamped backups, change to this option +later and even revert to the original, non-windows-compatible timestamp or even intermix different timestamp formats. +.TP +.B \-\-use-utc +When \-\-use-utc is enabled, the timestamp is always stored as UTC, +indicated by appended "Z" (for Zulu) instead of timezone +offset, thus becomes "2001-07-15T04-09-38Z". Using \-\-use-utc +is recommended when 1) a remote backup is performed, 2) source +and destination for backup/restore are in different timezones, 3) +you want to keep yourself from 'daylight saving' errors, 4) all +timestamped filenames are 5 characters shorter. +This option is fail-safe, you can continue your 'traditional' timestamped backups, change to this option +later and even revert to the original, non-windows-compatible timestamp or even intermix different timestamp formats. +.TP .BI "\-\-user-mapping-file " filename Map user names and ids according to the user mapping file .IR filename . @@ -577,6 +608,28 @@ http://www.w3.org/TR/NOTE-datetime. Basically they look like "2001-07-15T04:09:38-07:00", which means what it looks like. The "-07:00" section means the time zone is 7 hours behind UTC. +When +.B \-\-use-compatible-timestamps +is enabled, the above timestamp is created as following: "2001-07-15T04-09-38-07-00". This format +is allowed also on Windows filesystems where ":" colons are disallowed in filenames! +This option is fail-safe, you can continue your 'traditional' timestamped backups, change to this option +later and even revert to the original, non-windows-compatible timestamp or even intermix any. +When additionally +.B \-\-use-utc +is enabled, then the timestamp is +always used as UTC, indicated by appended "Z" (for Zulu) instead +of timezone offset, thus becomes "2001-07-15T04-09-38Z". See above option details for benefits. +This option is fail-safe, you can continue your 'traditional' timestamped backups, change to this option +later and even revert to the original, non-windows-compatible timestamp or even intermix them. + +Using +.B \-\-use-utc +is recommended when 1) a remote backup is performed, 2) source +and destination for backup/restore are in different timezones, 3) +you want to keep yourself from 'daylight saving' problems, 4) all +timestamped filenames are 5 characters shorter. +This option is fail-safe, you can continue your 'traditional' timestamped backups including timezone-info, +then use this option at any time, even intermixed. .PP Secondly, the .BI \-r , " \-\-restore-as-of" ", and " \-\-remove-older-than @@ -697,6 +750,7 @@ .BR \-\-include-globbing-filelist , .BR \-\-include-globbing-filelist-stdin , .BR \-\-include-filelist-stdin , +.BR \-\-override-quote-chars-and-fsabilities-from-file, and .BR \-\-include-regexp . Each file selection condition either matches or doesn't match a given @@ -920,6 +974,53 @@ the same as specifying "\-\-include dir/foo \-\-include dir/bar \-\-exclude **" on the command line. +.B "\-\---override-quote-chars-and-fsabilities-from-file override_cfg.txt" +.RE This option lets you override ANY filesystem attributes and characters to quote, read from a config file. +Similar to above \-\-override-chars-to-quote, but offers advanced functionality, you can specify INCLUDES, EXCLUDES, +each also as a range of values (\\0x00-\\0x30) and supports decimal/hexadecimal and octal values. +Use this option to create a single cross-platform-compatibility file, when you use several OS with different filesystems, +then all filenames are converted properly to be +.B completely cross-platform +compatible. +.RE +.B File-Format: +.RE ######################################################## + # Format description: escaped ('\\'-prefixed) numbers, + # supported are octal/hexadecimal/decimal values (even + # inter-mixed), separated by semicolon ';' + # or specifying a character-range with a hyphen ('-') + # (e.g. \\0x00-\\0x1F). For help in conversion from characters + # to decimal/hexadecimal/octal values, see URLs: + # http://www.asciitable.com/ + # http://www.asciitable.de/tabelle.html + ########################################################### + INCLUDE:\\0x00-\\0x1F;\\0x22;\\0x2A;\\0x2F;\\0x3A;\\0x3C;\\0x3E;\ +\\0x3F;\\0x5C;\\0x7C;\\0x7F;\\0x3B + # Description of above INCLUDE: Quote non-printable char-range + # decimal 0-31, and also quote: ", *, /, :, <, >, ?, \, |, + # and 127 (decimal; hex: 7F) (DEL) and ';' quotation + # char itself. This is required for Windows-compatibility. + EXCLUDE: +.RE + ############################################################ + # FS-ABILITIES: nameformat as in Python source-code, with + # '+' and '-' prefix to indicate enabling, resp. disabling a + # feature, separated by ';' from following other options. + # If you do not want an auto-detected filesystem-ability to + # be overwritten, simply remove the option from below list. + # + ## TODO: interaction and precedence from commandline with: + # --exclude-special-files (device-files;fifos;sockets;symbolic-links); + # --exclude-other-filesystems +.RE + ########### + FS-ABILITIES:+case_sensitive;+eas;+acls;+win_acls; \ + +escape_dos_devices;-resource_forks;-carbonfile;-hardlinks; \ + -fsync_dirs;-change_ownership;-high_perms;-symlink_perms; + ############################################################ +.RE + +.BR Finally, the .B \-\-include-regexp and [/code] [code] diff U3 eas_acls.py eas_acls.py --- eas_acls.py Sat Sep 27 01:08:31 2008 +++ eas_acls.py Sat Nov 01 11:15:45 2008 @@ -224,7 +224,7 @@ """Set self.entry_list and self.default_entry_list from text""" self.entry_list, self.default_entry_list = [], [] for line in text.split('\n'): - comment_pos = text.find('#') + comment_pos = line.find('#') ## AFAICT was BUG before: comment_pos = text.find('#') if comment_pos >= 0: line = line[:comment_pos] line = line.strip() if not line: continue diff U3 FilenameMapping.py FilenameMapping.py --- FilenameMapping.py Sat Jan 05 19:43:13 2008 +++ FilenameMapping.py Thu Oct 23 14:04:21 2008 @@ -65,6 +65,7 @@ def init_quoting_regexps(): """Compile quoting regular expressions""" global chars_to_quote_regexp, unquoting_regexp + assert chars_to_quote and type(chars_to_quote) is types.StringType, \ "Chars to quote: '%s'" % (chars_to_quote,) try: diff U3 fs_abilities.py fs_abilities.py --- fs_abilities.py Wed Oct 08 00:45:42 2008 +++ fs_abilities.py Tue Nov 04 13:11:27 2008 @@ -27,7 +27,7 @@ """ -import errno, os +import errno, os, re, string import Globals, log, TempFile, selection, robust, SetConnections, \ static, FilenameMapping, win_acls @@ -73,7 +73,7 @@ else: assert boolean == 0 val_text = 'Off' - addline(desc, val_text) + addline(desc, val_text) def get_title_line(): """Add the first line, mostly for decoration""" @@ -164,6 +164,9 @@ subdir.delete() return self + +# def show_active_fsabilities(self, ): + def set_ownership(self, testdir): """Set self.ownership to true iff testdir's ownership can be changed""" @@ -556,6 +559,11 @@ % (subdir.path), 4) self.escape_dos_devices = 1 + +## OM: Todo: Move function 'update_fs_abilities' herein? +# def update_fs_abilities(self, key, value): +###### END OF update_fs_abilities() + def get_readonly_fsa(desc_string, rp): """Return an fsa with given description_string @@ -633,6 +641,9 @@ class BackupSetGlobals(SetGlobals): """Functions for setting fsa related globals for backup session""" + +#---------------------------------------------------------------------- + def update_triple(self, src_support, dest_support, attr_triple): """Many of the settings have a common form we can handle here""" active_attr, write_attr, conn_attr = attr_triple @@ -645,6 +656,8 @@ SetConnections.UpdateGlobal(write_attr, 1) self.out_conn.Globals.set_local(conn_attr, 1) +#---------------------------------------------------------------------- + def set_must_escape_dos_devices(self, rbdir): """If local edd or src edd, then must escape """ try: @@ -657,21 +670,29 @@ log.Log("Backup: must_escape_dos_devices = %d" % \ (self.src_fsa.escape_dos_devices or local_edd), 4) +#---------------------------------------------------------------------- + def set_chars_to_quote(self, rbdir, force): - """Set chars_to_quote setting for backup session + """Set chars_to_quote setting for backup session. Unlike the other options, + the chars_to_quote setting also depends on the current settings in the + rdiff-backup-data directory, not just the current fs features. + """ - Unlike the other options, the chars_to_quote setting also - depends on the current settings in the rdiff-backup-data - directory, not just the current fs features. + ctq = [] - """ - (ctq, update) = self.compare_ctq_file(rbdir, - self.get_ctq_from_fsas(), force) + if Globals.is_not_None('override_chars_to_quote'): + wanted_ctq = self.get_ctq_and_fsabilities_from_file( Globals.get('path_octqff') ) + else: wanted_ctq = self.get_ctq_from_fsas() + +# WAS: (ctq, update) = self.compare_ctq_file(rbdir, self.get_ctq_from_fsas(), force) + (ctq, update) = self.compare_ctq_file(rbdir, wanted_ctq, force) SetConnections.UpdateGlobal('chars_to_quote', ctq) if Globals.chars_to_quote: FilenameMapping.set_init_quote_vals() return update +#---------------------------------------------------------------------- + def get_ctq_from_fsas(self): """Determine chars_to_quote just from filesystems, no ctq file""" ctq = [] @@ -684,12 +705,276 @@ if self.dest_fsa.win_reserved_filenames: if self.dest_fsa.extended_filenames: ctq.append('\000-\037') # Quote 0 - 31 - # Quote ", *, /, :, <, >, ?, \, |, and 127 (DEL) + # Quote ", *, /, :, <, >, ?, \, |, and 127 (decimal) (DEL) ctq.append('\"*/:<>?\\\\|\177') - if ctq: ctq.append(';') # Quote quoting char if quoting anything +# OM: WAS pot. bug before, because unconditionally only ';' was quoted, even if overridden with another character +# if ctq: ctq.append(';') # Quote quoting char if quoting anything + if ctq: ctq.append(Globals.get('quoting_char')) # Quote defined quoting-char if quoting anything return "".join(ctq) +######################################################################################### +## OM-test +## TODO: Should be a combination of filesystem-detected requirements ?!? +## Safe-version: use both detected by FS-capabilities AND from file => most sensible +## +## TODO: enhance also overriding filesystem-detected capabilities; e.g. +## use case-sensitivity even on case-insensitive FS, no hardlinking even if supported, etc. +## impl. here also! +Option, -Option +## +## TODO: later impl. option, if filesystem (SRC, DEST or intermediary!) is case-insensitive, +## do simple 'sliding window' test for each directory to find any potential collision AND escape +## only these. While writing this, a '--dry-run' option performing this action would be great, +## add value to this software and positively advocates on file-naming! +## TODO: find any existing project/code to do this! There are only some regexp required for this! + + + def get_ctq_and_fsabilities_from_file(self, filepath_ctq): + """Determine chars_to_quote just from file, overriding all other settings""" + + ctq, ctq_includes, ctq_excludes, custom_fs_abilities = [], [], [], [] + + try: + file_in = open(filepath_ctq, 'r') + except IOError: + log.Log("Error: Could not open 'override-quoted-chars-and-fsabilities'-file '%s'!" % (filepath_ctq), 1) + else: + for line in file_in.readlines(): + line = line.strip() + if not line: continue ## skip empty lines +# log.Log("DEBUG: get_ctq_from_file(): line read: '%s'\n" % line, 6) + if re.match("^#", line): ## skip the format description/examples + commented out lines +# log.Log("DEBUG: get_ctq_from_file(): comment skipped\n", 6) + continue + elif re.match("^FS-ABILITIES:(.*?)$", line): + m = re.match("^FS-ABILITIES:(.*?)$", line) ## How to include the assignment 'm = ' right in above line? + custom_fs_abilities = m.group(1).strip() + log.Log("DEBUG: get_ctq_from_file(): FS-ABILITIES: Filesystem overrides: '%s'\n" % ( custom_fs_abilities ), 6) + self.override_fs_abilities(custom_fs_abilities) + else: + m = re.match("^(IN|EX)CLUDE:(.*?)$", line) + if m.group(1) == 'IN': ctq_includes = self.decode_characters(m.group(2).strip()) + elif m.group(1) == 'EX': ctq_excludes = self.decode_characters(m.group(2).strip()) + log.Log("DEBUG: get_ctq_from_file(): %sCLUDE contents: '%s'\n" % (m.group(1), m.group(2)), 6) + file_in.close() + + ctq_decimals = self.diff_ctqff(ctq_includes, ctq_excludes) + result = self.collapse_ctqff(ctq_decimals) + +# log.Log("Number of includes: '%d', of excludes '%d'" % ( len(ctq_includes), len(ctq_excludes) ), 6) +# print "DEBUG: final DIFF-array of decimal character values (range only 0-255): ", ctq_decimals + log.Log("DEBUG: Final list of chars to quote: '%s'\n" % result, 6) + + return result + +#---------------------------------------------------------------------- + + def decode_characters(self, encoded_string): + """ + Reads characters in multiple formats (decimal/hexadecimal/octal), expands + any ranges provided and converts/unifies all to decimals for later easy array-processing. + Attention: only allows ASCII (0-255), no Unicode-points at this time! + + For help in conversion from/to decimal/hexadecimal/octal values, see URLs: + http://www.asciitable.com/ + http://www.asciitable.de/tabelle.html + """ + + final_chars_to_quote_list = [] + + for item in encoded_string.split(';'): + if not item: continue +# print "DEBUG: part '%s'" % ( item ) + + sublist = item.split('-') + + if (len(sublist) != 2): ## is a number-representation of a single character only + + replaced_number = string.replace( sublist[0], '\\', '') + ## Get the number from whatever Base-/Radix was specified, either decimal/hexadecimal or octal + char_number = int(replaced_number, 0) ## read + convert to decimal/base10 + final_chars_to_quote_list.append( char_number ) +## For debugging: +# if (char_number <= 255): char = chr(char_number) +# else: char = unichr(char_number) +# print "DEBUG: Single char appended: decimal '%d' as char '%s'" % ( char_number, char ) + + else: ## a range of characters specified, represented as numbers + + replaced_number0 = string.replace( sublist[0], '\\', '') + replaced_number1 = string.replace( sublist[1], '\\', '') + char_number0 = int(replaced_number0, 0) + char_number1 = int(replaced_number1, 0) + final_chars_to_quote_list.extend(range(char_number0, char_number1 + 1)) ## '+1' because range() excludes second parameter + +## For debugging: +# if (char_number0 <= 255): char0 = chr(char_number0) +# else: char0 = unichr(char_number0) +# if (char_number1 <= 255): char1 = chr(char_number1) +# else: char1 = unichr(char_number1) +# print "DEBUG: Char range appended: decimal '%d' - '%d' => '%s' - '%s'" % (char_number0, char_number1, char0, char1) + +# print "DEBUG: char-array of decimal values: ", final_chars_to_quote_list + return final_chars_to_quote_list + +#---------------------------------------------------------------------- + + def diff_ctqff(self, list_includes, list_excludes): + """ Actually find all decimal character values to quote, with any excluded values removed.""" + list_includes.sort() + list_excludes.sort() + final_list_ctqff = [item for item in list_includes if item not in list_excludes] + return final_list_ctqff + +#---------------------------------------------------------------------- + + def collapse_ctqff(self, listing): + """ + Collapses/compresses **more than two** consecutive integer numbers into a range: e.g. 0 1 2 3 4 8 10 11 => 0-4,8,10,11 + Problem: This is important or else a bug is thrown when quoted chars in range 0-31 decimal are given (which is required for Windows FSs) + of type: "TypeError: 'NoneType' object is not iterable" + Solution: The char-range 0-31 decimal must be written to 'quote'-file with '-' as a range, as in default quote-file for Windows FSs. + """ + + final_string, str_list, quoting_char_included = '', [], 0 + quoting_integer_number = ord(Globals.get('quoting_char')) ## If the quoting char itself is forgotten from the list, add it! + +## OM: Remark: this works for now, but 'feels' not ideal ... please improve. + accumulator, start, stop, iter = -1, -1, -1, -1 + + for item in listing: + iter = iter + 1 +# print "DEBUG: iterating item '%s' - number '%d'" % (item, iter) + + if ((quoting_char_included == 0) and (item == quoting_integer_number)): + quoting_char_included = 1 + + if iter == 0: start = item; accumulator = 0; continue + elif (listing[iter] == (listing[iter - 1] + 1)): ## consecutive integers, store +# print "DEBUG: consecutive numbers found '%r'-'%r'\n" % (listing[iter]-1, listing[iter]) + stop = item; accumulator = accumulator + 1 + elif accumulator >= 2: ## more than two consecutive integers, store +# print "DEBUG: accumulator '%s'\n" % (chr(start) + '-' + chr(stop)) + str_list.append(chr(start) + '-' + chr(stop)); + accumulator = 0; start = item + elif accumulator >= 1: ## only two consecutive integers, do not store as range, but simple append both +# print "DEBUG: accumulator '%s'\n" % (chr(start) + chr(stop)) + str_list.append(chr(start) + chr(stop)); + accumulator = 0; start = item + else: + accumulator = 0; appendix = chr(start) + str_list.append(appendix); start = item +# print "DEBUG: ELSE-clause, solitaire char only '%s' appended\n" % appendix + ## END for-loop + str_list.append(chr(start)) ## Last character needs to be appended, too! + + # Add defined quoting-char to existing list if not already included + if (str_list and (quoting_char_included == 0)): str_list.append(Globals.get('quoting_char')) + + return "".join(str_list) + +#---------------------------------------------------------------------- + + def override_fs_abilities(self, string_fs_abilities): + """ Attention: Only use when you know what you are doing, since this might break backups. """ + + """ +#OM: FS-ABILITIES: nameformat as in Python-code +#FS-ABILITIES:+case_sensitivity;+eas;+acls;+win_acls;+escape_dos_devices;-resource_forks;-carbonfile;-hardlinks;-fsync_dirs;-change_ownership;-high_perms;-symlink_perms; +## How about: --exclude-special-files (--exclude-device-files; --exclude-fifos;--exclude-sockets;--exclude-symbolic-links); --exclude-other-filesystems + """ + + fs_abilities_enabled, fs_abilities_disabled = [], [] + + for item in string_fs_abilities.split(';'): + if not item: continue +# print "DEBUG: item '%s'" % ( item ) + m = re.match("^(\+|-)(.*?)$", item) + if m.group(1) == '+': fs_abilities_enabled.append(m.group(2)) + elif m.group(1) == '-': fs_abilities_disabled.append(m.group(2)) + else: log.Log("Error! There's a '+' or '-' prefix missing from item '%s' in fs_abilities override file, fix this first!" % (m.group(2)), 6) + ## END FOR-LOOP + +# print "DEBUG: fs_abilities ENABLED: ", fs_abilities_enabled, " - DISABLED: ", fs_abilities_disabled + + for item in fs_abilities_enabled: self.update_fs_abilities(item, 1) + for item in fs_abilities_disabled: self.update_fs_abilities(item, 0) + + """ + TODO: interaction with the corresponding commandline options? +Globals.set("never_drop_acls", 1) ; Globals.set("carbonfile_active", 0) ; Globals.set("acls_active", 0) +Globals.set("win_acls_active", 0) ; Globals.set('preserve_hardlinks', 0) ; Globals.set('resource_forks_active', 0) + """ + +#---------------------------------------------------------------------- + + def info_updated_fsa(self, key, value): + print "DEBUG: update_fs_abilities(src,dest): '%s' was changed to '%s'." % (key, value) + Globals.set('override_fsabilities', 1) + +#---------------------------------------------------------------------- + + def update_fs_abilities(self, key, value): + +# print "DEBUG: update_fs_abilities(): processing '%s'" % key + +## OM: AIM: Wanted to create/lookup variable from string/key: +# I tried below to no avail with and without 'self.' prefix: +# src_key = 'self.src_fsa.' + key +# src_key = vars()['self.src_fsa.' + key] +# src_key = locals()['self.src_fsa.' + key] +# src_key = globals()['self.src_fsa.' + key] +# eval(src_key = 'self.src_fsa.' + key) +# exec( "src_key = 'self.src_fsa.' + key") +# +## OM: This is an UGLY KLUDGE/workaround for now. AIM: Wanted to create/lookup variable from string/key: +# src_key = "self.src_fsa." + key; print "Src-key: '%s'\n" % src_key + + fsa_attribute_changed = 1 + + if ( (key.find( 'extended_filenames' ) != -1) and ( self.src_fsa.extended_filenames != value ) ): + self.src_fsa.extended_filenames = self.dest_fsa.extended_filenames = value + elif ( (key.find( 'win_reserved_filenames' ) != -1) and ( self.src_fsa.win_reserved_filenames != value) ): + self.src_fsa.win_reserved_filenames = self.dest_fsa.win_reserved_filenames = value + elif ( (key.find( 'case_sensitive' ) != -1) and (self.src_fsa.case_sensitive != value) ): + self.src_fsa.case_sensitive = self.dest_fsa.case_sensitive = value + elif ( (key.find( 'ownership' ) != -1) and (self.src_fsa.ownership != value) ): + self.src_fsa.ownership = self.dest_fsa.ownership = value + elif ( (key.find( 'eas' ) != -1) and (self.src_fsa.eas != value) ): + self.src_fsa.eas = self.dest_fsa.eas = value + elif ( (key.find( 'win_acls' ) != -1) and (self.src_fsa.win_acls != value) ): + self.src_fsa.win_acls = self.dest_fsa.win_acls = value + elif ( (key.find( 'acls' ) != -1) and (self.src_fsa.acls != value) ): ### IMPORTANT! 'acls' check must be after 'win_acls' check! + self.src_fsa.acls = self.dest_fsa.acls = value + elif ( (key.find( 'hardlinks' ) != -1) and (self.src_fsa.hardlinks != value) ): + self.src_fsa.hardlinks = self.dest_fsa.hardlinks = value + elif ( (key.find( 'fsync_dirs' ) != -1) and (self.src_fsa.fsync_dirs != value) ): + self.src_fsa.fsync_dirs = value ## Not needed: self.dest_fsa.fsync_dirs + elif ( (key.find( 'dir_inc_perms' ) != -1) and (self.src_fsa.dir_inc_perms != value) ): + self.src_fsa.dir_inc_perms = self.dest_fsa.dir_inc_perms = value + elif ( (key.find( 'resource_forks' ) != -1) and (self.src_fsa.resource_forks != value) ): + self.src_fsa.resource_forks = self.dest_fsa.resource_forks = value + elif ( (key.find( 'carbonfile' ) != -1) and (self.src_fsa.carbonfile != value) ): + self.src_fsa.carbonfile = self.dest_fsa.carbonfile = value + elif ( (key.find( 'high_perms' ) != -1) and (self.src_fsa.high_perms != value) ): + self.src_fsa.high_perms = self.dest_fsa.high_perms = value + elif ( (key.find( 'escape_dos_devices' ) != -1) and (self.src_fsa.escape_dos_devices != value) ): + self.src_fsa.escape_dos_devices_dirs = self.dest_fsa.escape_dos_devices = value + elif ( (key.find( 'symlink_perms' ) != -1) and (self.src_fsa.symlink_perms != value) ): + self.src_fsa.symlink_perms = self.dest_fsa.symlink_perms = value + else: + fsa_attribute_changed = 0 + print "DEBUG: update_fs_abilities(): Option '%s' either not changed or not recognized - skipped!" % key + + if (fsa_attribute_changed != 0): self.info_updated_fsa(key, value) + +## OM: END of UGLY KLUDGE/workaround. + +#---------------------------------------------------------------------- + +######################################################################################### +## END OM-test + def compare_ctq_file(self, rbdir, suggested_ctq, force): """Compare ctq file with suggested result, return actual ctq""" ctq_rp = rbdir.append("chars_to_quote") @@ -733,9 +1018,11 @@ repository from the old quoting chars to the new ones.""" % (suggested_ctq, actual_ctq, ctq_rp.path)) +#---------------------------------------------------------------------- class RestoreSetGlobals(SetGlobals): """Functions for setting fsa-related globals for restore session""" + def update_triple(self, src_support, dest_support, attr_triple): """Update global settings for feature based on fsa results @@ -754,6 +1041,8 @@ self.out_conn.Globals.set_local(write_attr, 1) if src_support: self.in_conn.Globals.set_local(conn_attr, 1) +#---------------------------------------------------------------------- + def set_must_escape_dos_devices(self, rbdir): """If local edd or src edd, then must escape """ if getattr(self, "src_fsa", None) is not None: @@ -769,6 +1058,8 @@ log.Log("Restore: must_escape_dos_devices = %d" % \ (src_edd or local_edd), 4) +#---------------------------------------------------------------------- + def set_chars_to_quote(self, rbdir): """Set chars_to_quote from rdiff-backup-data dir""" if Globals.chars_to_quote is not None: return # already overridden @@ -781,9 +1072,11 @@ "assuming no quoting in backup repository.", 2) SetConnections.UpdateGlobal("chars_to_quote", "") +#---------------------------------------------------------------------- class SingleSetGlobals(RestoreSetGlobals): """For setting globals when dealing only with one filesystem""" + def __init__(self, conn, fsa): self.conn = conn self.dest_fsa = fsa @@ -816,6 +1109,7 @@ self.update_triple(self.dest_fsa.carbonfile, ('carbonfile_active', 'carbonfile_write', 'carbonfile_conn')) +#---------------------------------------------------------------------- def backup_set_globals(rpin, force): """Given rps for source filesystem and repository, set fsa globals @@ -848,6 +1142,10 @@ if update_quoting and force: FilenameMapping.update_quoting(Globals.rbdir) + if Globals.is_not_None('override_fsabilities'): + print "Changed Filesystem-Attributes:\nSRC-FSA:\n", src_fsa, "\nDEST-FSA:\n", dest_fsa + +#---------------------------------------------------------------------- def restore_set_globals(rpout): """Set fsa related globals for restore session, given in/out rps""" @@ -873,6 +1171,8 @@ rsg.set_escape_dos_devices() rsg.set_must_escape_dos_devices(Globals.rbdir) +#---------------------------------------------------------------------- + def single_set_globals(rp, read_only = None): """Set fsa related globals for operation on single filesystem""" if read_only: @@ -894,3 +1194,4 @@ ssg.set_escape_dos_devices() ssg.set_must_escape_dos_devices(Globals.rbdir) +#---------------------------------------------------------------------- diff U3 Globals.py Globals.py --- Globals.py Wed Jul 02 19:03:23 2008 +++ Globals.py Tue Nov 04 12:51:56 2008 @@ -23,12 +23,15 @@ # The current version of rdiff-backup -version = "$version" +version = "$version" # If this is set, use this value in seconds as the current time # instead of reading it from the clock. current_time = None +# Stores the local timezone for later UTC-conversion +local_timezone = None + # This determines how many bytes to read at a time when copying blocksize = 131072 @@ -158,6 +161,27 @@ # should be escaped (see FilenameMapping for more info). chars_to_quote = None quoting_char = ';' + +############################################################################## +## OM-test +## If set, the timestamps use the following format: "2008-09-01T04-49-04-07-00" +## (instead of "2008-09-01T04:49:04-07:00"). This creates truly cross-platform +## timestamps, e.g. required for transferring rdiff-backup archive to Windows filesystems. +use_compatible_timestamps = None + +## if set, uses the characters/ranges for overriding local + evg. remote filesystem attributes +## detected. Useful when you need to transfer a locally done backup later to another (e.g. Windows) filesystem. +## Specify in this file either the characters you want to replace or their octal values + ranges (e.g. '\000-\030'). +override_chars_to_quote = None +path_octqff = None +override_fsabilities = None +## TODO: merge with above chars_to_quote var?!? + +## If set, use UTC as base for all file-access/modification calculations. Removes the explicit +## timezone-offset, and appends 'Z' for Zulu to the timestamp, resulting in "2008-09-01T04-49-04-Z" +use_utc = None +## END OM-test +############################################################################## # If true, emit output intended to be easily readable by a # computer. False means output is intended for humans. diff U3 Main.py Main.py --- Main.py Sun Oct 12 16:46:00 2008 +++ Main.py Sun Nov 02 14:34:31 2008 @@ -85,8 +85,11 @@ "remove-older-than=", "restore-as-of=", "restrict=", "restrict-read-only=", "restrict-update-only=", "server", "ssh-no-compression", "tempdir=", "terminal-verbosity=", - "test-server", "user-mapping-file=", "verbosity=", "verify", - "verify-at-time=", "version"]) + "test-server", + ## OM-test + "use-utc", "use-compatible-timestamps", "override-quote-chars-and-fsabilities-from-file=", + ## END + "user-mapping-file=", "verbosity=", "verify", "verify-at-time=", "version"]) except getopt.error, e: commandline_error("Bad commandline options: " + str(e)) @@ -146,6 +149,25 @@ select_opts.append(("--include-globbing-filelist", "standard input")) select_files.append(sys.stdin) + + ## OM-test + ## Advantages: --use-utc: local/remote systems in different timezones do not cause problems + elif opt == "--override-quote-chars-and-fsabilities-from-file": + p_octqff = arg + Globals.set_integer('override_chars_to_quote', 1) + Globals.set('path_octqff', p_octqff) +# print "DEBUG: override_chars_to_quote-from-file '%s'" % p_octqff + +# BackupSetGlobals.get_ctq_from_file( Globals.get('path_octqff') ) +# ## was: fs_abilities.get_ctq_from_file() +# REM: required also? Globals.set('chars_to_quote', arg) + + elif opt == "--use-utc": + Globals.set("use_utc", 1) + elif opt == "--use-compatible-timestamps": + Globals.set("use_compatible_timestamps", 1) + ## END OM-test + elif opt == "--include-regexp": select_opts.append((opt, arg)) elif opt == "--list-at-time": restore_timestr, action = arg, "list-at-time" @@ -240,8 +262,7 @@ else: action = "backup" def commandline_error(message): - Log.FatalError(message + "\nSee the rdiff-backup manual page for " - "more information.") + Log.FatalError(message + "\nSee the rdiff-backup manual page for more information.") def misc_setup(rps): """Set default change ownership flag, umask, relay regexps""" @@ -476,6 +497,9 @@ Log.open_logfile(Globals.rbdir.append("backup.log")) checkdest_if_necessary(rpout) prevtime = backup_get_mirrortime() + + Log("DEBUG: prevtime is %s" % prevtime, 6) + if prevtime >= Time.curtime: Log.FatalError( """Time of Last backup is not in the past. This is probably caused by running two backups in less than a second. Wait a second a try again.""") diff U3 metadata.py metadata.py --- metadata.py Sat Sep 27 01:17:24 2008 +++ metadata.py Tue Oct 21 10:08:28 2008 @@ -542,10 +542,13 @@ def _writer_helper(self, prefix, flatfileclass, typestr, time): """Used in the get_xx_writer functions, returns a writer class""" + if time is None: timestr = Time.curtimestr else: timestr = Time.timetostring(time) filename = '%s.%s.%s' % (prefix, timestr, typestr) rp = Globals.rbdir.append(filename) + + log.Log("DEBUG: created filename '%s'\n" % filename, 6) assert not rp.lstat(), "File %s already exists!" % (rp.path,) assert rp.isincfile() return flatfileclass(rp, 'w', callback = self.add_incrp) diff U3 rpath.py rpath.py --- rpath.py Sun Oct 12 16:46:00 2008 +++ rpath.py Tue Oct 21 14:09:51 2008 @@ -352,10 +352,14 @@ assert rpath.conn is Globals.local_connection return open(rpath.path, "rb") +## This fn is erroneous? No! This returns the number of tokens!! def get_incfile_info(basename): """Returns None or tuple of (is_compressed, timestr, type, and basename)""" dotsplit = basename.split(".") + +# log.Log("DEBUG: get_incfile_info(%s): dotsplit: '%d'" % (basename, len(dotsplit) ), 6) + if dotsplit[-1] == "gz": compressed = 1 if len(dotsplit) < 4: return None @@ -364,7 +368,13 @@ compressed = None if len(dotsplit) < 3: return None timestring, ext = dotsplit[-2:] + + result_time = Time.stringtotime(timestring) +# log.Log("DEBUG: timestring: '%s' - ext '%s' - resulting timestring '%s'" % (timestring, ext, result_time ), 6) + if Time.stringtotime(timestring) is None: return None + + if not (ext == "snapshot" or ext == "dir" or ext == "missing" or ext == "diff" or ext == "data"): return None @@ -1181,12 +1191,15 @@ def isincfile(self): """Return true if path looks like an increment file - Also sets various inc information used by the *inc* functions. - """ + +# log.Log("DEBUG: inside isincfile() beginning", 9) + if self.index: basename = self.index[-1] else: basename = self.base + +# log.Log("DEBUG: basename is '%s'" % basename, 6) inc_info = get_incfile_info(basename) diff U3 Time.py Time.py --- Time.py Thu Jan 25 04:09:16 2007 +++ Time.py Tue Nov 04 12:50:55 2008 @@ -20,8 +20,9 @@ """Provide time related exceptions and functions""" import time, types, re, sys, calendar +import os ## For timezone-change import Globals - +import log class TimeException(Exception): pass @@ -60,11 +61,40 @@ global prevtime, prevtimestr prevtime, prevtimestr = timeinseconds, timestr + +## OM: currently testing which version is more cross-platform compatible: either this fn. or time.gmtime()-call +def set_zimezone_to_utc(reset_to_localtime_again=False): + """Alternative way to set timezone to UTC and reset it again to local timezone for all timestamps""" + if reset_to_localtime_again: + os.environ['TZ'] = Globals.get(local_timezone) ## os.environ['TZ']='Europe/Berlin' + else: + Globals.set('local_timezone', time.tzname) + os.environ['TZ']='' ### is UTC now + + time.tzset() + log.Log("DEBUG: timezone used now '%s' - before was: '%s' " % (time.tzname, Globals.get(local_timezone) ), 7) + + def timetostring(timeinseconds): """Return w3 datetime compliant listing of timeinseconds""" - s = time.strftime("%Y-%m-%dT%H:%M:%S", time.localtime(timeinseconds)) + """OR else use the combatible version e.g. for Windows""" + + if Globals.is_not_None('use_compatible_timestamps'): time_formatstring = "%Y-%m-%dT%H-%M-%S" + else: time_formatstring = "%Y-%m-%dT%H:%M:%S" + + debug_utc_used = None + if Globals.is_not_None('use_utc'): + seconds = time.gmtime(timeinseconds) ## opposite direction: seconds = calendar.timegm(timeinseconds) + debug_utc_used = 1 + else: + seconds = time.localtime(timeinseconds) + + s = time.strftime(time_formatstring, seconds) +# log.Log("DEBUG: Time.py: time_string is '%s' (UTC: '%s')" % (s, debug_utc_used), 8) + return s + gettzd(timeinseconds) + def stringtotime(timestring): """Return time in seconds from w3 timestring @@ -72,10 +102,19 @@ like a w3 datetime string, return None. """ + +## OM: AIM: The processing of separators should be fail-safe, even mixed separators +## should be properly recognized and handled! + + regexp = re.compile( '[-:_]' ) ## WAS: "-" +# log.Log("DEBUG: timestring to process is '%s'" % (timestring), 9 ) + try: date, daytime = timestring[:19].split("T") year, month, day = map(int, date.split("-")) - hour, minute, second = map(int, daytime.split(":")) + +## OM: re.split() is required, using 'string'.split() is NOT possible with regexp-char group. + hour, minute, second = map(int, re.split(regexp, daytime) ) assert 1900 < year < 2100, year assert 1 <= month <= 12 assert 1 <= day <= 31 @@ -144,11 +183,16 @@ """Return w3's timezone identification string. Expresed as [+/-]hh:mm. For instance, PDT is -07:00 during - dayling savings and -08:00 otherwise. Zone is coincides with what + dayling savings and -08:00 otherwise. Zone coincides with what localtime(), etc., use. If no argument given, use the current time. """ + + if Globals.is_not_None('use_utc'): +# log.Log("DEBUG: Globals(use_utc) is set, instantly returning 'Z' ('Zulu' UTC identifier).", 7) + return 'Z' + if timeinseconds is None: timeinseconds = time.time() dst_in_effect = time.daylight and time.localtime(timeinseconds)[8] if dst_in_effect: offset = -time.altzone/60 @@ -157,16 +201,21 @@ elif offset < 0: prefix = "-" else: return "Z" # time is already in UTC + if Globals.is_not_None('use_compatible_timestamps'): time_separator = '-' + else: time_separator = ':' + log.Log("DEBUG: timezone_format separator used: '%s'.\n" % time_separator, 8) + hours, minutes = map(abs, divmod(offset, 60)) assert 0 <= hours <= 23 assert 0 <= minutes <= 59 - return "%s%02d:%02d" % (prefix, hours, minutes) + return "%s%02d%s%02d" % (prefix, hours, time_separator, minutes) def tzdtoseconds(tzd): """Given w3 compliant TZD, return how far ahead UTC is""" + """Now accepts both ':' and '-' separators""" if tzd == "Z": return 0 assert len(tzd) == 6 # only accept forms like +08:00 for now - assert (tzd[0] == "-" or tzd[0] == "+") and tzd[3] == ":" + assert (tzd[0] == "-" or tzd[0] == "+") and ( tzd[3] == ":" or tzd[3] == "-" or tzd[3] == "_" ) return -60 * (60 * int(tzd[:3]) + int(tzd[4:])) def cmp(time1, time2): @@ -239,6 +288,8 @@ match = _genstr_date_regexp1.search(timestr) or \ _genstr_date_regexp2.search(timestr) if not match: error() + + ## OM: No need to change below format from ':', because not used for filenames and stringtotime() was made failsafe. timestr = "%s-%02d-%02dT00:00:00%s" % (match.group('year'), int(match.group('month')), int(match.group('day')), gettzd()) t = stringtotime(timestr) [/code] Cheers, Oliver Mulatz _______________________________________________ rdiff-backup-users mailing list at [email protected] http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
