Re: Rebuilding SIS attachment links from log
That works brilliantly! I revised my script now to take advantage of yours - #!/bin/bash # These variables need to be customized for your particular installation LISTFILE='/var/mail/files.fail' ATTACHMENT_STORAGE_BASE='/var/files/attachments' # These variables are based on current Dovecot behaviour and should not require changing HASH_FOLDER='hashes' function usage { echo "Dovecot Single-Instance-Storage Attachment Repair" echo "usage: dovesisfix [-d] [-t] [-v] [-h]" echo " -t | --test-only perform logfile analysis and show steps to be taken without any on-disk modification" echo " -v | --verbose provide verbose messages at each step" echo " -d | --debug provide additional debug messages" echo " -h | --helpthis screen" } while [ "$1" != "" ]; do case $1 in -d | --debug ) DEBUG=1 VERBOSE=1 ;; -t | --test-only ) TESTMODE=1 ;; -v | --verbose )VERBOSE=1 ;; -h | --help ) usage exit ;; * ) usage exit 1 esac shift done while read -r LINE do # Now extract the aa/bb/ prefix, the base attachment file name, and user hash CATEGORY_PATH="${LINE:0:5}" BASE_HASH="${LINE:6:40}" USER_HASH="${LINE:47:32}" ATTACH_SOURCE="$ATTACHMENT_STORAGE_BASE/$CATEGORY_PATH/$HASH_FOLDER/$BASE_HASH" ATTACH_TARGET="$ATTACHMENT_STORAGE_BASE/$CATEGORY_PATH/$BASE_HASH-$USER_HASH" # If in debug/verbose mode show operation about to occur if [ "$VERBOSE" = 1 ]; then echo "The file $ATTACH_SOURCE must be linked to $ATTACH_TARGET" fi # Verify that source exists if [ ! -f "$ATTACH_SOURCE" ]; then echo "ERROR: File $ATTACH_SOURCE does not exist. You must restore this from a backup and run this utility again." fi # This is a Good Thing. if [ "$DEBUG" = 1 ]; then echo "The file $ATTACH_SOURCE appears to be a valid file." fi # Check if user link mysteriously reappeared if [ -f "$ATTACH_TARGET" ]; then echo "INFO: File $ATTACH_TARGET exists. This may mean the fault has been previously corrected. Clearing/rotating the logfile $LOGFILE is appropriate now." continue fi # Prepare to create user link LINK_LINE="$ATTACH_SOURCE $ATTACH_TARGET" if [ "$DEBUG" = 1 ]; then echo "About to execute command: ln $LINK_LINE" fi # If test mode, do nothing if [ "$TESTMODE" = 1 ]; then continue fi # There's probably more tests I could/should do - but I don't know how # So...if we're not in test mode...time to do it to it. LINK_CREATED=$(ln $LINK_LINE) if [ "$VERBOSE" = 1 ]; then echo "Repair result for $ATTACH_TARGET - $LINK_CREATED" fi done < "$LISTFILE" On 3/4/2015 9:29 AM, Hardy Flor wrote: I use this script for review. For a correct result Dovecot should not run. #!/bin/bash attdir="/var/files/attachments" maildir="/var/mail" cd "$attdir" || exit 1 find -type f -printf "%P\n" | grep -v "hashes" | sort -u >"$maildir/attachments.s" cd "$maildir" || exit 2 rm "$maildir/files.ok" "$maildir/files.fail" 2>/dev/null for f in $(find -type f -name "m.*" -printf "%P\n"); do doveadm dump -t dbox "$f" | egrep "^msg.ext-ref" | while read z; do set -- $z while [ -n "$1" ]; do if [[ $1 == */* ]]; then test -r "$attdir/$1" && echo "$1" >>"$maildir/files.ok" || echo "$1" >>"$maildir/files.fail" fi shift done done done sort -u "$maildir/files.ok" >"$maildir/files.s" diff -Nu "$maildir/attachments.s" "$maildir/files.s" | tee "$maildir/files.diff" | egrep "^\-" Am 03.03.2015 um 22:08 schrieb Daniel Miller: This seems simple enough...I'm just not script wizard. If someone can throw together a starting point I can test and tweak it from there. It seems to me: 1. Read /var/mail/mail.err or specified logfile 2. For each "failed: read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line, a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists i. If attachmentHash is missing display such for possible backup searching. b. create link attachmentHash-userHash to hashes/attachmentHash 3. Continue to end of file Can this be done via "pure" BASH? Need sed/awk as well? -- Daniel L. Miller, VP - Engineering, SET AM Fire & Electronic Services, Inc. [AMFES] dmil...@amfes.com 702-312-5276
Re: Rebuilding SIS attachment links from log
I use this script for review. For a correct result Dovecot should not run. #!/bin/bash attdir="/var/files/attachments" maildir="/var/mail" cd "$attdir" || exit 1 find -type f -printf "%P\n" | grep -v "hashes" | sort -u >"$maildir/attachments.s" cd "$maildir" || exit 2 rm "$maildir/files.ok" "$maildir/files.fail" 2>/dev/null for f in $(find -type f -name "m.*" -printf "%P\n"); do doveadm dump -t dbox "$f" | egrep "^msg.ext-ref" | while read z; do set -- $z while [ -n "$1" ]; do if [[ $1 == */* ]]; then test -r "$attdir/$1" && echo "$1" >>"$maildir/files.ok" || echo "$1" >>"$maildir/files.fail" fi shift done done done sort -u "$maildir/files.ok" >"$maildir/files.s" diff -Nu "$maildir/attachments.s" "$maildir/files.s" | tee "$maildir/files.diff" | egrep "^\-" Am 03.03.2015 um 22:08 schrieb Daniel Miller: This seems simple enough...I'm just not script wizard. If someone can throw together a starting point I can test and tweak it from there. It seems to me: 1. Read /var/mail/mail.err or specified logfile 2. For each "failed: read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line, a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists i. If attachmentHash is missing display such for possible backup searching. b. create link attachmentHash-userHash to hashes/attachmentHash 3. Continue to end of file Can this be done via "pure" BASH? Need sed/awk as well?
Re: Rebuilding SIS attachment links from log
Daniel, please help me understand, since I met your same problem on 2015-02-26 as you did. > 2. For each "failed: > read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line, > a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists > i. If attachmentHash is missing display such for possible Is this a FEATURE of Dovecot SIS? Or a known bug described somewhere? Loosing detached attachments, I mean. > Can this be done via "pure" BASH? Need sed/awk as well? Well, you need a way to strip out the "-userHash" part from the string, so a little bit of sed and regex is needed. You also need to work out source and destination files. Still, personally I think the problem is somewhere else and should be fixed. Else, stop using SIS and let the SAN do the deduplication. Am I wrong? Paolo
Re: Rebuilding SIS attachment links from log
Well, with no guarantees or promises whatsoever...here's my first attempt. I'm certain someone else can come up with a much more robust solution but it's a starting point. Since I typically see a batch of the error messages during an FTS update, my current usage is: 1. doveadm fts rescan -u user-to-fix 2. Perform a fts search to do a full mailbox scan 3. Check the mail log to see the date/time of the errors (I usually have a window open with tail -f) 4. Something like "grep 'Mar 3 20:17' mail.log > mail.err" gives me a starting point execute dovesisfix and see if it helps. -- Daniel On 3/3/2015 1:08 PM, Daniel Miller wrote: This seems simple enough...I'm just not script wizard. If someone can throw together a starting point I can test and tweak it from there. It seems to me: 1. Read /var/mail/mail.err or specified logfile 2. For each "failed: read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line, a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists i. If attachmentHash is missing display such for possible backup searching. b. create link attachmentHash-userHash to hashes/attachmentHash 3. Continue to end of file Can this be done via "pure" BASH? Need sed/awk as well? #!/bin/bash # These variables need to be customized for your particular installation LOGFILE='/var/log/mail.err' ATTACHMENT_STORAGE_BASE='/var/mail/attachments' # These variables are based on current Dovecot behaviour and should not require changing HASH_FOLDER='hashes' # Initialization PREVIOUS_ERR='' ERR='' function usage { echo "Dovecot Single-Instance-Storage Attachment Repair" echo "usage: dovesisfix [-d] [-t] [-v] [-h]" echo " -t | --test-only perform logfile analysis and show steps to be taken without any on-disk modification" echo " -v | --verbose provide verbose messages at each step" echo " -d | --debug provide additional debug messages" echo " -h | --helpthis screen" } while [ "$1" != "" ]; do case $1 in -d | --debug ) DEBUG=1 VERBOSE=1 ;; -t | --test-only ) TESTMODE=1 ;; -v | --verbose )VERBOSE=1 ;; -h | --help ) usage exit ;; * ) usage exit 1 esac shift done while read -r LINE do ERR=$LINE # Format of log line has date, host, process, user, mail storage file, and then the # attachment path failure, followed by a duplicate of the path as an argument to open, # and then final details. # Verify this line is indeed a dovecot attachment error. Don't look for "dovecot" specifically # in case that name was changed - but the individual worker names are probably safe searches. # So we test against "attachments-connector" - hopefully that's good enough. TEST=$(echo "$ERR" | sed -n "s|.*attachments-connector.*|1|p") if [ "$TEST" != "1" ]; then # Not found - not relevant if [ "$DEBUG" = 1 ]; then echo "Skipping non-relevant log line. $ERR" fi continue fi # Remove prefacing details from log line - find start of attachment path within log line # This is a greedy match - so the second attachment path is returned along with the trailing info ATTACH_LINE_FILTER="s|.*$ATTACHMENT_STORAGE_BASE||" ATTACH_LINE=$(echo "$ERR" | sed "$ATTACH_LINE_FILTER") # Now extract the aa/bb/ prefix, the base attachment file name, and user hash CATEGORY_PATH="${ATTACH_LINE:1:5}" BASE_HASH="${ATTACH_LINE:7:40}" USER_HASH="${ATTACH_LINE:48:32}" ATTACH_SOURCE="$ATTACHMENT_STORAGE_BASE/$CATEGORY_PATH/$HASH_FOLDER/$BASE_HASH" ATTACH_TARGET="$ATTACHMENT_STORAGE_BASE/$CATEGORY_PATH/$BASE_HASH-$USER_HASH" # There appear to be duplicate lines - so to try to filter some out. if [ "$PREVIOUS_TARGET" = "$ATTACH_TARGET" ]; then if [ "$DEBUG" = 1 ]; then echo "Skipping duplicate log line for $ATTACH_SOURCE-$ATTACH_TARGET" fi continue fi PREVIOUS_TARGET=$ATTACH_TARGET # If in debug/verbose mode show operation about to occur if [ "$VERBOSE" = 1 ]; then echo "The file $ATTACH_SOURCE must be linked to $ATTACH_TARGET" fi # Verify that source exists if [ ! -f "$ATTACH_SOURCE" ]; then echo "ERROR: File $ATTACH_SOURCE does not exist. You must restore this from a backup and run this utility again." fi # This is a Good Thing. if [ "$DEBUG" = 1 ]; then echo "The file $ATTACH_SOURCE appears to be a valid file." fi # Check if user link mysteriously reappeared if [ -f "$ATTACH_TARGET" ]; then echo "INFO: File $ATTACH_TARGET exists. This may mean the fault h
Rebuilding SIS attachment links from log
This seems simple enough...I'm just not script wizard. If someone can throw together a starting point I can test and tweak it from there. It seems to me: 1. Read /var/mail/mail.err or specified logfile 2. For each "failed: read(/var/mail/attachments/aa/bb/attachmentHash-userHash" line, a. Confirm /var/mail/attachments/aa/bb/hashes/attachmentHash exists i. If attachmentHash is missing display such for possible backup searching. b. create link attachmentHash-userHash to hashes/attachmentHash 3. Continue to end of file Can this be done via "pure" BASH? Need sed/awk as well? -- Daniel