Re: Faster searching for Entourage

[Allen Watson]

Title: Re: Faster searching for Entourage

On or near 12/4/03 2:17 AM, Dan Warne at [EMAIL PROTECTED] observed:

>>>>> Beside the possibilities Jeremy mentioned, there are also archiving
>>>>> solutions that back up to FileMaker Pro databases, which can be indexed,
>>>>> and
>>>>> which can search my current archive of more than 35,000 messages in a
>>>>> second
>>>>> or two.
>>>>
>>>> Interesting... What are the solutions called?
>>>>
>>> The most full-featured one with full support by its author, John Carlsen, is
>>> eMessage Archiver (shareware). It comes with a runtime engine (meaning you
>>> do not have to own FileMaker to use this package.)
>
>
> I downloaded eMA and had a quick look, but it doesn't really have a very
> good interface... Very unlike Entourage. I also had a quick look for a full
> text search function (a VERY quick look) and couldn't find one. I was using
> the eMA Solution runtime, not FMP6 though.
>
FileMaker's basic "Find" command on the message text will do a full text search. It may take a long time to index the first time you use it, but after that it will maintain the indices. Same with any field you search. FMP6 also allows you to search and locate text strings within a text field (earlier versions just locate the record, but don't highlight the text within the field.)

> It's also unfortunate that eMA won't store email attachments. That's a deal
> breaker for me ... I have thousands of attachments, and their proximity to
> messages is important - I don't want to have them in a file directory
> separate from the message...

The only way I know of that you are going to get the attachments kept with the messages is to keep them in Entourage databases...which begs the question we started with of slow text searching.

If you drag multiple messages to a finder folder or the desktop, they'll get saved with the attachments encoded along with them. Searching them will also, therefore, search the encoded attachments (needlessly), which will waste considerable time if you have large attachments. For fast text searching of text files or mbox files, investigate OS X's "grep" command! There are utilities that give grep a GUI, although not too sophisticated; see "grepArtee" for one. You don't have to use regular expressions; you can search for plain text strings if you want.

I even wrote an AppleScript that will ask for a directory and then do a grep search of text files in that directory (and any folders it contains), and then report the file names to you, so you can select one to open. It opens files with BBEdit if you have it, and then locates the string in the BBEdit window; if you don’t have BBEdit it just does a “Finder” open command and lets the file open in whatever is its native application. If you know some regular expressions, you can use them in the search string. “.” is the wildcard character; “.+” is “any character one or more times”.

Try it:
-- Search Text, by Allen Watson, searchs for strings within text files.
property searchTerm : ""
property newLine : ASCII character 10
property startDisk : path to startup disk as string

on run
   -- Search through files and list files containing a match
   set rootFolder to choose folder with prompt "Root folder for search?"
    set doSearch to true
   repeat while doSearch
       set {searchTerm, theBut} to {text returned, button returned} of (display dialog "Term to search for?" default answer searchTerm buttons {"Exact case", "Ignore case", "grep Help"} default button 2)
        if theBut = "grep Help" then
           display dialog "Basic regular expressions:
. = any character
+ = previous character repeated one or more times
* = previous character repeated zero or more times
^ = Start of line
$ = end of line
\\r = return
[...] = match any character between brackets, e.g. [abc]
[x-y] = match any character in range from x to y"
            set {searchTerm, theBut} to {text returned, button returned} of (display dialog "Term to search for?" default answer searchTerm buttons {"Exact case", "Ignore case"} default button 2)
        end if
       if theBut is "Exact case" then
           set sw to "-lr "
        else
           set sw to "-lri "
        end if
       set searchTerm to unixifyFileSpec(searchTerm)
        set f to rootFolder as text
       set x to path to startup disk as text
       if f begins with x then
           set l to length of x
           set f to text (l + 1) thru -1 of f
       end if
       set f to unixifyFileSpec(f)
        
        set theScript to "grep " & sw & searchTerm & " " & f & "*"
        try
           set theResult to do shell script theScript
       on error
           set f to "Volumes/" & f
           set theScript to "grep " & sw & searchTerm & " " & f & "*"
            try
               set theResult to do shell script theScript
           on error theErr
               display dialog "Problem most likely no matches." & return & theErr
               set theResult to {}
            end try
       end try
       set oldDelims to AppleScript's text item delimiters
       set AppleScript's text item delimiters to {return}
        set listResult to text items of theResult
       set AppleScript's text item delimiters to oldDelims
       set foundCount to count listResult
       display dialog "I found " & foundCount & " files that match. Try again, or select one to insert, or several to examine/edit?" buttons {"Insert one", "Edit some", "Try again"} default button 1
       if button returned of result is "Insert one" then
           set doSearch to false
           if foundCount = 0 then return
           set desiredOne to choose from list listResult with prompt "Choose one file to insert." without multiple selections allowed
           if desiredOne is false then return
           set desiredOne to macifyFilespec(item 1 of desiredOne)
            set t to read alias desiredOne
           set the clipboard to t
           display dialog "Contents of \"" & desiredOne & "\" is on the clipboard."
        else if button returned of result is "Edit some" then
           set doSearch to false
           if foundCount = 0 then return
           set desiredFiles to choose from list listResult with prompt "Please choose one or more files to examine." with multiple selections allowed
           if desiredFiles is false then return
           repeat with aFile in desiredFiles
               set aFile to macifyFilespec(aFile)
                set aFile to alias aFile
               tell application "Finder" to set useBB to exists application file id "R*ch"
                if useBB then
                   tell application "Finder" to set theApp to name of application file id "R*ch"
                    using terms from application "BBEdit"
                        tell application theApp
                           open aFile
                           activate
                           find searchTerm searching in text 1 of text window 1 options {search mode:grep, starting at top:true, wrap around:false, backwards:false, case sensitive:false, match words:false, extend selection:false} with selecting match
                       end tell
                   end using terms from
               else
                   tell application "Finder"
                        open alias aFile
                   end tell
               end if
           end repeat
       else
           -- Try a different search
       end if
   end repeat
end run
on unixifyFileSpec(f)
    set oldDelims to AppleScript's text item delimiters
   set AppleScript's text item delimiters to ":"
    set f to text items of f
   set AppleScript's text item delimiters to "/"
    set f to f as text
   
    -- Escape spaces
   set AppleScript's text item delimiters to " "
    set f to text items of f
   set AppleScript's text item delimiters to "\\ "
    set f to f as text
   
    set AppleScript's text item delimiters to oldDelims
   return f
end unixifyFileSpec

on macifyFilespec(f)
    set oldDelims to AppleScript's text item delimiters
   set AppleScript's text item delimiters to "/"
    set f to text items of f
   set AppleScript's text item delimiters to ":"
    set f to f as text
   
    -- UnEscape spaces
   set AppleScript's text item delimiters to "\\ "
    set f to text items of f
   set AppleScript's text item delimiters to " "
    set f to f as text
   
    set AppleScript's text item delimiters to oldDelims
   if f begins with "Volumes:" then
       set f to text 9 thru -1 of f
   else
       set f to startDisk & f
   end if
   return f
end macifyFilespec


--
Microsoft MVP for Entourage/OE/Word (MVPs are volunteers)
Allen Watson <[EMAIL PROTECTED]> Entourage FAQ site: <http://www.entourage.mvps.org/>
AppleScripts for Outlook Express and Entourage:
 <http://members.thinkaccess.net/[EMAIL PROTECTED]/Scripts/>
Entourage Help Pages: <http://www.entourage.mvps.org/>