Hello FMEers, The desire to have control of FME just before and just after translation has been discussed and debated in this list for many years now. People have wanted to do different setup things before a translation begins, and different post-processing of log files or other translation artifacts after a translation ends.
At last this is possible, in a very powerful way. We've added a way to run an arbitrary Tcl script, from within the FME engine, just before and just after the translation begins/ends respectively. The script has access to as many of the translation artifacts (like counts, logfile, etc) as we could possibly do. The possibilities are immense, and the applications include customized logging (to databases even) of translation results to site-specific event logs, post processing of the translation results to rename files, pre processing to set up a translation environment, you name it. The draft documentation below contains seven examples which attempt to illustrate the breadth of the possibilities, including one example which inserts a record of the translation's activity into an Oracle database using the Oratcl package. This functionality has been available in the FME 2005 betas for nearly a month now, so enjoy! Dale ---------------------------------------------------------------------- Dale Lutz������������� Safe Software Inc.���������������� [EMAIL PROTECTED] VP Development�������� Surrey, BC, CANADA������� phone: (604) 501-9985 ���������������������� http://www.safe.com�������� fax: (604) 501-9965 ---------------------------------------------------------------------- Translation Begin/End Hooks =========================== When the FME translation engine is incorporated into larger systems, it may be useful to have the FME perform some additional site specific custom processing either before or after each translation. The FME_BEGIN_TCL and FME_END_TCL mapping file directives specify Tcl scripts to be run either before or after (respectively) the actual translation. The FME_END_TCL script has access to a number of variables which contain the statistics and status of the translation, and as such can be used to write customized logs or insert records into a database (perhaps using the TclODBC or Oratcl Tcl extensions -- see examples 6 and 7 respectively) to make a record of the translation's activity. The FME_END_TCL script can also read through the logfile produced by the FME to extract certain messages and record those results as well. The FME_BEGIN_TCL and FME_END_TCL scripts will share a Tcl interpretter, which means that the FME_BEGIN_TCL script can set up some Tcl global variables that will later be available to the FME_END_TCL script. However, in most scenarios it is likely that only an FME_END_TCL script will be needed. Lastly, the FME_BEGIN_TCL and FME_END_TCL interpretter is completely separate from any other Tcl interpretters used in FME; for example, the @TCL and @Tcl2 FME functions do not share this interpretter. The scripts to be executed for FME_BEGIN_TCL or FME_END_TCL can be specified right in the mapping file, or they can be "source'd" in from an external file. If they are specified in the mapping file, then any quotation marks must be escaped with a \ and continuation characters must be used to join all the lines in the script together into one logical FME mapping file line. This also then requires that the ; statement separator be used in the Tcl code. Alternately, if the Tcl script is placed in an external file and source'd in, then native Tcl quoting and formatting may be used. For example: FME_END_TCL proc finalize {} { \ ... tcl code here ... \ }; \ finalize; or FME_END_TCL source $(FME_MF_DIR_UNIX)/finalization.tcl Note that if the script contains a procedure definition, the script must call the procedure it defined, otherwise the procedure itself will not be executed and no processing will occur. If any errors are encountered during the processing of either the FME_BEGIN_TCL or FME_END_TCL script, the errors are written into the logfile. If there is no logfile, the errors will not be reported. These hooks are not available within the definition of a custom format. They can, of course, be used in a mapping file or workspace that itself employs a custom format. The FME Objects Tcl interface may not be used within the either of the FME_BEGIN_TCL or FME_END_TCL scripts. This is because using FME Objects at the translation start/end times would cause a configuration conflict within the FME translation system. However, the FME_BEGIN_TCL/FME_END_TCL scripts can themselves start new processes via the "exec" Tcl command, and these new processes could involve FME Objects Tcl without any conflict. FME_BEGIN_TCL ------------- The FME_BEGIN_TCL directive specifies a Tcl script to execute just prior to the start of translation. The script is executed after the mapping file has been completely parsed, and after the logfile has been opened, but before any of the readers or writers have begun to do their processing. The syntax is: FME_BEGIN_TCL <tcl script> where the <tcl script> is any valid tcl script to execute. The script may access the following global Tcl variables: Global Variable Contents ================================== ======================================================================== FME_MappingFileId : The value of the MAPPING_FILE_ID directive specified in the mapping file FME_END_TCL ----------- The FME_BEGIN_TCL directive specifies a Tcl script to execute just after translation has completed, either sucessfully or prematurely due to an error being encountered. If the translation ended due to an error, the script is executed after all cleanup is done, all reader and writers are shut down, and the logfile has been closed. If the translation was successful, the script is executed after all the the readers and writers have finished their work, and the logfile has been closed. The script has access to a number of Tcl global variables that contain statistics and other information about the translation. The script may access the following global Tcl variables, as well as any global variables set up in the FME_BEGIN_TCL. Note that if the translation failed, only the FME_Status and FME_FailureMessage will hold any values. Global Variable Contents ================================== ======================================================================== FME_Status : 0 if the trnaslation failed and 1 if it was successful FME_FailureMessage : The failure message if the translation failed, blank if the translation succeeded FME_FeaturesRead(<featureType>) : Each element in this array, indexed by feature type, holds the number of features read for that feature type FME_TotalFeaturesRead : The total number of features read FME_FeaturesWritten(<featureType>) : Each element in this array, indexed by feature type, holds the number of features written for that feature type FME_TotalFeaturesWritten : The total number of features written FME_TotalCoordinates : The total number of coordinates output FME_MappingFileId : The value of MAPPING_FILE_ID keyword if specified in the mapping file FME_ElapsedTime : The elapsed time from just before the FME_BEGIN_TCL script was called until just before the FME_END_TCL was called. FME_CPUTime : The CPU time from just before the FME_BEGIN_TCL was called until just before the FME_END_TCL was called. FME_LogFileName : The filename of the logfile used by this translation. Will be blank if no logfile was used. FME_StartingSeconds : The seconds since epoch when the translation was started (as returned by [clock seconds]) FME_StartingTimeStamp : The starting time of the translation, formatted as YYYY-MM-DD HH:MM:SS FME_EndingSeconds : The seconds since epoch when the translation ended (as returned by [clock seconds]) FME_EndingTimeStamp : The ending time of the translation, formatted as YYYY-MM-DD HH:MM:SS Example 1: ========== When placed into a complete mapping file, this mapping file fragment uses the translation begin/end hooks to cause the translation start and end times to be written to standard output. FME_BEGIN_TCL set gTranslationStartTime [clock format [clock seconds]] FME_END_TCL puts \"Translation started time: $gTranslationStartTime \nTranslation end time: [clock format [clock seconds]]\" Example 2: ========== When placed into a complete mapping file, this mapping file fragment uses the translation end hook and the global FME_Status variable to output a translation success or failure message. Notice that it first defines a procedure which will do this testing and output, and then it actually calls the procedure to do the work. FME_END_TCL proc finally {} { \ global FME_Status; \ if {$FME_Status == "1"} { \ puts \"Translation was successful\"; \ } else { \ puts \"Translation was NOT successful\"; \ }; \ }; \ finally This same example could also be written without using a procedure: FME_END_TCL if {$FME_Status == "1"} { \ puts \"Translation was successful\"; \ } else { \ puts \"Translation was NOT successful\"; \ } Example 3: ========== This example shows how macros used by the mapping file can be passed into a beginning or ending script as an argument to a procedure. This could be used to move, copy, email, backup or otherwise post/pre-process the destination datasets. In this example, the destination dataset is backed up prior to the start of translation to save any existing output file that would normally be overwritten. The mapping file fragment is: # The macro being used in this mapping file to set the destination # dataset directory is DestDataset. DWG_DATASET "$(DestDataset)" # Source in the Tcl script to run at the conclusion of this # translation. The script is stored in the same directory as # this mapping file. FME_BEGIN_TCL source $(FME_MF_DIR_UNIX)/backup.tcl; backup {$(DestDataset)} And the contents of the "backup.tcl" file are: proc backup {filename} { if {[file exists $filename]} { file copy -force $filename $filename.bak } } Example 4: ========== This example shows how the FME_END_TCL could be used to chain together translations, based on the results of a particular translation. In this example, if the original translation fails, then another fme session will be initiated. Note that in the Tcl file, the 2> NUL: routes any log and error messages to the null device, which effectively causes them to be ignored. The mapping file fragment is: # Source in the Tcl script to run at the conclusion of this # translation. The script is stored in the same directory as # this mapping file. FME_END_TCL source $(FME_MF_DIR_UNIX)/tryAnother.tcl And the contents of the "tryAnother.tcl" file are: set outputFile [open c:/temp/status.txt w+] if { $FME_Status == "1" } { puts $outputFile "Translation was successful" } else { puts $outputFile "Translation failed -- running alternate translation" exec fme.exe alternateTranslation.fme 2> NUL: } close $outputFile Example 5: ========== This example uses an external script to output a complete set of translation statistics to a summary file at the end of translation. The mapping file fragment is: # Set a mapping file id so that the Tcl finalization procedure "knows" which # mapping file was being run MAPPING_FILE_ID Shape to AutoCAD # Log all the message numbers so that we can later pull out only those # we are interested in. LOG_MESSAGE_NUMBERS yes # Source in the Tcl script to run at the conclusion of this # translation. The script is stored in the same directory as # this mapping file. FME_END_TCL source $(FME_MF_DIR_UNIX)/tclFinalization.tcl And the contents of the "tclFinalization.tcl" file are: # Open a file for writing out the translation stats set outputFile [open c:/temp/stats_out.txt w+] # Check for the translation status and output the desired message if { $FME_Status == "1" } { puts $outputFile "Translation was successful" } else { puts $outputFile "Translation failed" puts $outputFile "Error message was: $FME_FailureMessage" }; puts $outputFile "" # Output the unique Mapping file identifier set in the mapping file using MAPPING_FILE_ID puts $outputFile "Translation summary for $FME_MappingFileId" # Output the number of features read\written per feature type. puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile " Features read summary " puts $outputFile "---------------------------------------------------------------------------- ---" # Loop through a sorted listed of the feature types that were read, and output the # count for each one set formatSpec "%-65s: %12s" foreach featType [lsort [array names FME_FeaturesRead]] { puts $outputFile [format $formatSpec $featType $FME_FeaturesRead($featType)] } puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile [format $formatSpec "Total Features Read" $FME_TotalFeaturesRead]; puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile "" puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile " Features Written summary " puts $outputFile "---------------------------------------------------------------------------- ---" # Loop through a sorted listed of the feature types that were written, and output the # count for each one foreach featType [lsort [array names FME_FeaturesWritten]] { puts $outputFile [format $formatSpec $featType $FME_FeaturesWritten($featType)] } puts $outputFile "" puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile [format $formatSpec "Total Features Written" $FME_TotalFeaturesWritten] puts $outputFile [format $formatSpec "Total Coordinates Written" $FME_TotalCoordinates] puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile "" # Look for any lines in the logfile that were warnings, and output a count of them to # the summary file. Also, check if there was any "unexpected input remover" # statistics line and report a non-zero count if there was (this may happen # when workbench generated mapping files are run against input datasets # with different feature types than those that were expected) # And also fish out the system status log message (which is #246014) and copy # it into the output file set logFileExists [file exists $FME_LogFileName] set warnings 0 if {$logFileExists} { set logFile [open $FME_LogFileName r] while {[gets $logFile line] >= 0} { if {[regexp {WARN} $line]} { incr warnings } elseif {[regexp {#246014} $line]} { puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile $line puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile "" } elseif {[regexp {Unexpected Input Remover} $line]} { set totalFeatures 0 set acceptedFeatures 0 set rejectedFeatures 0 set line [regsub {^.*Unexpected} $line {Unexpected}] catch {scan $line "Unexpected Input Remover(TestFactory): Tested %d input features -- %d features passed, %d features failed." totalFeatures acceptedFeatures rejectedFeatures} if {$rejectedFeatures > 0} { puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile [format $formatSpec "Features with Unexpected Feature Types" $rejectedFeatures] puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile "" } } } close $logFile } puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile [format $formatSpec "Logfile $FME_LogFileName Warnings" $warnings] puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile "" puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile [format $formatSpec "Total Elapsed Time (seconds)" $FME_ElapsedTime] puts $outputFile [format $formatSpec "Total CPU Time (seconds)" $FME_CPUTime] puts $outputFile "---------------------------------------------------------------------------- ---" puts $outputFile "" # And close the file close $outputFile Example 6: ========== This example uses an external script to insert a record of the translation's activity into a database using the TclODBC package. In this example, an Access database is created (if necessary) and populated with a row for each translation that completes. Sidebar: Installing TclODBC To use TclODBC within the FME environment, follow these steps: 1) Read about the TclODBC package at http://wiki.tcl.tk/2151 2) Download the TclODBC package from http://sourceforge.net/projects/tclodbc 3) Unzip the result 4) Edit the supplied setup.tcl to change: if [string match *lib $i] { to if [string match *lib* $i] { 5) In a command prompt in the directory holding setup.tcl, run "fme setup.tcl". This will install the TclODBC package in the FME's tcl environment. 6) Ensure any scripts which require the TclODBC package start with: package require tclodbc To invoke the sample script below script which does the database insertion, the mapping file contains this line: FME_END_TCL source $(FME_MF_DIR_UNIX)/recordTranslationODBC.tcl And the contents of the "recordTranslationODBC.tcl" file are: # ========================================================================= # # recordTranslationODBC.tcl # # This script records the execution of a translation using # the TclODBC package. This example includes the creation of the # Access database and table for storing the translation results. # One row for each translation run is inserted into the table. # Note that for an acutal production system the "puts" statements # would be removed. package require tclodbc # ========================================================================= # # Set up some variables that are used within to create and connect # to the database via ODBC set driver "Microsoft Access Driver (*.mdb)" set dbfile c:/translations.mdb set dsn XLATION # ========================================================================= # Create the database if it isn't already there. We wouldn't do this if the # database was a server-based one, but for this example we're just using Access. if {![file exists $dbfile]} { puts "Creating database $dbfile" database configure config_dsn $driver [list "CREATE_DB=\"$dbfile\" General"] } else { puts "Using existing database $dbfile" } # ========================================================================= # Make an ODBC datasource for this database, and connect to it # First always remove the DSN if it was there already. If we were # using ODBC to connect to a "real" database, then we'd just assume # the DSN is already valid catch {database configure remove_dsn $driver "DSN=$dsn"} database configure add_dsn $driver [list "DSN=$dsn" "DBQ=$dbfile"] database db $dsn # ========================================================================= # Create the table we want to insert into if it wasn't there already if {[llength [db tables XLATION_RESULTS]] == 0} { puts "Creating XLATION_RESULTS table in database $dbfile" db "CREATE TABLE XLATION_RESULTS ( MappingFileID VARCHAR(50), StartTime TIMESTAMP, EndTime TIMESTAMP, CpuTime DOUBLE, Successful CHAR(3), NumFeatures INTEGER)" } else { puts "XLATION_RESULTS table present in database $dbfile" } # ========================================================================= # All of that was just setup, now we actually can insert our row, # commit it, and disconnect from the datasource set Success yes if {$FME_Status == 0} {set Success no} db "INSERT INTO XLATION_RESULTS (MappingFileID, StartTime, EndTime, CpuTime, Successful, NumFeatures) VALUES ('$FME_MappingFileId', \{ts '$FME_StartingTimeStamp'\}, \{ts '$FME_EndingTimeStamp'\}, $FME_CPUTime, '$Success', $FME_TotalFeaturesWritten)" db commit db disconnect # ========================================================================= # If you were connecting to a server-based database, you probably # would NOT remove the ODBC DSN at the end. But since we were file based # we will remove it database configure remove_dsn $driver "DSN=$dsn" Example 7: ========== This example uses an external script to insert a record of the translation's activity into an Oracle database using the Oratcl package. Sidebar: Installing Oratcl To use Oratcl within the FME environment, follow these steps: 1) Install and configure Oracle client software on your computer. Verify you can access your database using sqlPlus. 2) Read about and download the Oratcl package at http://oratcl.sourceforge.net 3) Unzip the result into $FME_HOME/tcl_library/oratcl4.1 (you will have to create this directory, and the 4.1 may need to change to match version of Oratcl you've downloaded). After unzipping, move the msvcr70.dll and oratcl41.dll to $FME_HOME/tcl_library (up one level). 4) Ensure any scripts which require the Oratcl package start with: package require Oratcl Note -- it is important that the O be uppercase in Oratcl To invoke the sample script below script which does the database insertion, the mapping file contains this line: FME_END_TCL source $(FME_MF_DIR_UNIX)/recordTranslationOracle.tcl And the contents of the "recordTranslationOracle.tcl" file are: # ========================================================================= # # recordTranslationOracle.tcl # # This script records the execution of a translation using # the Oratcl package. This example includes the creation of the # XLATION_RESULTS table for storing the translation results. # One row for each translation run is inserted into the table. package require Oratcl # ========================================================================= # Login to the Oracle service set username scott set password tiger set service amidala set loginHandle [oralogon $username/[EMAIL PROTECTED] # ========================================================================= # Determine if the xlation_results table we wish to record results to exists set tableExists no set statementHandle [oraopen $loginHandle] orasql $statementHandle "select * from user_tables where table_name = 'XLATION_RESULTS'" while {[orafetch $statementHandle -datavariable row] == 0} { set tableExists yes } # ========================================================================= # Create the xlation_results table if we need to if {$tableExists == "no"} { orasql $statementHandle "CREATE TABLE XLATION_RESULTS ( MappingFileID VARCHAR(50), StartTime TIMESTAMP, EndTime TIMESTAMP, CpuTime FLOAT, XlationSuccessful CHAR(3), NumFeatures INTEGER)" } # ========================================================================= # Insert the translation results into the table set Success yes if {$FME_Status == 0} {set Success no} orasql $statementHandle "INSERT INTO XLATION_RESULTS (MappingFileID, StartTime, EndTime, CpuTime, XlationSuccessful, NumFeatures) VALUES ('$FME_MappingFileId', timestamp '$FME_StartingTimeStamp', timestamp '$FME_EndingTimeStamp', $FME_CPUTime, '$Success', $FME_TotalFeaturesWritten)" -commit # ========================================================================= # Shut down the statement handle oraclose $statementHandle # ========================================================================= # Logout of our oracle service oralogoff $loginHandle Get the maximum benefit from your FME, FME Objects, or SpatialDirect via our Professional Services team. Visit www.safe.com/services for details. Yahoo! Groups Links <*> To visit your group on the web, go to: http://groups.yahoo.com/group/fme/ <*> To unsubscribe from this group, send an email to: [EMAIL PROTECTED] <*> Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
