Added: manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/script.xml URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/script.xml?rev=1617058&view=auto ============================================================================== --- manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/script.xml (added) +++ manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/script.xml Sun Aug 10 07:45:35 2014 @@ -0,0 +1,437 @@ +<?xml version="1.0"?> + +<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" + "http://forrest.apache.org/dtd/document-v20.dtd";> + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<document> + + <header> + <title>ManifoldCF Scripting Language</title> + </header> + + <body> + <section> + <title>Overview</title> + <p></p> + <p>The ManifoldCF scripting language allows symbolic communication with the ManifoldCF API Service + in order to define connections and jobs, and perform crawls. The language provides support for JSON-like hierarchical + documents, as well as the ability to construct properly encoded REST URLs. It also has support for simple control flow + and error handling.</p> + </section> + + <section> + <title>How to use the script interpreter</title> + <p></p> + <p>The ManifoldCF script interpreter can be used in two ways - either as a real-time shell (executing a script as it is typed), + or interpreting a script file. The main class of the interpreter is <em>org.apache.manifoldcf.scriptengine.ScriptParser</em>, + and the two ways of invoking it are:</p> + <source> +java -cp ... org.apache.manifoldcf.scriptengine.ScriptParser + </source> + <p>or:</p> + <source> +java -cp ... org.apache.manifoldcf.scriptengine.ScriptParser <script_file> <arg1> ... <argN> + </source> + <p>If you choose to invoke ScriptParser in interactive mode, simply type your script one line at a time. Any errors will be reported immediately, + and the ScriptParser will accordingly exit. You can also type ^Z to terminate the script.</p> + <p>If you use ScriptParser with a scripting file, that file will be read and interpreted. The arguments you provide will be loaded into an array + of strings, which is accessible from your script as the variable named <em>__args__</em>.</p> + <section> + <title>Running the script interpreter by hand</title> + <p></p> + <p>When you build ManifoldCF, the required dependent jars for the scripting language are copied to <em>dist/script-engine/lib</em>. + You can run the interpreter in interactive mode by typing:</p> + <source> +cd dist\script-engine +run-script.bat <args> + </source> + <p>Or, on Linux:</p> + <source> +cd dist/script-engine +run-script.sh <args> + </source> + <p>You will need to set the environment variable <strong>ENGINE_HOME</strong> to point at the <em>dist/script-engine</em> directory beforehand, so that + the scripts can locate the appropriate jars.</p> + </section> + <section> + <title>Running the script interpreter using Ant</title> + <p></p> + <p>You can also start the script interpreter with all the correct required jars using Ant. Simply type the following:</p> + <source> +ant run-script-interpreter + </source> + <p>This will start the script interpreter in interactive mode only.</p> + </section> + <section> + <title>Running the script interpreter using Maven</title> + <p></p> + <p>You can also run the script interpreter using maven. The commands are:</p> + <source> +cd framework/script-engine +mvn exec:exec + </source> + <p>This, once again, will start the interpreter in interactive mode.</p> + </section> + </section> + + <section> + <title>Script language syntax</title> + <p></p> + <p>A ManifoldCF script is not sensitive to whitespace or indenting. All comments begin with a '#' character and end with the end of that line. + Unquoted tokens can include alphanumeric characters, plus '_', '$', and '@'. Numeric tokens always begin with a number ('0'-'9'), and are + considered floating-point if they include a decimal point ('.'). Otherwise they are integers. String tokens can be quoted with either a double quote + ('"') or a single quote, and within strings characters can be escaped with a preceding backslash ('\').</p> + <p>A ManifoldCF script has a syntax that is readily described with a BNF grammar. See below.</p> + <source> +program: +--> statements + +statements: +--> statement1 ... statementN + +statement: +--> 'set' expression '=' expression ';' +--> 'print' expression ';' +--> 'if' expression 'then' statements ['else' statements] ';' +--> 'while' expression 'do' statements ';' +--> 'break' ';' +--> 'error' expression ';' +--> 'insert' expression 'into' expression ['at' expression] ';' +--> 'remove' expression 'from' expression ';' +--> 'wait' expression ';' +--> 'GET' expression '=' expression ';' +--> 'PUT' expression '=' expression 'to' expression ';' +--> 'POST' expression '=' expression 'to' expression ';' +--> 'DELETE' expression ';' + +expression: +--> '(' expression ')' +--> expression '&&' expression +--> expression '||' expression +--> '!' expression +--> expression '&' expression +--> expression '|' expression +--> expression '==' expression +--> expression '!=' expression +--> expression '>=' expression +--> expression '<=' expression +--> expression '>' expression +--> expression '<' expression +--> expression '+' expression +--> expression '-' expression +--> expression '*' expression +--> expression '/' expression +--> '-' expression +--> '[' [expression [',' expression ...]] ']' +--> '{' [expression [',' expression ...]] '}' +--> '<<' expression ':' expression ':' [expression '=' expression [',' expression '=' expression ...]] ':' [expression [',' expression ...]] '>>' +--> expression '[' expression ']' +--> expression '.' token +--> token +--> string +--> number +--> 'true' | 'false' +--> 'null' +--> 'new' newexpression +--> 'isnull' expression + +newexpression: +--> 'url' expression +--> 'connectionname' expression +--> 'configuration' +--> 'configurationnode' expression +--> 'array' +--> 'dictionary' +--> 'queryarg' expression ['=' expression] + + </source> + </section> + + <section> + <title>Script language variables</title> + <p></p> + <p>Variables in the ManifoldCF scripting language determine the behavior of all aspects of expression evaluation, with the exception of operator precedence. + In particular, every canonical variable has the ability to support arbitrary <em>attributes</em> (which are named properties of the variable), + <em>subscripts</em> (children which are accessed by a numeric subscript), and all other <em>operations</em>, such as '+' or '=='. Not all kinds of + variable instance will in fact support all such features. Should you try to use a feature with a variable that does not support it, you will receive a + ScriptException telling you what you did wrong.</p> + <p>Since the actual operation details are bound to the variable, for binary operations the left-hand variable typically determines what actually takes place. For example:</p> + <source> +print 3+7; + [java] 10 +print "3"+7; + [java] 37 + </source> + <p>There is, of course, a way to caste a variable to a different type. For example:</p> + <source> +print "3".__int__+7; + [java] 10 + </source> + <p>Here, we are using the built-in attribute <em>__int__</em> to obtain the integer equivalent of the original string variable "3". See the following table for + a list of some of the standard attributes and their meanings:</p> + <table> + <caption>Standard attributes</caption> + <tr><th>Attribute name</th><th>Meaning</th></tr> + <tr><td>__script__</td><td>Returns the script code that would create this variable</td></tr> + <tr><td>__string__</td><td>Returns the string value of the variable, if any</td></tr> + <tr><td>__int__</td><td>Returns the integer value of the variable, if any</td></tr> + <tr><td>__float__</td><td>Returns the floating-point value of the variable, if any</td></tr> + <tr><td>__boolean__</td><td>Returns the boolean value of the variable, if any</td></tr> + <tr><td>__size__</td><td>Typically returns the number of subscript children</td></tr> + <tr><td>__type__</td><td>Returns the 'type' of the variable</td></tr> + <tr><td>__value__</td><td>Returns the 'value' of the variable</td></tr> + <tr><td>__dict__</td><td>Returns a dictionary equivalent of the variable</td></tr> + <tr><td>__OK__</td><td>Returns a boolean 'true' if the variable was "OK", false otherwise</td></tr> + <tr><td>__NOTFOUND__</td><td>Returns a boolean 'true' if the variable was "NOTFOUND", false otherwise</td></tr> + <tr><td>__CREATED__</td><td>Returns a boolean 'true' if the variable was "CREATED", false otherwise</td></tr> + </table> + <p>Obviously, only some variables will support each of the standard attributes. You will receive a script exception if you try to obtain a non-existent + attribute for a variable.</p> + <section> + <title>Integers</title> + <p>Integer variable types are created by non-quoted numeric values that do not have a '.' in them. For example, the character '4' will create an integer + variable type with a value of 4.</p> + <p>The operations supported for this variable type, and their meanings, are listed in the table below:</p> + <table> + <caption>Integer operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>binary +</td><td>Addition, yielding an integer</td><td>4+7</td></tr> + <tr><td>binary -</td><td>Subtraction, yielding an integer</td><td>7-4</td></tr> + <tr><td>binary *</td><td>Multiplication, yielding an integer</td><td>7*4</td></tr> + <tr><td>binary /</td><td>Division, yielding an integer</td><td>7/4</td></tr> + <tr><td>unary -</td><td>Negation, yielding an integer</td><td>-4</td></tr> + <tr><td>binary ==</td><td>Equality comparison, yielding a boolean</td><td>7 == 4</td></tr> + <tr><td>binary !=</td><td>Inequality comparison, yielding a boolean</td><td>7 != 4</td></tr> + <tr><td>binary >=</td><td>Greater or equals comparison, yielding a boolean</td><td>7 >= 4</td></tr> + <tr><td>binary <=</td><td>Less or equals comparison, yielding a boolean</td><td>7 <= 4</td></tr> + <tr><td>binary ></td><td>Greater comparison, yielding a boolean</td><td>7 > 4</td></tr> + <tr><td>binary <</td><td>Less comparison, yielding a boolean</td><td>7 < 4</td></tr> + <tr><td>binary &</td><td>Bitwise AND, yielding an integer</td><td>7 & 4</td></tr> + <tr><td>binary |</td><td>Bitwise OR, yielding an integer</td><td>7 | 4</td></tr> + <tr><td>unary !</td><td>Bitwise NOT, yielding an integer</td><td>! 7</td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em>, <em>__string__</em>, <em>__int__</em>, and <em>__float__</em> are supported + by integer types.</p> + </section> + <section> + <title>Strings</title> + <p>String variable types are created by quoted sequences of characters. For example, the character '"hello world"' will create a string + variable type with an (unquoted) value of "hello world".</p> + <p>The operations supported for this variable type, and their meanings, are listed in the table below:</p> + <table> + <caption>String operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>binary +</td><td>Concatenation, yielding a string</td><td>"hi" + "there"</td></tr> + <tr><td>binary ==</td><td>Equality comparison, yielding a boolean</td><td>"hi" == "there"</td></tr> + <tr><td>binary !=</td><td>Inequality comparison, yielding a boolean</td><td>"hi" != "there"</td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em>, <em>__string__</em>, <em>__int__</em>, and <em>__float__</em> are supported + by string types.</p> + </section> + <section> + <title>Floating-point numbers</title> + <p>Float variable types are created by non-quoted numeric values that have a '.' in them. For example, the token '4.1' will create a float + variable type with a value of 4.1</p> + <p>The operations supported for this variable type, and their meanings, are listed in the table below:</p> + <table> + <caption>Float operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>binary +</td><td>Addition, yielding a float</td><td>4.0+7.0</td></tr> + <tr><td>binary -</td><td>Subtraction, yielding a float</td><td>7.0-4.0</td></tr> + <tr><td>binary *</td><td>Multiplication, yielding a float</td><td>7.0*4.0</td></tr> + <tr><td>binary /</td><td>Division, yielding a float</td><td>7.0/4.0</td></tr> + <tr><td>unary -</td><td>Negation, yielding a float</td><td>-4.0</td></tr> + <tr><td>binary ==</td><td>Equality comparison, yielding a boolean</td><td>7.0 == 4.0</td></tr> + <tr><td>binary !=</td><td>Inequality comparison, yielding a boolean</td><td>7.0 != 4.0</td></tr> + <tr><td>binary >=</td><td>Greater or equals comparison, yielding a boolean</td><td>7.0 >= 4.0</td></tr> + <tr><td>binary <=</td><td>Less or equals comparison, yielding a boolean</td><td>7.0 <= 4.0</td></tr> + <tr><td>binary ></td><td>Greater comparison, yielding a boolean</td><td>7.0 > 4.0</td></tr> + <tr><td>binary <</td><td>Less comparison, yielding a boolean</td><td>7.0 < 4.0</td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em>, <em>__string__</em>, <em>__int__</em>, and <em>__float__</em> are supported + by float types.</p> + </section> + <section> + <title>Booleans</title> + <p>Boolean variable types are created by the keywords <strong>true</strong> and <strong>false</strong>. For example, the code 'true' will create a boolean + variable type with a value of "true".</p> + <p>The operations supported for this variable type, and their meanings, are listed in the table below:</p> + <table> + <caption>Boolean operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>binary ==</td><td>Equality comparison, yielding a boolean</td><td>7.0 == 4.0</td></tr> + <tr><td>binary !=</td><td>Inequality comparison, yielding a boolean</td><td>7.0 != 4.0</td></tr> + <tr><td>binary &&</td><td>AND logical operation, yielding a boolean</td><td>true && false</td></tr> + <tr><td>binary ||</td><td>OR logical operation, yielding a boolean</td><td>true || false</td></tr> + <tr><td>binary &</td><td>AND logical operation, yielding a boolean</td><td>true & false</td></tr> + <tr><td>binary |</td><td>OR logical operation, yielding a boolean</td><td>true | false</td></tr> + <tr><td>unary !</td><td>NOT logical operation, yielding a boolean</td><td>! true</td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em> and <em>__boolean__</em> are supported + by boolean types.</p> + </section> + <section> + <title>Arrays</title> + <p>Array variable types are created by an initializer of the form <strong>[</strong> [<em>expression</em> [<strong>,</strong> <em>expression</em> ...]] <strong>]</strong>. For example, the script code '[3, 4]' will create an array + variable type with two values, the integer "3" and the integer "4".</p> + <p>The operations supported for this variable type, and their meanings, are listed in the table below:</p> + <table> + <caption>Array operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>subscript []</td><td>Find the specified subscript variable, yielding the variable</td><td>[3,4] [0]</td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em> and <em>__size__</em> are supported + by array types, as well as the <em>insert</em> and <em>remove</em> statements.</p> + </section> + <section> + <title>Dictionaries</title> + <p>Dictionary variable types are created using the "new" operator, e.g. <strong>new</strong> <strong>dictionary</strong>.</p> + <p>The operations supported for this variable type, and their meanings, are listed in the table below:</p> + <table> + <caption>Array operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>subscript []</td><td>Find the specified key, yielding the keyed variable</td><td>mydict ["keyname"]</td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em> and <em>__size__</em> are supported + by dictionary types.</p> + </section> + <section> + <title>Configurations</title> + <p>Configuration variables contain the equivalent of the JSON used to communicate with the ManifoldCF API. They can be created using an initializer + of the form <strong>{</strong> [<em>expression</em> [<strong>,</strong> <em>expression</em> ...]] <strong>}</strong>. For example, the script code '{ << "outputconnector" : "" : : , << "description" : "Solr" : : >>, << "class_name" : "org.apache.manifoldcf.agents.output.solr.SolrConnector" : : >> >> }' + would create a configuration variable equivalent to one that might be returned from the ManifoldCF API if it was queried for the output connectors registered by the system.</p> + <p>The operations supported for this variable type, and their meanings are listed in the table below:</p> + <table> + <caption>Configuration operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>subscript []</td><td>Find the specified child configuration node variable, yielding the variable</td><td>myconfig [0]</td></tr> + <tr><td>binary +</td><td>Append a configuration child node variable to the list</td><td>myconfig + << "something" : "somethingvalue" : : >></td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em>, <em>__dict__</em>, and <em>__size__</em> are supported + by configuration variable types, as well as the <em>insert</em> and <em>remove</em> statements.</p> + </section> + <section> + <title>Configuration nodes</title> + <p>Configuration node variable types are children of configuration variable types or configuration node variable types. They have several components, as listed below:</p> + <ul> + <li>A type</li> + <li>A value</li> + <li>Attributes, described as a set of name/value pairs</li> + <li>Children, which must be configuration node variable types</li> + </ul> + <p>Configuration node variable types can be created using an initializer of the form <strong><<</strong> <em>expression</em> <strong>:</strong> <em>expression</em> <strong>:</strong> [<em>expression</em> <strong>=</strong> <em>expression</em> [<strong>,</strong> <em>expression</em> <strong>=</strong> <em>expression</em> ...]] <strong>:</strong> [<em>expression</em> [<strong>,</strong> <em>expression</em> ... ]] '>>'. + The first expression represents the type of the node. The second is the node's value. The series of '=' expressions represents attribute names and values. The last series represents + the children of the node. For example, the script code '<< "description" : "Solr" : : >>' represents a node of type 'description' with a value of 'Solr', with no attributes or children.</p> + <p>The operations supported for this variable type, and their meanings are listed in the table below:</p> + <table> + <caption>Configuration node operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>subscript []</td><td>Find the specified child configuration node variable, yielding the variable</td><td>myconfig [0]</td></tr> + <tr><td>binary +</td><td>Append a configuration child node variable to the list</td><td>myconfig + << "something" : "somethingvalue" : : >></td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em>, <em>__string__</em>, <em>__size__</em>, <em>__type__</em>, <em>__dict__</em> and <em>__value__</em> are supported + by configuration node variable types, as well as the <em>insert</em> and <em>remove</em> statements.</p> + </section> + <section> + <title>URLs</title> + <p>URL variable types exist to take care of the details of URL encoding while assembling the REST URL's needed to describe objects in ManifoldCF's REST API. A URL variable + type can be created using a 'new' operation of the form <strong>new</strong> <strong>url</strong> <em>expression</em>, where the expression is the already-encoded root path. For example, the script code 'new url "http://localhost:8345/mcf-api-service/json";' + would create a URL variable type with the root path "http://localhost:8345/mcf-api-service/json";.</p> + <p>The operations supported for this variable type, and their meanings are listed in the table below:</p> + <table> + <caption>URL operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>binary ==</td><td>Equals comparison, yielding a boolean</td><td>url1 == url2</td></tr> + <tr><td>binary !=</td><td>Non-equals comparison, yielding a boolean</td><td>url1 != url2</td></tr> + <tr><td>binary +</td><td>Append and encode another path or query argument element, yielding a URL</td><td>url1 + "repositoryconnections"</td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em> and <em>__string__</em> are supported + by URL variable types.</p> + </section> + <section> + <title>Query Arguments</title> + <p>Query Argument variable types exist to take care of the details of URL encoding while assembling the query arguments of a REST URL for ManifoldCF's REST API. A Query Argument variable + type can be created using a 'new' operation of the form <strong>new</strong> <strong>queryarg</strong> <em>expression</em> [<strong>=</strong> <em>expression</em>], where the first expression is the + query argument name, and the second optional expression is the query argument value. For example, the script code 'new queryarg "report" = "simple"' + would create a Query Argument variable type representing the query argument "report=simple". To add query arguments to a URL, simply add them using the '+' operator, + for example "urlvar = urlvar + new queryarg 'report' = 'simple';" .</p> + <p>The operations supported for this variable type, and their meanings are listed in the table below:</p> + <table> + <caption>Query Argument operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>binary ==</td><td>Equals comparison, yielding a boolean</td><td>arg1 == arg2</td></tr> + <tr><td>binary !=</td><td>Non-equals comparison, yielding a boolean</td><td>arg1 != arg2</td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em> and <em>__string__</em> are supported + by Query Argument variable types.</p> + </section> + <section> + <title>Connection names</title> + <p>Connection name variable types exist to perform the extra URL encoding needed for ManifoldCF's REST API. Connection names must be specially encoded so that they do not + contain slash characters ('/'). Connection name variable types take care of this encoding.</p> + <p>You can create a connection name variable type using the following syntax: <strong>new</strong> <strong>connectionname</strong> <em>expression</em>, where the expression is the name of the connection. + </p> + <p>The operations supported for this variable type, and their meanings are listed in the table below:</p> + <table> + <caption>URL operations</caption> + <tr><th>Operation</th><th>Meaning</th><th>Example</th></tr> + <tr><td>binary ==</td><td>Equals comparison, yielding a boolean</td><td>cn1 == cn2</td></tr> + <tr><td>binary !=</td><td>Non-equals comparison, yielding a boolean</td><td>cn1 != cn2</td></tr> + </table> + <p>In addition, the standard attributes <em>__script__</em> and <em>__string__</em> are supported + by connection name variable types.</p> + </section> + <section> + <title>Results</title> + <p>Result variable types capture the result of a GET, PUT, POST, or DELETE statement. They consist of two parts:</p> + <ul> + <li>A result code</li> + <li>A result configuration value</li> + </ul> + <p>There is no way to directly create a result variable type, nor does it support any operations. However, the standard attributes <em>__script__</em>, <em>__string__</em>, + <em>__value__</em>, <em>__OK__</em>, <em>__NOTFOUND__</em>, and <em>__CREATED__</em> are all supported by result variable types.</p> + </section> + </section> + + <section> + <title>Statements</title> + <p>The statements available to a ManifoldCF script programmer are designed to support interaction with the ManifoldCF API. Thus, there is support for + all four HTTP verbs, as well as basic variable setting and control flow. The table below describes each statement type:</p> + <table> + <caption>Statement types</caption> + <tr><th>Statement</th><th>Meaning</th><th>Example</th></tr> + <tr><td>'set' expression '=' expression ';'</td><td>Sets the variable described by the first expression with the value computed for the second</td><td>set myvar = 4 + 5;</td></tr> + <tr><td>'print' expression ';'</td><td>Prints the string value of the expression to stdout</td><td>print "hello world";</td></tr> + <tr><td>'if' expression 'then' statements ['else' statements] ';'</td><td>If the boolean value of the expression is 'true', executes the first set of statements, otherwise executes the (optional) second set</td><td>if true then print "hello"; else print "there"; ;</td></tr> + <tr><td>'while' expression 'do' statements ';'</td><td>While expression is true, execute the specified statements, and repeat</td><td>while count > 0 do set count = count - 1; ;</td></tr> + <tr><td>'break' ';'</td><td>Exits from the nearest enclosing while loop</td><td>while true do break; ;</td></tr> + <tr><td>'error' expression ';'</td><td>Aborts the script with a script exception based on the string value of the expression</td><td>error "bad stuff";</td></tr> + <tr><td>'wait' expression ';'</td><td>Waits the number of milliseconds corresponding to the integer value of the expression</td><td>wait 1000;</td></tr> + <tr><td>'insert' expression 'into' expression ['at' expression] ';'</td><td>Inserts the first expression into the second variable expression, either at the end or optionally at the position specified by the third expression</td><td>insert 4 into myarray at 0 ;</td></tr> + <tr><td>'delete' expression 'from' expression ';'</td><td>Deletes the element described by the first expression from the second expression</td><td>delete 0 from myarray ;</td></tr> + <tr><td>'GET' expression '=' expression ';'</td><td>Perform an HTTP GET from the URL specified in the second expression capturing the result in the first expression</td><td>GET result = new url "http://localhost:8345/mcf-api-service/json/repositoryconnections"; ;</td></tr> + <tr><td>'DELETE' expression '=' expression ';'</td><td>Perform an HTTP DELETE on the URL specified in the second expression capturing the result in the first expression</td><td>DELETE result = myurl ;</td></tr> + <tr><td>'PUT' expression '=' expression 'to' expression ';'</td><td>Perform an HTTP PUT of the second expression to the URL specified in the third expression capturing the result in the first expression</td><td>PUT result = configurationObject to myurl ;</td></tr> + <tr><td>'POST' expression '=' expression 'to' expression ';'</td><td>Perform an HTTP POST of the second expression to the URL specified in the third expression capturing the result in the first expression</td><td>POST result = configurationObject to myurl ;</td></tr> + </table> + </section> + </body> + +</document>
Added: manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/technical-resources.xml URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/technical-resources.xml?rev=1617058&view=auto ============================================================================== --- manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/technical-resources.xml (added) +++ manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/technical-resources.xml Sun Aug 10 07:45:35 2014 @@ -0,0 +1,60 @@ +<?xml version="1.0" encoding="utf-8"?> + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<document> +<header><title>ManifoldCF - å¼å人åèµæº</title></header> +<properties> +</properties> +<body> + + <section id="securityarchitecturediagrams"> + <title>å®å ¨ç»æå¾</title> + <p>ä¸é¢æ¯ManifoldCFçå®å ¨ç»æå¾ï¼</p> + <br/><br/> + <figure src="images/zh_CN/SecurityArchitecture.png" alt="ManifoldCF Security Architecture" width="80%"/> + <br/><br/> + <p>å¦ä½¿ç¨Apache认è¯ï¼æ¨¡åå为ï¼</p> + <br/><br/> + <figure src="images/zh_CN/LCFApacheArchitecture.png" alt="ManifoldCF Security Architecture, with Apache" width="80%"/> + <br/><br/> + </section> + + <section id="processarchitecturediagrams"> + <title>æµç¨ç»æå¾</title> + <p>ä¸é¢æ¯ManifoldCFçæµç¨ç»æå¾ï¼</p> + <br/><br/> + <figure src="images/zh_CN/MCFProcessArchitecture.png" alt="ManifoldCF Process Architecture" width="80%"/> + <br/><br/> + </section> + + <section id="howtowriteconnectors"> + <title>å建è¿æ¥å¨</title> + <p>éè¿å建è¿æ¥å¨å¯è¿ä¸æ¥äºè§£ManifoldCFå·¥ç¨ã亦å¯å°ææè´¡èµ éæ¤å·¥ç¨ã + </p> + <ul> + <li><a href="writing-output-connectors.html">å¦ä½å建è¾åºè¿æ¥å¨</a></li> + <li><a href="writing-transformation-connectors.html">å¦ä½å建转æ¢è¿æ¥å¨</a></li> + <li><a href="writing-mapping-connectors.html">å¦ä½å建ç¨æ·æ å°è¿æ¥å¨</a></li> + <li><a href="writing-authority-connectors.html">å¦ä½å建æéè¿æ¥å¨</a></li> + <li><a href="writing-repository-connectors.html">å¦ä½å建åå¨åºè¿æ¥å¨</a></li> + </ul> + </section> + +</body> +</document> Propchange: manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/technical-resources.xml ------------------------------------------------------------------------------ svn:executable = * Propchange: manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/technical-resources.xml ------------------------------------------------------------------------------ svn:mime-type = text/plain;charse=utf-8 Added: manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-authority-connectors.xml URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-authority-connectors.xml?rev=1617058&view=auto ============================================================================== --- manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-authority-connectors.xml (added) +++ manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-authority-connectors.xml Sun Aug 10 07:45:35 2014 @@ -0,0 +1,142 @@ +<?xml version="1.0"?> + +<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" + "http://forrest.apache.org/dtd/document-v20.dtd";> + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<document> + + <header> + <title>Writing authority connectors</title> + </header> + + <body> + <section> + <title>Writing an Authority Connector</title> + <p></p> + <p>An authority connector to a repository allows a repository's security model to be enforced by a search engine. Its only function is to convert a user name (which is often a Kerberos principal name) into a set of _access tokens_.</p> + <p></p> + <p>The definition of an access token within ManifoldCF for a given repository is completely defined by the connectors that deal with that repository, with one exception. That exception is for Active Directory. Active Directory is so prevalent as a repository authorization mechanism that ManifoldCF currently treats it as the "default" authority - that is, if you don't specify another authority when you define a repository connection, ManifoldCF presumes that you mean that Active Directory should be the controlling authority for the connection. In that case, an access token is simply an Active Directory SID.</p> + <p></p> + <p>For those repositories that do not use Active Directory as their authorization mechanism, an authority connector should be written, along with the repository connector for the repository. Access tokens in that case represent a contract between your implementation of the authority connector for the repository, and the repository connector for the repository. They must work together to define access tokens that will limit document access when used properly within any search engine query.</p> + <p></p> + <p>As is the case with all connectors under the ManifoldCF umbrella, an authority connector consists of a single parts:</p> + <p></p> + <ul> + <li>A class implementing an interface (in this case, <em>org.apache.manifoldcf.authorities.interfaces.IAuthorityConnector</em>)</li> + </ul> + <p></p> + <section> + <title>Key concepts</title> + <p></p> + <p>The authority connector abstraction makes use of, or introduces, the following concepts:</p> + <p></p> + <table> + <tr><th>Concept</th><th>What it is</th></tr> + <tr><td>Configuration parameters</td><td>A hierarchical structure, internally represented as an XML document, which describes a specific configuration of a specific authority connector, i.e. <strong>how</strong> the connector should do its job; see <em>org.apache.manifoldcf.core.interfaces.ConfigParams</em></td></tr> + <tr><td>Authority connection</td><td>An authority connector instance that has been furnished with configuration data</td></tr> + <tr><td>User name</td><td>The name of a user, which is often a Kerberos principal name, e.g. <em>[email protected]</em></td></tr> + <tr><td>Access token</td><td>An arbitrary string, which is only meaningful within the context of a specific authority connector, that describes a quantum of authorization</td></tr> + <tr><td>Connection management/threading/pooling model</td><td>How an individual authority connector class instance is managed and used</td></tr> + <tr><td>Service interruption</td><td>A specific kind of exception that signals ManifoldCF that the output repository is unavailable, and gives a best estimate of when it might become available again; see <em>org.apache.manifoldcf.agents.interfaces.ServiceInterruption</em></td></tr> + </table> + <p></p> + </section> + <section> + <title>Implementing the Authority Connector class</title> + <p></p> + <p>A very good place to start is to read the javadoc for the authority connector interface. You will note that the javadoc describes the usage and pooling model for a connector class pretty thoroughly. It is very important to understand the model thoroughly in order to write reliable connectors! Use of static variables, for one thing, must be done in a very careful way, to avoid issues that would be hard to detect with a cursory test.</p> + <p></p> + <p>The second thing to do is to examine some of the provided authority connector implementations. The Documentum connector, the LiveLink connector, the Memex connector, and the Meridio connector all include authority connectors which demonstrate (to some degree) the sorts of techniques you will need for an effective implementation. You will also note that all of these connectors extend a framework-provided authority connector base class, found at <em>org.apache.manifoldcf.authorities.authorities.BaseAuthorityConnector</em>. This base class furnishes some basic bookkeeping logic for managing the connector pool, as well as default implementations of some of the less typical functionality a connector may have. For example, connectors are allowed to have database tables of their own, which are instantiated when the connector is registered, and are torn down when the connector is removed. This is, however, not very typical, and the base implementation reflects that.</p> + <p></p> + <section> + <title>Principle methods</title> + <p></p> + <p>The principle methods an implementer should be concerned with for creating an authority connector are the following:</p> + <p></p> + <table> + <tr><th>Method</th><th>What it should do</th></tr> + <tr><td><strong>getAuthorizationResponse()</strong></td><td>Obtain the authorization response, given a user name</td></tr> + <tr><td><strong>outputConfigurationHeader()</strong></td><td>Output the head-section part of an authority connection <em>ConfigParams</em> editing page</td></tr> + <tr><td><strong>outputConfigurationBody()</strong></td><td>Output the body-section part of an authority connection <em>ConfigParams</em> editing page</td></tr> + <tr><td><strong>processConfigurationPost()</strong></td><td>Receive and process form data from an authority connection <em>ConfigParams</em> editing page</td></tr> + <tr><td><strong>viewConfiguration()</strong></td><td>Output the viewing HTML for an authority connection <em>ConfigParams</em> object</td></tr> + </table> + <p></p> + <p>These methods come in two broad classes: (a) functional methods for doing the work of the connector; (b) UI methods for configuring a connection. Together they do the heavy lifting of your connector.</p> + <p></p> + <p>The <em>getAuthorizationResponse()</em> method returns an <em>AuthorizationResponse</em> object, which can describe a number of conditions:</p> + <p></p> + <table> + <tr><th>Condition</th><th>Meaning</th></tr> + <tr><td>RESPONSE_OK</td><td>The access tokens for the user were successfully obtained from the repository, and are being returned</td></tr> + <tr><td>RESPONSE_UNREACHABLE</td><td>The repository is currently unreachable, and appropriate disabling tokens are being returned</td></tr> + <tr><td>RESPONSE_USERNOTFOUND</td><td>The user was not found within the repository, and appropriate disabling tokens are being returned</td></tr> + <tr><td>RESPONSE_USERUNAUTHORIZED</td><td>The user was found, but was in some way disabled, and appropriate disabling tokens are being returned</td></tr> + </table> + <p></p> + <p></p> + <p>In all cases, the connector returns access tokens. But in the case where token lookup has failed in some way, it is the responsibility of the connector to insure that inappropriate content is not viewed. Usually, this is done by ingesting a "global deny" token attached to all documents from the given repository, and then having the associated authority connector return this global deny token when error conditions apply.</p> + <p></p> + <p></p> + </section> + <section> + <title>Notes on connector UI methods</title> + <p></p> + <p>The crawler UI uses a tabbed layout structure, and thus each of these elements must properly implement the tabbed model. This means that the "header" methods above must add the desired tab names to a specified array, and the "body" methods must provide appropriate HTML which handles both the case where a tab is displayed, and where it is not displayed. Also, it makes sense to use the appropriate css definitions, so that the connector UI pages have a similar look-and-feel to the rest of ManifoldCF's crawler ui. We strongly suggest starting with one of the supplied authority connector's UI code, both for a description of the arguments to each page, and for some decent ideas of ways to organize your connector's UI code.</p> + <p></p> + </section> + </section> + <section> + <title>Implementation support provided by the framework</title> + <p></p> + <p>ManifoldCF's framework provides a number of helpful services designed to make the creation of a connector easier. These services are summarized below. (This is not an exhaustive list, by any means.)</p> + <p></p> + <ul> + <li>Lock management and synchronization (see <em>org.apache.manifoldcf.core.interfaces.LockManagerFactory</em>)</li> + <li>Cache management (see <em>org.apache.manifoldcf.core.interfaces.CacheManagerFactory</em>)</li> + <li>Local keystore management (see <em>org.apache.manifoldcf.core.KeystoreManagerFactory</em>)</li> + <li>Database management (see <em>org.apache.manifoldcf.core.DBInterfaceFactory</em>)</li> + </ul> + <p></p> + <p>For UI method support, these too are very useful:</p> + <p></p> + <ul> + <li>Multipart form processing (see <em>org.apache.manifoldcf.ui.multipart.MultipartWrapper</em>)</li> + <li>HTML encoding (see <em>org.apache.manifoldcf.ui.util.Encoder</em>)</li> + <li>HTML formatting (see <em>org.apache.manifoldcf.ui.util.Formatter</em>)</li> + </ul> + <p></p> + </section> + <section> + <title>DO's and DON'T DO's</title> + <p></p> + <p>It's always a good idea to make use of an existing infrastructure component, if it's meant for that purpose, rather than inventing your own. There are, however, some limitations we recommend you adhere to.</p> + <p></p> + <ul> + <li>DO make use of infrastructure components described in the section above</li> + <li>DON'T make use of infrastructure components that aren't mentioned, without checking first</li> + <li>NEVER write connector code that directly uses framework database tables, other than the ones installed and managed by your connector</li> + </ul> + <p></p> + <p>If you are tempted to violate these rules, it may well mean you don't understand something important. At the very least, we'd like to know why. Send email to [email protected] with a description of your problem and how you are tempted to solve it.</p> + </section> + </section> + </body> +</document> \ No newline at end of file Added: manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-mapping-connectors.xml URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-mapping-connectors.xml?rev=1617058&view=auto ============================================================================== --- manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-mapping-connectors.xml (added) +++ manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-mapping-connectors.xml Sun Aug 10 07:45:35 2014 @@ -0,0 +1,144 @@ +<?xml version="1.0"?> + +<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" + "http://forrest.apache.org/dtd/document-v20.dtd";> + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<document> + + <header> + <title>Writing user mapping connectors</title> + </header> + + <body> + <section> + <title>Writing a User Mapping Connector</title> + <p></p> + <p>A user mapping connector allows a user name to be transformed in a manner that depends on the functionality of the connector. In some cases, no connection to + an external repository is required (for example, simple string transformations), while in some cases one might imagine such a connector consulting with (say) an + LDAP system to look up a specific name.</p> + <p></p> + <p>A user name is just a string, which is designed to represent a user identity. Some user names have specific forms - for instance, Active Directory user names are + often represented in the form <code>user@domain</code>. But, most importantly, the exact name used can often depend on the particular system being addressed.</p> + <p></p> + <p>As is the case with all connectors under the ManifoldCF umbrella, a user mapping connector consists of a single part:</p> + <p></p> + <ul> + <li>A class implementing an interface (in this case, <em>org.apache.manifoldcf.authorities.interfaces.IMappingConnector</em>)</li> + </ul> + <p></p> + <section> + <title>Key concepts</title> + <p></p> + <p>The mapping connector abstraction makes use of, or introduces, the following concepts:</p> + <p></p> + <table> + <tr><th>Concept</th><th>What it is</th></tr> + <tr><td>Configuration parameters</td><td>A hierarchical structure, internally represented as an XML document, which describes a specific configuration of a specific mapping connector, i.e. <strong>how</strong> the connector should do its job; see <em>org.apache.manifoldcf.core.interfaces.ConfigParams</em></td></tr> + <tr><td>Mapping connection</td><td>An mapping connector instance that has been furnished with configuration data</td></tr> + <tr><td>User name</td><td>The name of a user, which is often a Kerberos principal name, e.g. <em>[email protected]</em></td></tr> + <tr><td>Connection management/threading/pooling model</td><td>How an individual mapping connector class instance is managed and used</td></tr> + </table> + <p></p> + </section> + <section> + <title>Implementing the Mapping Connector class</title> + <p></p> + <p>A very good place to start is to read the javadoc for the mapping connector interface. You will note that the javadoc describes the usage and pooling model for a + connector class pretty thoroughly. It is very important to understand the model thoroughly in order to write reliable connectors! Use of static variables, for one thing, + must be done in a very careful way, to avoid issues that would be hard to detect with a cursory test.</p> + <p></p> + <p>The second thing to do is to examine some of the provided mapping connector implementations. The only connector presently included (the Regular Expression + user mapping connector) demonstrates some of the sorts of techniques you will need for an effective + implementation. You will also note that all of these connectors extend a framework-provided mapping connector base class, found at + <em>org.apache.manifoldcf.authorities.mappers.BaseMappingConnector</em>. This base class furnishes some basic bookkeeping logic for managing the + connector pool, as well as default implementations of some of the less typical functionality a connector may have. For example, connectors are allowed to have + database tables of their own, which are instantiated when the connector is registered, and are torn down when the connector is removed. This is, however, not + very typical, and the base implementation reflects that.</p> + <p></p> + <section> + <title>Principle methods</title> + <p></p> + <p>The principle methods an implementer should be concerned with for creating a mapping connector are the following:</p> + <p></p> + <table> + <tr><th>Method</th><th>What it should do</th></tr> + <tr><td><strong>mapUser()</strong></td><td>Given an input user name, find the corresponding output user name</td></tr> + <tr><td><strong>outputConfigurationHeader()</strong></td><td>Output the head-section part of a mapping connection <em>ConfigParams</em> editing page</td></tr> + <tr><td><strong>outputConfigurationBody()</strong></td><td>Output the body-section part of a mapping connection <em>ConfigParams</em> editing page</td></tr> + <tr><td><strong>processConfigurationPost()</strong></td><td>Receive and process form data from a mapping connection <em>ConfigParams</em> editing page</td></tr> + <tr><td><strong>viewConfiguration()</strong></td><td>Output the viewing HTML for a mapping connection <em>ConfigParams</em> object</td></tr> + </table> + <p></p> + <p>These methods come in two broad classes: (a) functional methods for doing the work of the connector; (b) UI methods for configuring a connection. Together they + do the heavy lifting of your connector.</p> + <p></p> + <p></p> + </section> + <section> + <title>Notes on connector UI methods</title> + <p></p> + <p>The crawler UI uses a tabbed layout structure, and thus each of these elements must properly implement the tabbed model. This means that the "header" methods + above must add the desired tab names to a specified array, and the "body" methods must provide appropriate HTML which handles both the case where a tab is + displayed, and where it is not displayed. Also, it makes sense to use the appropriate css definitions, so that the connector UI pages have a similar look-and-feel + to the rest of ManifoldCF's crawler ui. We strongly suggest starting with one of the supplied mapping connector's UI code, both for a description of the arguments + to each page, and for some decent ideas of ways to organize your connector's UI code.</p> + <p></p> + </section> + </section> + <section> + <title>Implementation support provided by the framework</title> + <p></p> + <p>ManifoldCF's framework provides a number of helpful services designed to make the creation of a connector easier. These services are summarized below. + (This is not an exhaustive list, by any means.)</p> + <p></p> + <ul> + <li>Lock management and synchronization (see <em>org.apache.manifoldcf.core.interfaces.LockManagerFactory</em>)</li> + <li>Cache management (see <em>org.apache.manifoldcf.core.interfaces.CacheManagerFactory</em>)</li> + <li>Local keystore management (see <em>org.apache.manifoldcf.core.KeystoreManagerFactory</em>)</li> + <li>Database management (see <em>org.apache.manifoldcf.core.DBInterfaceFactory</em>)</li> + </ul> + <p></p> + <p>For UI method support, these too are very useful:</p> + <p></p> + <ul> + <li>Multipart form processing (see <em>org.apache.manifoldcf.ui.multipart.MultipartWrapper</em>)</li> + <li>HTML encoding (see <em>org.apache.manifoldcf.ui.util.Encoder</em>)</li> + <li>HTML formatting (see <em>org.apache.manifoldcf.ui.util.Formatter</em>)</li> + </ul> + <p></p> + </section> + <section> + <title>DO's and DON'T DO's</title> + <p></p> + <p>It's always a good idea to make use of an existing infrastructure component, if it's meant for that purpose, rather than inventing your own. There are, however, + some limitations we recommend you adhere to.</p> + <p></p> + <ul> + <li>DO make use of infrastructure components described in the section above</li> + <li>DON'T make use of infrastructure components that aren't mentioned, without checking first</li> + <li>NEVER write connector code that directly uses framework database tables, other than the ones installed and managed by your connector</li> + </ul> + <p></p> + <p>If you are tempted to violate these rules, it may well mean you don't understand something important. At the very least, we'd like to know why. Send email + to [email protected] with a description of your problem and how you are tempted to solve it.</p> + </section> + </section> + </body> +</document> \ No newline at end of file Added: manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-output-connectors.xml URL: http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-output-connectors.xml?rev=1617058&view=auto ============================================================================== --- manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-output-connectors.xml (added) +++ manifoldcf/trunk/site/src/documentation/content/xdocs/zh_CN/writing-output-connectors.xml Sun Aug 10 07:45:35 2014 @@ -0,0 +1,149 @@ +<?xml version="1.0"?> + +<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" + "http://forrest.apache.org/dtd/document-v20.dtd";> + +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +<document> + + <header> + <title>Writing output connectors</title> + </header> + + <body> + <section> + <title>Writing an Output Connector</title> + <p></p> + <p>An output connector furnishes the mechanism by which content that has been fetched from a repository gets handed to a back-end repository for processing. It also furnishes a mechanism for removing previously-processed content from that back end repository.</p> + <p></p> + <p>As is the case with all connectors under the ManifoldCF umbrella, an output connector consists of a single part, which is:</p> + <p></p> + <ul> + <li>A class implementing an interface (in this case, <em>org.apache.manifoldcf.agents.interfaces.IOutputConnector</em>)</li> + </ul> + <p></p> + <section> + <title>Key concepts</title> + <p></p> + <p>The output connector abstraction makes use of, or introduces, the following concepts:</p> + <p></p> + <table> + <tr><th>Concept</th><th>What it is</th></tr> + <tr><td>Configuration parameters</td><td>A hierarchical structure, internally represented as an XML document, which describes a specific configuration of a specific output connector, i.e. <strong>how</strong> the connector should do its job; see <em>org.apache.manifoldcf.core.interfaces.ConfigParams</em></td></tr> + <tr><td>Output connection</td><td>An output connector instance that has been furnished with configuration data</td></tr> + <tr><td>Document URI</td><td>The unique URI (or, in some cases, file IRI) of a document, which is meant to be displayed in search engine results as the link to the document</td></tr> + <tr><td>Repository document</td><td>An object that describes a document's contents, including raw document data (as a stream), metadata (as either strings or streams), and access tokens; see <em>org.apache.manifoldcf.agents.interfaces.RepositoryDocument</em></td></tr> + <tr><td>Connection management/threading/pooling model</td><td>How an individual output connector class instance is managed and used</td></tr> + <tr><td>Activity infrastructure</td><td>The framework API provided to specific methods allowing those methods to perform specific actions within the framework, e.g. recording activities; see <em>org.apache.manifoldcf.agents.interfaces.IOutputAddActivity</em> and <em>org.apache.manifoldcf.agents.interfaces.IOutputRemoveActivity</em></td></tr> + <tr><td>Output specification</td><td>A hierarchical structure, internally represented as an XML document, which describes <strong>what</strong> a specific output connector should do in the context of a specific job; see <em>org.apache.manifoldcf.agents.interfaces.OutputSpecification</em></td></tr> + <tr><td>Output version string</td><td>A simple string, used for comparison purposes, that allows ManifoldCF to figure out if an ingestion operation needs to be repeated as a result of changes to the output specification in effect for a document</td></tr> + <tr><td>Service interruption</td><td>A specific kind of exception that signals ManifoldCF that the output repository is unavailable, and gives a best estimate of when it might become available again; see <em>org.apache.manifoldcf.agents.interfaces.ServiceInterruption</em></td></tr> + </table> + <p></p> + <p></p> + </section> + <section> + <title>Implementing the Output Connector class</title> + <p></p> + <p>A very good place to start is to read the javadoc for the output connector interface. You will note that the javadoc describes the usage and pooling model for a connector class pretty thoroughly. It is very important to understand the model thoroughly in order to write reliable connectors! Use of static variables, for one thing, must be done in a very careful way, to avoid issues that would be hard to detect with a cursory test.</p> + <p></p> + <p>The second thing to do is to examine some of the provided output connector implementations. The GTS connector, the SOLR connector, the OpenSearchServer connector, the ElasticSearch connector, and the Null Output connector all are output connectors which demonstrate (to some degree) the sorts of techniques you will need for an effective implementation. You will also note that all of these connectors extend a framework-provided output connector base class, found at <em>org.apache.manifoldcf.agents.output.BaseOutputConnector</em>. This base class furnishes some basic bookkeeping logic for managing the connector pool, as well as default implementations of some of the less typical functionality a connector may have. For example, connectors are allowed to have database tables of their own, which are instantiated when the connector is registered, and are torn down when the connector is removed. This is, however, not very typical, and the base implementation reflects that.</ p> + <p></p> + <section> + <title>Principle methods</title> + <p></p> + <p>The principle methods an implementer should be concerned with for creating an output connector are the following:</p> + <p></p> + <table> + <tr><th>Method</th><th>What it should do</th></tr> + <tr><td><strong>checkMimetypeIndexable()</strong></td><td>Decide whether a document with a given mime type is indexable or not</td></tr> + <tr><td><strong>checkDocumentIndexable()</strong></td><td>Decide whether a file is indexable or not</td></tr> + <tr><td><strong>checkLengthIndexable()</strong></td><td>Decide whether a document of a given length is indexable or not</td></tr> + <tr><td><strong>checkURLIndexable()</strong></td><td>Decide whether a document with a given URL is indexable or not</td></tr> + <tr><td><strong>getPipelineDescription()</strong></td><td>Use the supplied output specification to come up with an output version string</td></tr> + <tr><td><strong>addOrReplaceDocument()</strong></td><td>Add or replace the specified document within the target repository, or signal if the document cannot be handled</td></tr> + <tr><td><strong>removeDocument()</strong></td><td>Remove the specified document from the target repository</td></tr> + <tr><td><strong>outputConfigurationHeader()</strong></td><td>Output the head-section part of an output connection <em>ConfigParams</em> editing page</td></tr> + <tr><td><strong>outputConfigurationBody()</strong></td><td>Output the body-section part of an output connection <em>ConfigParams</em> editing page</td></tr> + <tr><td><strong>processConfigurationPost()</strong></td><td>Receive and process form data from an output connection <em>ConfigParams</em> editing page</td></tr> + <tr><td><strong>viewConfiguration()</strong></td><td>Output the viewing HTML for an output connection <em>ConfigParams</em> object</td></tr> + <tr><td><strong>outputSpecificationHeader()</strong></td><td>Output the head-section part of a <em>Specification</em> editing page</td></tr> + <tr><td><strong>outputSpecificationBody()</strong></td><td>Output the body-section part of a <em>Specification</em> editing page</td></tr> + <tr><td><strong>processSpecificationPost()</strong></td><td>Receive and process form data from a <em>Specification</em> editing page</td></tr> + <tr><td><strong>viewSpecification()</strong></td><td>Output the viewing page for a <em>Specification</em> object</td></tr> + </table> + <p></p> + <p>These methods come in three broad classes: (a) functional methods for doing the work of the connector; (b) UI methods for configuring a connection; and (c) UI methods for editing the output specification for a job. Together they do the heavy lifting of your connector. But before you can write any code at all, you need to plan things out a bit.</p> + <p></p> + </section> + <section> + <title>Choosing the form of the output version string</title> + <p></p> + <p>The output version string is used by ManifoldCF to determine whether or not the output specification or configuration changed in such a way as to require that the document be reprocessed. ManifoldCF therefore requests the output version string for any document that is ready for processing, and usually does not process the document again if the returned output version string agrees with the output version string it has stored.</p> + <p></p> + <p>Thinking about it more carefully, it is clear that what an output connector writer needs to do is include everything in the output version string that could potentially affect how the document gets ingested, save that which is specific to the repository connector. That may include bits of output connector configuration information, as well as data from the output specification. When it's time to ingest, it's usually the correct thing to do to obtain the necessary data for ingestion out of the output version string, rather than calculating it or fetching it anew, because that guarantees that the document processing was done in a manner that agrees with its recorded output version string, thus eliminating any chance of ManifoldCF getting confused.</p> + <p></p> + </section> + <section> + <title>Notes on connector UI methods</title> + <p></p> + <p>The crawler UI uses a tabbed layout structure, and thus each of the UI methods must properly implement the tabbed model. This means that the "header" methods above must add the desired tab names to a specified array, and the "body" methods must provide appropriate HTML which handles both the case where a tab is displayed, and where it is not displayed. Also, it makes sense to use the appropriate css definitions, so that the connector UI pages have a similar look-and-feel to the rest of ManifoldCF's crawler ui. We strongly suggest starting with one of the supplied connector's UI code, both for a description of the arguments to each method, and for some decent ideas of ways to organize your connector's UI code.</p> + <p></p> + <p>Please also note that it is good practice to name the form fields in your HTML in such a way that they cannot collide with form fields that may come from the framework's HTML or any specific repository connector's HTML. The <em>OutputSpecification</em> HTML especially may be prone to collisions, because within any given job, this HTML is included in the same page as HTML from the chosen repository connector.</p> + <p></p> + <p></p> + </section> + </section> + <section> + <title>Implementation support provided by the framework</title> + <p></p> + <p>ManifoldCF's framework provides a number of helpful services designed to make the creation of a connector easier. These services are summarized below. (This is not an exhaustive list, by any means.)</p> + <p></p> + <ul> + <li>Lock management and synchronization (see <em>org.apache.manifoldcf.core.interfaces.LockManagerFactory</em>)</li> + <li>Cache management (see <em>org.apache.manifoldcf.core.interfaces.CacheManagerFactory</em>)</li> + <li>Local keystore management (see <em>org.apache.manifoldcf.core.KeystoreManagerFactory</em>)</li> + <li>Database management (see <em>org.apache.manifoldcf.core.DBInterfaceFactory</em>)</li> + </ul> + <p></p> + <p>For UI method support, these too are very useful:</p> + <p></p> + <ul> + <li>Multipart form processing (see <em>org.apache.manifoldcf.ui.multipart.MultipartWrapper</em>)</li> + <li>HTML encoding (see <em>org.apache.manifoldcf.ui.util.Encoder</em>)</li> + <li>HTML formatting (see <em>org.apache.manifoldcf.ui.util.Formatter</em>)</li> + </ul> + <p></p> + </section> + <section> + <title>DO's and DON'T DO's</title> + <p></p> + <p>It's always a good idea to make use of an existing infrastructure component, if it's meant for that purpose, rather than inventing your own. There are, however, some limitations we recommend you adhere to.</p> + <p></p> + <ul> + <li>DO make use of infrastructure components described in the section above</li> + <li>DON'T make use of infrastructure components that aren't mentioned, without checking first</li> + <li>NEVER write connector code that directly uses framework database tables, other than the ones installed and managed by your connector</li> + </ul> + <p></p> + <p>If you are tempted to violate these rules, it may well mean you don't understand something important. At the very least, we'd like to know why. Send email to [email protected] with a description of your problem and how you are tempted to solve it.</p> + </section> + </section> + </body> +</document> \ No newline at end of file
