Modified: manifoldcf/site/publish/release/trunk/en_US/writing-repository-connectors.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/en_US/writing-repository-connectors.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/en_US/writing-repository-connectors.html (original) +++ manifoldcf/site/publish/release/trunk/en_US/writing-repository-connectors.html Thu Mar 21 21:37:09 2013 @@ -323,15 +323,17 @@ document.write("Last Published: " + docu <li>SharePoint (local keystore, web services)</li> -<li>RSS (local keystore, binning)</li> +<li>RSS (local keystore, binning, fuzzy xml parsing)</li> -<li>Web (local database schema, local keystore, binning, events and prerequisites, cache management)</li> +<li>Web (local database schema, local keystore, binning, events and prerequisites, cache management, fuzzy xml parsing)</li> + +<li>Wiki (binning, rigorous xml parsing)</li> </ul> <p></p> <p>You will also note that all of these connectors extend a framework-provided repository connector base class, found at <em>org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector</em>. This base class furnishes some basic bookkeeping logic for managing the connector pool, as well as default implementations of some of the less typical functionality a connector may have. For example, connectors are allowed to have database tables of their own, which are instantiated when the connector is registered, and are torn down when the connector is removed. This is, however, not very typical, and the base implementation reflects that.</p> <p></p> -<a name="N10103"></a><a name="Principle+methods"></a> +<a name="N10106"></a><a name="Principle+methods"></a> <h4>Principle methods</h4> <p></p> <p>The principle methods an implementer should be concerned with for creating a repository connector are the following:</p> @@ -390,7 +392,7 @@ document.write("Last Published: " + docu <p></p> <p>These methods come in three broad classes: (a) functional methods for doing the work of the connector; (b) UI methods for configuring a connection; and (c) UI methods for editing the document specification for a job. Together they do the heavy lifting of your connector. But before you can write any code at all, you need to plan things out a bit.</p> <p></p> -<a name="N101B6"></a><a name="Model"></a> +<a name="N101B9"></a><a name="Model"></a> <h4>Model</h4> <p></p> <p>Each connector must declare a specific model which it adheres to. These models basically describe what the <strong>addSeedDocuments()</strong> method actually does, and are described below.</p> @@ -402,6 +404,14 @@ document.write("Last Published: " + docu </tr> <tr> +<td colspan="1" rowspan="1"><em>MODEL_ALL</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method supplies all specified documents on each call</td> +</tr> + +<tr> +<td colspan="1" rowspan="1"><em>MODEL_PARTIAL</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> does not return a complete list of documents that match the criteria and time interval, because some of those documents are no longer discoverable</td> +</tr> + +<tr> <td colspan="1" rowspan="1"><em>MODEL_ADD</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method supplies at least all the matching documents that have been added to the repository, within the specified time interval</td> </tr> @@ -414,16 +424,29 @@ document.write("Last Published: " + docu </tr> <tr> -<td colspan="1" rowspan="1"><em>MODEL_PARTIAL</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> does not return a complete list of documents that match the criteria and time interval, because some of those documents are no longer discoverable</td> +<td colspan="1" rowspan="1"><em>MODEL_CHAINED_ADD</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method, plus documents reachable by discovery from seeds, supplies at least all the matching documents that have been added to the repository, within the specified time interval</td> +</tr> + +<tr> +<td colspan="1" rowspan="1"><em>MODEL_CHAINED_ADD_CHANGE</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method, plus documents reachable by discovery from seeds, supplies at least those matching documents that have been added or changed in the repository, within the specified time interval</td> +</tr> + +<tr> +<td colspan="1" rowspan="1"><em>MODEL_CHAINED_ADD_CHANGE_DELETE</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method, plus documents reachable by discovery from seeds, supplies at least those matching documents that have been added, changed, or removed in the repository, within the specified time interval</td> </tr> </table> <p></p> -<p>Note that the choice of model is actually much more subtle than the above description might indicate. It may, for one thing, be affected by characteristics of the repository, such as whether the repository considers a document to have been changed if its security information was changed. This would mean that, even though most document changes are picked up and thus one might be tempted to declare the connector to be <em>MODEL_ADD_CHANGE</em>, the correct choice would in fact be <em>MODEL_ADD</em>.</p> +<p>Note that the choice of model is actually much more subtle than the above description might indicate. It may, for one thing, be affected by characteristics of + the repository, such as whether the repository considers a document to have been changed if its security information was changed. This would mean that, + even though most document changes are picked up and thus one might be tempted to declare the connector to be <em>MODEL_ADD_CHANGE</em>, the + correct choice would in fact be <em>MODEL_ADD</em>.</p> +<p></p> +<p>Another subtle point is what documents the connector is actually supposed to return by means of the <strong>addSeedDocuments()</strong> method. The + start time and end time parameters handed to the method do not have to be strictly adhered to, for instance; it is always okay to return more documents. It is never + okay for the connector to return fewer documents than were requested, on the other hand.</p> <p></p> -<p>Another subtle point is what documents the connector is actually supposed to return by means of the <strong>addSeedDocuments()</strong> method. The start time and end time parameters handed to the method do not have to be strictly adhered to, for instance; it is always okay to return more documents. It is never okay for the connector to return fewer documents than were requested, on the other hand.</p> -<p></p> -<a name="N10221"></a><a name="Choosing+a+document+identifier+format"></a> +<a name="N1025C"></a><a name="Choosing+a+document+identifier+format"></a> <h4>Choosing a document identifier format</h4> <p></p> <p>In order to decide on the format for a document identifier, you need to understand what this identifier is used for, and what it represents. A document identifier usually corresponds to some entity within the source repository, such as a document or a folder. Note that there is <strong>no</strong> requirement that the identifier represent indexable content.</p> @@ -442,27 +465,43 @@ document.write("Last Published: " + docu <p></p> <p>We highly recommend that no additional information be included in the document identifier, other than what is needed for the above, as that will almost certainly cause problems.</p> <p></p> -<a name="N1024A"></a><a name="Choosing+the+form+of+the+document+version+string"></a> +<a name="N10285"></a><a name="Choosing+the+form+of+the+document+version+string"></a> <h4>Choosing the form of the document version string</h4> <p></p> -<p>The document version string is used by ManifoldCF to determine whether or not the document or configuration changed in such a way as to require that the document be reprocessed. ManifoldCF therefore requests the version string for any document that is ready for processing, and usually does not process the document again if the returned version string agrees with the version string it has stored.</p> -<p></p> -<p>Thinking about it more carefully, it is clear that what a connector writer needs to do is include everything in the version string that could potentially affect how the document gets processed. That may include the version of the document in the repository, bits of configuration information, metadata, and even access tokens (if the underlying repository versions these things independently from the document itself). Storing all of that information in the version string seems like a lot - but the string is unlimited in length, and it actually serves another useful purpose to do it that way. Specifically, when it comes time to do the actual processing, it's often the correct thing to do to obtain the necessary data out of the version string, rather than calculating it or fetching it anew. That way of working guarantees that the document processing was done in a manner that agrees with its recorded version string, thus eliminating any chance of ManifoldCF getting confused .</p> -<p></p> -<p>For longer data that needs to persist between the <strong>getDocumentVersions()</strong> method call and the <strong>processDocuments()</strong> method call, the connector is welcome to save this information in a temporary disk file. To help make sure nothing leaks which this approach is used, the IRepositoryConnector interface has a method that will be called to clean up any temporary files that might have been created in the handling of a given document identifier.</p> +<p>The document version string is used by ManifoldCF to determine whether or not the document or configuration changed in such a way as to require that the document + be reprocessed. ManifoldCF therefore requests the version string for any document that is ready for processing, and usually does not process the document again if the + returned version string agrees with the version string it has stored.</p> +<p></p> +<p>Thinking about it more carefully, it is clear that what a connector writer needs to do is include everything in the version string that could potentially affect how the + document gets processed. That may include the version of the document in the repository, bits of configuration information, metadata, and even access tokens (if the + underlying repository versions these things independently from the document itself). Storing all of that information in the version string seems like a lot - but the string + is unlimited in length, and it actually serves another useful purpose to do it that way. Specifically, when it comes time to do the actual processing, it's often the correct + thing to do to obtain the necessary data out of the version string, rather than calculating it or fetching it anew. That way of working guarantees that the document + processing was done in a manner that agrees with its recorded version string, thus eliminating any chance of ManifoldCF getting confused.</p> +<p></p> +<p>For longer data that needs to persist between the <strong>getDocumentVersions()</strong> method call and the <strong>processDocuments()</strong> method + call, the connector is welcome to save this information in a temporary disk file. To help make sure nothing leaks which this approach is used, the IRepositoryConnector + interface has a method that will be called to clean up any temporary files that might have been created in the handling of a given document identifier.</p> <p></p> -<a name="N10268"></a><a name="Notes+on+connector+UI+methods"></a> +<a name="N102A3"></a><a name="Notes+on+connector+UI+methods"></a> <h4>Notes on connector UI methods</h4> <p></p> -<p>The crawler UI uses a tabbed layout structure, and thus each of these elements must properly implement the tabbed model. This means that the "header" methods above must add the desired tab names to a specified array, and the "body" methods must provide appropriate HTML which handles both the case where a tab is displayed, and where it is not displayed. Also, it makes sense to use the appropriate css definitions, so that the connector UI pages have a similar look-and-feel to the rest of ManifoldCF's crawler ui. We strongly suggest starting with one of the supplied connector's UI code, both for a description of the arguments to each page, and for some decent ideas of ways to organize your connector's UI code. </p> +<p>The crawler UI uses a tabbed layout structure, and thus each of these elements must properly implement the tabbed model. This means that the "header" methods + above must add the desired tab names to a specified array, and the "body" methods must provide appropriate HTML which handles both the case where a tab is + displayed, and where it is not displayed. Also, it makes sense to use the appropriate css definitions, so that the connector UI pages have a similar look-and-feel to the + rest of ManifoldCF's crawler ui. We strongly suggest starting with one of the supplied connector's UI code, both for a description of the arguments to each page, and + for some decent ideas of ways to organize your connector's UI code. </p> +<p></p> +<p>Please also note that it is good practice to name the form fields in your HTML in such a way that they cannot collide with form fields that may come from the + framework's HTML or any specific output connector's HTML. The <em>DocumentSpecification</em> editing HTML especially may be prone to collisions, because + within any given job, this HTML is included in the same page as HTML from the chosen output connector.</p> <p></p> -<p>Please also note that it is good practice to name the form fields in your HTML in such a way that they cannot collide with form fields that may come from the framework's HTML or any specific output connector's HTML. The <em>DocumentSpecification</em> editing HTML especially may be prone to collisions, because within any given job, this HTML is included in the same page as HTML from the chosen output connector.</p> <p></p> -<p></p> -<a name="N10281"></a><a name="Implementation+support+provided+by+the+framework"></a> +<a name="N102BC"></a><a name="Implementation+support+provided+by+the+framework"></a> <h3 class="h4">Implementation support provided by the framework</h3> <p></p> -<p>ManifoldCF's framework provides a number of helpful services designed to make the creation of a connector easier. These services are summarized below. (This is not an exhaustive list, by any means.)</p> +<p>ManifoldCF's framework provides a number of helpful services designed to make the creation of a connector easier. These services are summarized below. + (This is not an exhaustive list, by any means.)</p> <p></p> <ul> @@ -488,10 +527,11 @@ document.write("Last Published: " + docu </ul> <p></p> -<a name="N102C8"></a><a name="DO%27s+and+DON%27T+DO%27s"></a> +<a name="N10303"></a><a name="DO%27s+and+DON%27T+DO%27s"></a> <h3 class="h4">DO's and DON'T DO's</h3> <p></p> -<p>It's always a good idea to make use of an existing infrastructure component, if it's meant for that purpose, rather than inventing your own. There are, however, some limitations we recommend you adhere to.</p> +<p>It's always a good idea to make use of an existing infrastructure component, if it's meant for that purpose, rather than inventing your own. There are, however, + some limitations we recommend you adhere to.</p> <p></p> <ul> @@ -503,7 +543,8 @@ document.write("Last Published: " + docu </ul> <p></p> -<p>If you are tempted to violate these rules, it may well mean you don't understand something important. At the very least, we'd like to know why. Send email to [email protected] with a description of your problem and how you are tempted to solve it.</p> +<p>If you are tempted to violate these rules, it may well mean you don't understand something important. At the very least, we'd like to know why. Send email + to [email protected] with a description of your problem and how you are tempted to solve it.</p> </div> </div> @@ -525,6 +566,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +-->
Modified: manifoldcf/site/publish/release/trunk/index.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/index.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/index.html (original) +++ manifoldcf/site/publish/release/trunk/index.html Thu Mar 21 21:37:09 2013 @@ -127,6 +127,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <div id="logos"></div> <!--+ |end bottomstrip Modified: manifoldcf/site/publish/release/trunk/ja_JP/concepts.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/concepts.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/concepts.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/concepts.html Thu Mar 21 21:37:09 2013 @@ -318,6 +318,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +--> Modified: manifoldcf/site/publish/release/trunk/ja_JP/end-user-documentation.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/end-user-documentation.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/end-user-documentation.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/end-user-documentation.html Thu Mar 21 21:37:09 2013 @@ -2128,6 +2128,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +--> Modified: manifoldcf/site/publish/release/trunk/ja_JP/end-user-documentation.pdf URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/end-user-documentation.pdf?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== Binary files - no diff available. Modified: manifoldcf/site/publish/release/trunk/ja_JP/faq.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/faq.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/faq.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/faq.html Thu Mar 21 21:37:09 2013 @@ -197,6 +197,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +--> Modified: manifoldcf/site/publish/release/trunk/ja_JP/included-connectors.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/included-connectors.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/included-connectors.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/included-connectors.html Thu Mar 21 21:37:09 2013 @@ -253,6 +253,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +--> Modified: manifoldcf/site/publish/release/trunk/ja_JP/index.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/index.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/index.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/index.html Thu Mar 21 21:37:09 2013 @@ -200,6 +200,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <div id="logos"></div> <!--+ |end bottomstrip Modified: manifoldcf/site/publish/release/trunk/ja_JP/javadoc.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/javadoc.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/javadoc.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/javadoc.html Thu Mar 21 21:37:09 2013 @@ -244,6 +244,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +--> Modified: manifoldcf/site/publish/release/trunk/ja_JP/technical-resources.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/technical-resources.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/technical-resources.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/technical-resources.html Thu Mar 21 21:37:09 2013 @@ -240,6 +240,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +--> Modified: manifoldcf/site/publish/release/trunk/ja_JP/writing-authority-connectors.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/writing-authority-connectors.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/writing-authority-connectors.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/writing-authority-connectors.html Thu Mar 21 21:37:09 2013 @@ -420,6 +420,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +--> Modified: manifoldcf/site/publish/release/trunk/ja_JP/writing-output-connectors.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/writing-output-connectors.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/writing-output-connectors.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/writing-output-connectors.html Thu Mar 21 21:37:09 2013 @@ -440,6 +440,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +--> Modified: manifoldcf/site/publish/release/trunk/ja_JP/writing-repository-connectors.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/ja_JP/writing-repository-connectors.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/ja_JP/writing-repository-connectors.html (original) +++ manifoldcf/site/publish/release/trunk/ja_JP/writing-repository-connectors.html Thu Mar 21 21:37:09 2013 @@ -323,15 +323,17 @@ document.write("Last Published: " + docu <li>SharePoint (local keystore, web services)</li> -<li>RSS (local keystore, binning)</li> +<li>RSS (local keystore, binning, fuzzy xml parsing)</li> -<li>Web (local database schema, local keystore, binning, events and prerequisites, cache management)</li> +<li>Web (local database schema, local keystore, binning, events and prerequisites, cache management, fuzzy xml parsing)</li> + +<li>Wiki (binning, rigorous xml parsing)</li> </ul> <p></p> <p>You will also note that all of these connectors extend a framework-provided repository connector base class, found at <em>org.apache.manifoldcf.crawler.connectors.BaseRepositoryConnector</em>. This base class furnishes some basic bookkeeping logic for managing the connector pool, as well as default implementations of some of the less typical functionality a connector may have. For example, connectors are allowed to have database tables of their own, which are instantiated when the connector is registered, and are torn down when the connector is removed. This is, however, not very typical, and the base implementation reflects that.</p> <p></p> -<a name="N10103"></a><a name="Principle+methods"></a> +<a name="N10106"></a><a name="Principle+methods"></a> <h4>Principle methods</h4> <p></p> <p>The principle methods an implementer should be concerned with for creating a repository connector are the following:</p> @@ -390,7 +392,7 @@ document.write("Last Published: " + docu <p></p> <p>These methods come in three broad classes: (a) functional methods for doing the work of the connector; (b) UI methods for configuring a connection; and (c) UI methods for editing the document specification for a job. Together they do the heavy lifting of your connector. But before you can write any code at all, you need to plan things out a bit.</p> <p></p> -<a name="N101B6"></a><a name="Model"></a> +<a name="N101B9"></a><a name="Model"></a> <h4>Model</h4> <p></p> <p>Each connector must declare a specific model which it adheres to. These models basically describe what the <strong>addSeedDocuments()</strong> method actually does, and are described below.</p> @@ -402,6 +404,14 @@ document.write("Last Published: " + docu </tr> <tr> +<td colspan="1" rowspan="1"><em>MODEL_ALL</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method supplies all specified documents on each call</td> +</tr> + +<tr> +<td colspan="1" rowspan="1"><em>MODEL_PARTIAL</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> does not return a complete list of documents that match the criteria and time interval, because some of those documents are no longer discoverable</td> +</tr> + +<tr> <td colspan="1" rowspan="1"><em>MODEL_ADD</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method supplies at least all the matching documents that have been added to the repository, within the specified time interval</td> </tr> @@ -414,16 +424,29 @@ document.write("Last Published: " + docu </tr> <tr> -<td colspan="1" rowspan="1"><em>MODEL_PARTIAL</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> does not return a complete list of documents that match the criteria and time interval, because some of those documents are no longer discoverable</td> +<td colspan="1" rowspan="1"><em>MODEL_CHAINED_ADD</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method, plus documents reachable by discovery from seeds, supplies at least all the matching documents that have been added to the repository, within the specified time interval</td> +</tr> + +<tr> +<td colspan="1" rowspan="1"><em>MODEL_CHAINED_ADD_CHANGE</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method, plus documents reachable by discovery from seeds, supplies at least those matching documents that have been added or changed in the repository, within the specified time interval</td> +</tr> + +<tr> +<td colspan="1" rowspan="1"><em>MODEL_CHAINED_ADD_CHANGE_DELETE</em></td><td colspan="1" rowspan="1">The <strong>addSeedDocuments()</strong> method, plus documents reachable by discovery from seeds, supplies at least those matching documents that have been added, changed, or removed in the repository, within the specified time interval</td> </tr> </table> <p></p> -<p>Note that the choice of model is actually much more subtle than the above description might indicate. It may, for one thing, be affected by characteristics of the repository, such as whether the repository considers a document to have been changed if its security information was changed. This would mean that, even though most document changes are picked up and thus one might be tempted to declare the connector to be <em>MODEL_ADD_CHANGE</em>, the correct choice would in fact be <em>MODEL_ADD</em>.</p> +<p>Note that the choice of model is actually much more subtle than the above description might indicate. It may, for one thing, be affected by characteristics of + the repository, such as whether the repository considers a document to have been changed if its security information was changed. This would mean that, + even though most document changes are picked up and thus one might be tempted to declare the connector to be <em>MODEL_ADD_CHANGE</em>, the + correct choice would in fact be <em>MODEL_ADD</em>.</p> +<p></p> +<p>Another subtle point is what documents the connector is actually supposed to return by means of the <strong>addSeedDocuments()</strong> method. The + start time and end time parameters handed to the method do not have to be strictly adhered to, for instance; it is always okay to return more documents. It is never + okay for the connector to return fewer documents than were requested, on the other hand.</p> <p></p> -<p>Another subtle point is what documents the connector is actually supposed to return by means of the <strong>addSeedDocuments()</strong> method. The start time and end time parameters handed to the method do not have to be strictly adhered to, for instance; it is always okay to return more documents. It is never okay for the connector to return fewer documents than were requested, on the other hand.</p> -<p></p> -<a name="N10221"></a><a name="Choosing+a+document+identifier+format"></a> +<a name="N1025C"></a><a name="Choosing+a+document+identifier+format"></a> <h4>Choosing a document identifier format</h4> <p></p> <p>In order to decide on the format for a document identifier, you need to understand what this identifier is used for, and what it represents. A document identifier usually corresponds to some entity within the source repository, such as a document or a folder. Note that there is <strong>no</strong> requirement that the identifier represent indexable content.</p> @@ -442,27 +465,43 @@ document.write("Last Published: " + docu <p></p> <p>We highly recommend that no additional information be included in the document identifier, other than what is needed for the above, as that will almost certainly cause problems.</p> <p></p> -<a name="N1024A"></a><a name="Choosing+the+form+of+the+document+version+string"></a> +<a name="N10285"></a><a name="Choosing+the+form+of+the+document+version+string"></a> <h4>Choosing the form of the document version string</h4> <p></p> -<p>The document version string is used by ManifoldCF to determine whether or not the document or configuration changed in such a way as to require that the document be reprocessed. ManifoldCF therefore requests the version string for any document that is ready for processing, and usually does not process the document again if the returned version string agrees with the version string it has stored.</p> -<p></p> -<p>Thinking about it more carefully, it is clear that what a connector writer needs to do is include everything in the version string that could potentially affect how the document gets processed. That may include the version of the document in the repository, bits of configuration information, metadata, and even access tokens (if the underlying repository versions these things independently from the document itself). Storing all of that information in the version string seems like a lot - but the string is unlimited in length, and it actually serves another useful purpose to do it that way. Specifically, when it comes time to do the actual processing, it's often the correct thing to do to obtain the necessary data out of the version string, rather than calculating it or fetching it anew. That way of working guarantees that the document processing was done in a manner that agrees with its recorded version string, thus eliminating any chance of ManifoldCF getting confused .</p> -<p></p> -<p>For longer data that needs to persist between the <strong>getDocumentVersions()</strong> method call and the <strong>processDocuments()</strong> method call, the connector is welcome to save this information in a temporary disk file. To help make sure nothing leaks which this approach is used, the IRepositoryConnector interface has a method that will be called to clean up any temporary files that might have been created in the handling of a given document identifier.</p> +<p>The document version string is used by ManifoldCF to determine whether or not the document or configuration changed in such a way as to require that the document + be reprocessed. ManifoldCF therefore requests the version string for any document that is ready for processing, and usually does not process the document again if the + returned version string agrees with the version string it has stored.</p> +<p></p> +<p>Thinking about it more carefully, it is clear that what a connector writer needs to do is include everything in the version string that could potentially affect how the + document gets processed. That may include the version of the document in the repository, bits of configuration information, metadata, and even access tokens (if the + underlying repository versions these things independently from the document itself). Storing all of that information in the version string seems like a lot - but the string + is unlimited in length, and it actually serves another useful purpose to do it that way. Specifically, when it comes time to do the actual processing, it's often the correct + thing to do to obtain the necessary data out of the version string, rather than calculating it or fetching it anew. That way of working guarantees that the document + processing was done in a manner that agrees with its recorded version string, thus eliminating any chance of ManifoldCF getting confused.</p> +<p></p> +<p>For longer data that needs to persist between the <strong>getDocumentVersions()</strong> method call and the <strong>processDocuments()</strong> method + call, the connector is welcome to save this information in a temporary disk file. To help make sure nothing leaks which this approach is used, the IRepositoryConnector + interface has a method that will be called to clean up any temporary files that might have been created in the handling of a given document identifier.</p> <p></p> -<a name="N10268"></a><a name="Notes+on+connector+UI+methods"></a> +<a name="N102A3"></a><a name="Notes+on+connector+UI+methods"></a> <h4>Notes on connector UI methods</h4> <p></p> -<p>The crawler UI uses a tabbed layout structure, and thus each of these elements must properly implement the tabbed model. This means that the "header" methods above must add the desired tab names to a specified array, and the "body" methods must provide appropriate HTML which handles both the case where a tab is displayed, and where it is not displayed. Also, it makes sense to use the appropriate css definitions, so that the connector UI pages have a similar look-and-feel to the rest of ManifoldCF's crawler ui. We strongly suggest starting with one of the supplied connector's UI code, both for a description of the arguments to each page, and for some decent ideas of ways to organize your connector's UI code. </p> +<p>The crawler UI uses a tabbed layout structure, and thus each of these elements must properly implement the tabbed model. This means that the "header" methods + above must add the desired tab names to a specified array, and the "body" methods must provide appropriate HTML which handles both the case where a tab is + displayed, and where it is not displayed. Also, it makes sense to use the appropriate css definitions, so that the connector UI pages have a similar look-and-feel to the + rest of ManifoldCF's crawler ui. We strongly suggest starting with one of the supplied connector's UI code, both for a description of the arguments to each page, and + for some decent ideas of ways to organize your connector's UI code. </p> +<p></p> +<p>Please also note that it is good practice to name the form fields in your HTML in such a way that they cannot collide with form fields that may come from the + framework's HTML or any specific output connector's HTML. The <em>DocumentSpecification</em> editing HTML especially may be prone to collisions, because + within any given job, this HTML is included in the same page as HTML from the chosen output connector.</p> <p></p> -<p>Please also note that it is good practice to name the form fields in your HTML in such a way that they cannot collide with form fields that may come from the framework's HTML or any specific output connector's HTML. The <em>DocumentSpecification</em> editing HTML especially may be prone to collisions, because within any given job, this HTML is included in the same page as HTML from the chosen output connector.</p> <p></p> -<p></p> -<a name="N10281"></a><a name="Implementation+support+provided+by+the+framework"></a> +<a name="N102BC"></a><a name="Implementation+support+provided+by+the+framework"></a> <h3 class="h4">Implementation support provided by the framework</h3> <p></p> -<p>ManifoldCF's framework provides a number of helpful services designed to make the creation of a connector easier. These services are summarized below. (This is not an exhaustive list, by any means.)</p> +<p>ManifoldCF's framework provides a number of helpful services designed to make the creation of a connector easier. These services are summarized below. + (This is not an exhaustive list, by any means.)</p> <p></p> <ul> @@ -488,10 +527,11 @@ document.write("Last Published: " + docu </ul> <p></p> -<a name="N102C8"></a><a name="DO%27s+and+DON%27T+DO%27s"></a> +<a name="N10303"></a><a name="DO%27s+and+DON%27T+DO%27s"></a> <h3 class="h4">DO's and DON'T DO's</h3> <p></p> -<p>It's always a good idea to make use of an existing infrastructure component, if it's meant for that purpose, rather than inventing your own. There are, however, some limitations we recommend you adhere to.</p> +<p>It's always a good idea to make use of an existing infrastructure component, if it's meant for that purpose, rather than inventing your own. There are, however, + some limitations we recommend you adhere to.</p> <p></p> <ul> @@ -503,7 +543,8 @@ document.write("Last Published: " + docu </ul> <p></p> -<p>If you are tempted to violate these rules, it may well mean you don't understand something important. At the very least, we'd like to know why. Send email to [email protected] with a description of your problem and how you are tempted to solve it.</p> +<p>If you are tempted to violate these rules, it may well mean you don't understand something important. At the very least, we'd like to know why. Send email + to [email protected] with a description of your problem and how you are tempted to solve it.</p> </div> </div> @@ -525,6 +566,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +--> Modified: manifoldcf/site/publish/release/trunk/linkmap.html URL: http://svn.apache.org/viewvc/manifoldcf/site/publish/release/trunk/linkmap.html?rev=1459538&r1=1459537&r2=1459538&view=diff ============================================================================== --- manifoldcf/site/publish/release/trunk/linkmap.html (original) +++ manifoldcf/site/publish/release/trunk/linkmap.html Thu Mar 21 21:37:09 2013 @@ -396,6 +396,17 @@ document.write("Last Published: " + docu Copyright © 2009-2013 <a href="http://www.apache.org/licenses/";>The Apache Software Foundation.</a> </div> +<div class="copyright"> + Apache ManifoldCF, ManifoldCF, Apache Forrest, Forrest, Apache Solr, Solr, Apache, the Apache feather logo, the Apache Forrest + logo, and the Apache ManifoldCF logo are trademarks of The Apache Software Foundation. + Documentum and EMC are a trademarks of EMC Corporation. + SharePoint, Windows, and Microsoft are trademarks of Microsoft, Inc. + FileNet P8 and IBM are trademarks of IBM, Inc. + LiveLink and OpenText are trademarks of OpenText, Inc. + QBase, MetaCarta, and GTS are trademarks of QBase, Inc. + Meridio and Autonomy are trademarks of Hewlett Packard, Inc. + Alfresco is a trademark of Alfresco Software, Inc. + </div> <!--+ |end bottomstrip +-->
