Author: kwright
Date: Sun Jun 30 14:06:40 2013
New Revision: 1498126
URL: http://svn.apache.org/r1498126
Log:
Fix for CONNECTORS-740.
Added:
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/ParameterEnum.java
(with props)
manifoldcf/trunk/site/src/documentation/resources/images/en_US/filesystem-job-output-path.PNG
(with props)
Modified:
manifoldcf/trunk/CHANGES.txt
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/FileOutputParam.java
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/filesystem/FileConnector.java
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_en_US.properties
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_ja_JP.properties
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_en_US.properties
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_ja_JP.properties
manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.html
manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.js
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
manifoldcf/trunk/site/src/documentation/resources/images/en_US/filesystem-job-paths.PNG
Modified: manifoldcf/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/CHANGES.txt?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
--- manifoldcf/trunk/CHANGES.txt (original)
+++ manifoldcf/trunk/CHANGES.txt Sun Jun 30 14:06:40 2013
@@ -3,6 +3,11 @@ $Id$
======================= 1.3-dev =====================
+CONNECTORS-740: Add documentation for filesystem connector changes
+and for filesystem output connector. Also refactor filesystem connector
+for better UI.
+(Karl Wright)
+
CONNECTORS-738: Update how-to-build-and-deploy to reflect new
connectors.
(Karl Wright)
Modified:
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/FileOutputParam.java
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/FileOutputParam.java?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/FileOutputParam.java
(original)
+++
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/FileOutputParam.java
Sun Jun 30 14:06:40 2013
@@ -22,27 +22,14 @@ package org.apache.manifoldcf.agents.out
import java.util.HashMap;
import java.util.Map;
-import
org.apache.manifoldcf.agents.output.filesystem.FileOutputParam.ParameterEnum;
-
/**
* Parameters data for the elasticsearch output connector.
*/
public class FileOutputParam extends HashMap<ParameterEnum, String>
{
-
- /** Parameters constants */
- public enum ParameterEnum {
- ROOTPATH("/tmp");
-
- final protected String defaultValue;
-
- private ParameterEnum(String defaultValue) {
- this.defaultValue = defaultValue;
- }
- }
-
private static final long serialVersionUID = -140994685772720029L;
+
protected FileOutputParam(ParameterEnum[] params) {
super(params.length);
}
Added:
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/ParameterEnum.java
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/ParameterEnum.java?rev=1498126&view=auto
==============================================================================
---
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/ParameterEnum.java
(added)
+++
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/ParameterEnum.java
Sun Jun 30 14:06:40 2013
@@ -0,0 +1,31 @@
+/* $Id$ */
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.manifoldcf.agents.output.filesystem;
+
+/** Parameters constants */
+public enum ParameterEnum {
+ ROOTPATH("");
+
+ final protected String defaultValue;
+
+ private ParameterEnum(String defaultValue) {
+ this.defaultValue = defaultValue;
+ }
+}
Propchange:
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/ParameterEnum.java
------------------------------------------------------------------------------
svn:eol-style = native
Propchange:
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/agents/output/filesystem/ParameterEnum.java
------------------------------------------------------------------------------
svn:keywords = Id
Modified:
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/filesystem/FileConnector.java
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/filesystem/FileConnector.java?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/filesystem/FileConnector.java
(original)
+++
manifoldcf/trunk/connectors/filesystem/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/filesystem/FileConnector.java
Sun Jun 30 14:06:40 2013
@@ -111,52 +111,39 @@ public class FileConnector extends org.a
*@param repositoryPath is the document repositoryPath.
*@return the document uri.
*/
- protected String convertToURI(String documentIdentifier, String[]
repositoryPaths)
+ protected static String convertToWGETURI(String path)
throws ManifoldCFException
{
//
// Note well: This MUST be a legal URI!!!
try
{
- String path = new File(documentIdentifier).getAbsolutePath();
- for (String repositoryPath : repositoryPaths) {
- if (path.startsWith(repositoryPath)) {
- StringBuffer sb = new StringBuffer();
- path = path.replaceFirst(repositoryPath, "");
- if (path.startsWith("/")) {
- path = path.replaceFirst("/", "");
- }
- String[] tmp = path.split("/", 3);
- String scheme = "";
- String host = "";
- String other = "";
- try {
- scheme = tmp[0];
- } catch (ArrayIndexOutOfBoundsException e) {
- scheme = "http";
- }
- try {
- host = tmp[1];
- } catch (ArrayIndexOutOfBoundsException e) {
- host = "localhost";
- }
- try {
- other = "/" + tmp[2];
- } catch (ArrayIndexOutOfBoundsException e) {
- other = "/";
- }
- return new URI(scheme + "://" + host + other).toURL().toString();
- }
- }
- return convertToURI(documentIdentifier);
+ StringBuffer sb = new StringBuffer();
+ String[] tmp = path.split("/", 3);
+ String scheme = "";
+ String host = "";
+ String other = "";
+ if (tmp.length >= 1)
+ scheme = tmp[0];
+ else
+ scheme = "http";
+ if (tmp.length >= 2)
+ host = tmp[1];
+ else
+ host = "localhost";
+ if (tmp.length >= 3)
+ other = "/" + tmp[2];
+ else
+ other = "/";
+ return new URI(scheme + "://" + host + other).toURL().toString();
}
- catch (URISyntaxException e)
+ catch (java.net.MalformedURLException e)
{
- throw new ManifoldCFException("Bad url",e);
+ throw new ManifoldCFException("Bad url: "+e.getMessage(),e);
}
- catch (IOException e)
+ catch (URISyntaxException e)
{
- throw new ManifoldCFException("Bad url",e);
+ throw new ManifoldCFException("Bad url: "+e.getMessage(),e);
}
}
@@ -165,7 +152,7 @@ public class FileConnector extends org.a
*@param documentIdentifier is the document identifier.
*@return the document uri.
*/
- protected String convertToURI(String documentIdentifier)
+ protected static String convertToURI(String documentIdentifier)
throws ManifoldCFException
{
//
@@ -225,23 +212,12 @@ public class FileConnector extends org.a
{
int i = 0;
- /*
- * get filepathtouri value
- */
- boolean filePathToUri = false;
- i = 0;
- while (i < spec.getChildCount()) {
- SpecificationNode sn = spec.getChild(i++);
- if (sn.getType().equals("filepathtouri")) {
- filePathToUri = Boolean.valueOf(sn.getValue());
- }
- }
-
String[] rval = new String[documentIdentifiers.length];
i = 0;
while (i < rval.length)
{
- File file = new File(documentIdentifiers[i]);
+ String documentIdentifier = documentIdentifiers[i];
+ File file = new File(documentIdentifier);
if (file.exists())
{
if (file.isDirectory())
@@ -262,9 +238,17 @@ public class FileConnector extends org.a
{
// Get the file's modified date.
long lastModified = file.lastModified();
+
+ // Check if the path is to be converted. We record that info in
the version string so that we'll reindex documents whose
+ // URI's change.
+ String convertPath = findConvertPath(spec, file);
StringBuilder sb = new StringBuilder();
- if (filePathToUri)
+ if (convertPath != null)
+ {
+ // Record the path.
sb.append("+");
+ pack(sb,convertPath,'+');
+ }
else
sb.append("-");
sb.append(new
Long(lastModified).toString()).append(":").append(new
Long(fileLength).toString());
@@ -349,30 +333,17 @@ public class FileConnector extends org.a
// We still need to check based on file data.
if (checkIngest(file,spec))
{
- int j = 0;
-
- /*
- * get repository paths
- */
- j = 0;
- List<String> repositoryPaths = new ArrayList<String>();
- while ( j < spec.getChildCount())
- {
- SpecificationNode sn = spec.getChild(j++);
- if (sn.getType().equals("startpoint"))
- {
- if (sn.getAttributeValue("path").length() > 0) {
- repositoryPaths.add(sn.getAttributeValue("path"));
- }
- }
- }
/*
* get filepathtouri value
*/
- boolean filePathToUri = false;
+ String convertPath = null;
if (version.length() > 0 && version.startsWith("+"))
- filePathToUri = true;
+ {
+ StringBuilder unpack = new StringBuilder();
+ unpack(unpack, version, 1, '+');
+ convertPath = unpack.toString();
+ }
long startTime = System.currentTimeMillis();
String errorCode = "OK";
@@ -394,15 +365,17 @@ public class FileConnector extends org.a
data.setFileName(fileName);
data.setMimeType(mapExtensionToMimeType(fileName));
data.setModifiedDate(new Date(file.lastModified()));
- if (filePathToUri) {
-
data.addField("uri",convertToURI(documentIdentifier,repositoryPaths.toArray(new
String[0])));
- // MHL for other metadata
-
activities.ingestDocument(documentIdentifier,version,convertToURI(documentIdentifier,repositoryPaths.toArray(new
String[0])),data);
+ String uri;
+ if (convertPath != null) {
+ // WGET-compatible input; convert back to external URI
+ uri = convertToWGETURI(convertPath);
+ data.addField("uri",uri);
} else {
+ uri = convertToURI(documentIdentifier);
data.addField("uri",file.toString());
- // MHL for other metadata
-
activities.ingestDocument(documentIdentifier,version,convertToURI(documentIdentifier),data);
}
+ // MHL for other metadata
+
activities.ingestDocument(documentIdentifier,version,uri,data);
fileLength = new Long(fileBytes);
}
finally
@@ -429,6 +402,34 @@ public class FileConnector extends org.a
}
}
+ /** This method finds the part of the path that should be converted to a URI.
+ * Returns null if the path should not be converted.
+ *@param spec is the document specification.
+ *@param documentIdentifier is the document identifier.
+ *@return the part of the path to be converted, or null.
+ */
+ protected static String findConvertPath(DocumentSpecification spec, File
theFile)
+ {
+ String fullpath = theFile.getAbsolutePath().replaceAll("\\","/");
+ for (int j = 0; j < spec.getChildCount(); j++)
+ {
+ SpecificationNode sn = spec.getChild(j);
+ if (sn.getType().equals("startpoint"))
+ {
+ String path = sn.getAttributeValue("path").replaceAll("\\","/");
+ String convertToURI = sn.getAttributeValue("converttouri");
+ if (path.length() > 0 && convertToURI != null &&
convertToURI.equals("true"))
+ {
+ if (!path.endsWith("/"))
+ path += "/";
+ if (fullpath.startsWith(path))
+ return fullpath.substring(path.length());
+ }
+ }
+ }
+ return null;
+ }
+
/** Map an extension to a mime type */
protected static String mapExtensionToMimeType(String fileName)
{
@@ -529,7 +530,6 @@ public class FileConnector extends org.a
throws ManifoldCFException, IOException
{
tabsArray.add(Messages.getString(locale,"FileConnector.Paths"));
- tabsArray.add(Messages.getString(locale,"FileConnector.FilePathToURITab"));
out.print(
"<script type=\"text/javascript\">\n"+
@@ -578,6 +578,7 @@ public class FileConnector extends org.a
" <tr class=\"formheaderrow\">\n"+
" <td class=\"formcolumnheader\"></td>\n"+
" <td class=\"formcolumnheader\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.RootPath") + "</nobr></td>\n"+
+" <td class=\"formcolumnheader\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.ConvertToURI") + "<br/>" +
Messages.getBodyString(locale,"FileConnector.ConvertToURIExample")+
"</nobr></td>\n"+
" <td class=\"formcolumnheader\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.Rules") + "</nobr></td>\n"+
" </tr>\n"
);
@@ -590,6 +591,14 @@ public class FileConnector extends org.a
{
String pathDescription = "_"+Integer.toString(k);
String pathOpName = "specop"+pathDescription;
+
+ String path = sn.getAttributeValue("path");
+ String convertToURIString = sn.getAttributeValue("converttouri");
+
+ boolean convertToURI = false;
+ if (convertToURIString != null && convertToURIString.equals("true"))
+ convertToURI = true;
+
out.print(
" <tr class=\""+(((k % 2)==0)?"evenformrow":"oddformrow")+"\">\n"+
" <td class=\"formcolumncell\">\n"+
@@ -601,7 +610,13 @@ public class FileConnector extends org.a
" </td>\n"+
" <td class=\"formcolumncell\">\n"+
" <nobr>\n"+
-"
"+org.apache.manifoldcf.ui.util.Encoder.bodyEscape(sn.getAttributeValue("path"))+"
\n"+
+" "+org.apache.manifoldcf.ui.util.Encoder.bodyEscape(path)+" \n"+
+" </nobr>\n"+
+" </td>\n"+
+" <td class=\"formcolumncell\">\n"+
+" <input type=\"hidden\" name=\"converttouri"+pathDescription+"\"
value=\""+(convertToURI?"true":"false")+"\">\n"+
+" <nobr>\n"+
+"
"+(convertToURI?Messages.getBodyString(locale,"FileConnector.Yes"):Messages.getBodyString(locale,"FileConnector.No"))+"
\n"+
" </nobr>\n"+
" </td>\n"+
" <td class=\"boxcell\">\n"+
@@ -730,11 +745,11 @@ public class FileConnector extends org.a
if (k == 0)
{
out.print(
-" <tr class=\"formrow\"><td class=\"message\" colspan=\"3\">" +
Messages.getBodyString(locale,"FileConnector.NoDocumentsSpecified") +
"</td></tr>\n"
+" <tr class=\"formrow\"><td class=\"formcolumnmessage\" colspan=\"4\">"
+ Messages.getBodyString(locale,"FileConnector.NoDocumentsSpecified") +
"</td></tr>\n"
);
}
out.print(
-" <tr class=\"formrow\"><td class=\"lightseparator\"
colspan=\"3\"><hr/></td></tr>\n"+
+" <tr class=\"formrow\"><td class=\"lightseparator\"
colspan=\"4\"><hr/></td></tr>\n"+
" <tr class=\"formrow\">\n"+
" <td class=\"formcolumncell\">\n"+
" <nobr>\n"+
@@ -747,7 +762,12 @@ public class FileConnector extends org.a
" </td>\n"+
" <td class=\"formcolumncell\">\n"+
" <nobr>\n"+
-" <input type=\"text\" size=\"80\" name=\"specpath\"
value=\"\"/>\n"+
+" <input type=\"text\" size=\"30\" name=\"specpath\"
value=\"\"/>\n"+
+" </nobr>\n"+
+" </td>\n"+
+" <td class=\"formcolumncell\">\n"+
+" <nobr>\n"+
+" <input name=\"converttouri\" type=\"checkbox\"
value=\"true\"/>\n"+
" </nobr>\n"+
" </td>\n"+
" <td class=\"formcolumncell\">\n"+
@@ -769,8 +789,17 @@ public class FileConnector extends org.a
if (sn.getType().equals("startpoint"))
{
String pathDescription = "_"+Integer.toString(k);
+
+ String path = sn.getAttributeValue("path");
+ String convertToURIString = sn.getAttributeValue("converttouri");
+
+ boolean convertToURI = false;
+ if (convertToURIString != null && convertToURIString.equals("true"))
+ convertToURI = true;
+
out.print(
-"<input type=\"hidden\" name=\"specpath"+pathDescription+"\"
value=\""+org.apache.manifoldcf.ui.util.Encoder.attributeEscape(sn.getAttributeValue("path"))+"\"/>\n"+
+"<input type=\"hidden\" name=\"specpath"+pathDescription+"\"
value=\""+org.apache.manifoldcf.ui.util.Encoder.attributeEscape(path)+"\"/>\n"+
+"<input type=\"hidden\" name=\"converttouri"+pathDescription+"\"
value=\""+(convertToURI?"true":"false")+"\">\n"+
"<input type=\"hidden\" name=\"specchildcount"+pathDescription+"\"
value=\""+Integer.toString(sn.getChildCount())+"\"/>\n"
);
@@ -798,43 +827,6 @@ public class FileConnector extends org.a
);
}
-
- /*
- * get filepathtouri value
- */
- boolean filePathToUri = false;
- i = 0;
- while (i < ds.getChildCount()) {
- SpecificationNode sn = ds.getChild(i++);
- if (sn.getType().equals("filepathtouri")) {
- filePathToUri = Boolean.valueOf(sn.getValue());
- }
- }
-
- /*
- * File path to URI tab
- */
- if
(tabName.equals(Messages.getString(locale,"FileConnector.FilePathToURITab"))) {
- out.print(
-"<table class=\"displaytable\">\n"+
-" <tr><td colspan=\"2\" class=\"separator\"><hr/></td></tr>\n"+
-" <tr>\n"+
-" <td class=\"description\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.FilePathToURI") + "</nobr></td>\n"+
-" <td class=\"value\">\n"+
-" <input name=\"filepathtouri\" type=\"checkbox\" value=\"true\"" +
(filePathToUri ? "checked" : "") +"/> " +
Messages.getBodyString(locale,"FileConnector.FilePathToURIExample") + "\n" +
-" </td>\n"+
-" </tr>\n"+
-"</table>\n"
- );
- } else {
- /*
- * File path to URI tab hiddens
- */
- out.print(
-"<input type=\"hidden\" name=\"filepathtouri\" value=\"" +
org.apache.manifoldcf.ui.util.Encoder.attributeEscape(Boolean.toString(filePathToUri))
+ "\"/>\n"
- );
- }
-
}
/** Process a specification post.
@@ -871,8 +863,13 @@ public class FileConnector extends org.a
}
// Path inserts won't happen until the very end
String path = variableContext.getParameter("specpath"+pathDescription);
+ String convertToURI =
variableContext.getParameter("converttouri"+pathDescription);
+
SpecificationNode node = new SpecificationNode("startpoint");
node.setAttribute("path",path);
+ if (convertToURI != null)
+ node.setAttribute("converttouri",convertToURI);
+
// Now, get the number of children
String y =
variableContext.getParameter("specchildcount"+pathDescription);
int childCount = Integer.parseInt(y);
@@ -934,9 +931,13 @@ public class FileConnector extends org.a
if (op != null && op.equals("Add"))
{
String path = variableContext.getParameter("specpath");
+ String convertToURI = variableContext.getParameter("converttouri");
+
SpecificationNode node = new SpecificationNode("startpoint");
node.setAttribute("path",path);
-
+ if (convertToURI != null)
+ node.setAttribute("converttouri",convertToURI);
+
// Now add in the defaults; these will be "include all directories"
and "include all files".
SpecificationNode sn = new SpecificationNode("include");
sn.setAttribute("type","file");
@@ -951,25 +952,6 @@ public class FileConnector extends org.a
}
}
- /*
- * "filepathtouri"
- */
- String filepathtouri = variableContext.getParameter("filepathtouri");
- if (filepathtouri != null) {
- SpecificationNode sn;
- int i = 0;
- while (i < ds.getChildCount()) {
- if (ds.getChild(i).getType().equals("filepathtouri")) {
- ds.removeChild(i);
- } else {
- i++;
- }
- }
- sn = new SpecificationNode("filepathtouri");
- sn.setValue(filepathtouri);
- ds.addChild(ds.getChildCount(),sn);
- }
-
return null;
}
@@ -983,81 +965,120 @@ public class FileConnector extends org.a
public void viewSpecification(IHTTPOutput out, Locale locale,
DocumentSpecification ds)
throws ManifoldCFException, IOException
{
- int i = 0;
-
out.print(
"<table class=\"displaytable\">\n"+
" <tr>\n"+
-" <td colspan=\"2\" class=\"message\">" +
Messages.getAttributeString(locale,"FileConnector.Paths") + "</td>\n"+
-" </tr>\n"
+" <td class=\"description\">" +
Messages.getAttributeString(locale,"FileConnector.Paths2") + "</td>\n"+
+" <td class=\"boxcell\">\n"+
+" <table class=\"formtable\">\n"+
+" <tr class=\"formheaderrow\">\n"+
+" <td class=\"formcolumnheader\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.RootPath") + "</nobr></td>\n"+
+" <td class=\"formcolumnheader\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.ConvertToURI") + "<br/>" +
Messages.getBodyString(locale,"FileConnector.ConvertToURIExample")+
"</nobr></td>\n"+
+" <td class=\"formcolumnheader\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.Rules") + "</nobr></td>\n"+
+" </tr>\n"
);
-
- i = 0;
- boolean seenAny = false;
- while (i < ds.getChildCount())
+
+ int k = 0;
+ for (int i = 0; i < ds.getChildCount(); i++)
{
- SpecificationNode sn = ds.getChild(i++);
+ SpecificationNode sn = ds.getChild(i);
if (sn.getType().equals("startpoint"))
{
- if (seenAny == false)
- {
- seenAny = true;
- }
+ String path = sn.getAttributeValue("path");
+ String convertToURIString = sn.getAttributeValue("converttouri");
+ boolean convertToURI = false;
+ if (convertToURIString != null && convertToURIString.equals("true"))
+ convertToURI = true;
+
out.print(
-" <tr>\n"+
-" <td
class=\"description\">"+org.apache.manifoldcf.ui.util.Encoder.bodyEscape(sn.getAttributeValue("path"))+":"+"</td>\n"+
-" <td class=\"value\">\n"
+" <tr class=\""+(((k % 2)==0)?"evenformrow":"oddformrow")+"\">\n"+
+" <td class=\"formcolumncell\">\n"+
+" <nobr>\n"+
+" "+org.apache.manifoldcf.ui.util.Encoder.bodyEscape(path)+" \n"+
+" </nobr>\n"+
+" </td>\n"+
+" <td class=\"formcolumncell\">\n"+
+" <nobr>\n"+
+"
"+(convertToURI?Messages.getBodyString(locale,"FileConnector.Yes"):Messages.getBodyString(locale,"FileConnector.No"))+"
\n"+
+" </nobr>\n"+
+" </td>\n"+
+" <td class=\"boxcell\">\n"+
+" <table class=\"formtable\">\n"+
+" <tr class=\"formheaderrow\">\n"+
+" <td class=\"formcolumnheader\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.IncludeExclude") +
"</nobr></td>\n"+
+" <td class=\"formcolumnheader\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.FileDirectory") + "</nobr></td>\n"+
+" <td class=\"formcolumnheader\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.Match") + "</nobr></td>\n"+
+" </tr>\n"
);
- int j = 0;
- while (j < sn.getChildCount())
+
+ int l = 0;
+ for (int j = 0; j < sn.getChildCount(); j++)
{
- SpecificationNode excludeNode = sn.getChild(j++);
+ SpecificationNode excludeNode = sn.getChild(j);
+
+ String nodeFlavor = excludeNode.getType();
+ String nodeType = excludeNode.getAttributeValue("type");
+ String nodeMatch = excludeNode.getAttributeValue("match");
out.print(
-" "+(excludeNode.getType().equals("include")?"Include ":"")+"\n"+
-" "+(excludeNode.getType().equals("exclude")?"Exclude ":"")+"\n"+
-" "+(excludeNode.getAttributeValue("type").equals("file")?"file
":"")+"\n"+
-" "+(excludeNode.getAttributeValue("type").equals("directory")?"directory
":"")+"\n"+
-"
"+org.apache.manifoldcf.ui.util.Encoder.bodyEscape(excludeNode.getAttributeValue("match"))+"<br/>\n"
+" <tr class=\""+(((l % 2)==0)?"evenformrow":"oddformrow")+"\">\n"+
+" <td class=\"formcolumncell\">\n"+
+" <nobr>\n"+
+" "+nodeFlavor+"\n"+
+" </nobr>\n"+
+" </td>\n"+
+" <td class=\"formcolumncell\">\n"+
+" <nobr>\n"+
+" "+nodeType+"\n"+
+" </nobr>\n"+
+" </td>\n"+
+" <td class=\"formcolumncell\">\n"+
+" <nobr>\n"+
+"
"+org.apache.manifoldcf.ui.util.Encoder.bodyEscape(nodeMatch)+"\n"+
+" </nobr>\n"+
+" </td>\n"+
+" </tr>\n"
+ );
+ l++;
+ }
+
+ if (l == 0)
+ {
+ out.print(
+" <tr><td class=\"formcolumnmessage\" colspan=\"3\">" +
Messages.getBodyString(locale,"FileConnector.NoRulesDefined") + "</td></tr>\n"
);
}
+
out.print(
-" </td>\n"+
-" </tr>\n"
+" </table>\n"+
+" </td>\n"
);
+
+ out.print(
+" </tr>\n"
+ );
+
+ k++;
}
+
}
- if (seenAny == false)
+
+ if (k == 0)
{
out.print(
-" <tr><td class=\"message\">" +
Messages.getBodyString(locale,"FileConnector.NoDocumentsSpecified") +
"</td></tr>\n"
+" <tr><td class=\"formcolumnmessage\" colspan=\"3\">" +
Messages.getBodyString(locale,"FileConnector.NoDocumentsSpecified") +
"</td></tr>\n"
);
}
+
out.print(
-"</table>\n"
+" </table>\n"+
+" </td>\n"+
+" </tr>\n"
);
-
- /*
- * get filepathtouri value
- */
- boolean filePathToUri = false;
- i = 0;
- while (i < ds.getChildCount()) {
- SpecificationNode sn = ds.getChild(i++);
- if (sn.getType().equals("filepathtouri")) {
- filePathToUri = Boolean.valueOf(sn.getValue());
- }
- }
out.print(
-"<table class=\"displaytable\">\n"+
-" <tr><td colspan=\"2\" class=\"separator\"><hr/></td></tr>\n"+
-" <tr>\n"+
-" <td class=\"description\"><nobr>" +
Messages.getBodyString(locale,"FileConnector.FilePathToURI") + "</nobr></td>\n"+
-" <td class=\"value\">" +
org.apache.manifoldcf.ui.util.Encoder.bodyEscape(Boolean.toString(filePathToUri))
+ "</td>\n"+
-" </tr>\n"+
"</table>\n"
);
-
+
}
// Protected static methods
Modified:
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_en_US.properties
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_en_US.properties?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_en_US.properties
(original)
+++
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_en_US.properties
Sun Jun 30 14:06:40 2013
@@ -14,6 +14,5 @@
# limitations under the License.
FileConnector.PathTabName=Output Path
-FileConnector.Path=Output Path
-FileConnector.RootPath=Root path
-
+FileConnector.RootPath=Root path:
+FileConnector.RootPathCannotBeNull=Root path cannot be null
Modified:
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_ja_JP.properties
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_ja_JP.properties?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_ja_JP.properties
(original)
+++
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/agents/output/filesystem/common_ja_JP.properties
Sun Jun 30 14:06:40 2013
@@ -14,5 +14,5 @@
# limitations under the License.
FileConnector.PathTabName=åºåãã¹
-FileConnector.Path=åºåãã¹
FileConnector.RootPath=ã«ã¼ããã¹
+FileConnector.RootPathCannotBeNull=Root path cannot be null
Modified:
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_en_US.properties
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_en_US.properties?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_en_US.properties
(original)
+++
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_en_US.properties
Sun Jun 30 14:06:40 2013
@@ -16,6 +16,10 @@
FileConnector.Paths=Repository Paths
FileConnector.Paths2=Repository Paths:
FileConnector.RootPath=Root path
+FileConnector.ConvertToURI=Convert path to URI?
+FileConnector.ConvertToURIExample= (e.g. http/xyz/index.html =>
http://xyz/index.html)
+FileConnector.Yes=Yes
+FileConnector.No=No
FileConnector.Rules=Rules
FileConnector.Delete=Delete
FileConnector.DeletePath=Delete path #
@@ -35,7 +39,3 @@ FileConnector.InsertNewMatchForPath=Inse
FileConnector.DeletePath=Delete path #
FileConnector.AddNewMatchForPath=Add new match for path #
FileConnector.AddNewPath=Add new path
-
-FileConnector.FilePathToURITab=Convert file path to URI
-FileConnector.FilePathToURI=Convert file path to URI:
-FileConnector.FilePathToURIExample=EX)
$REPOSITORY_PATH/http/localhost/index.html => http://localhost/index.html
Modified:
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_ja_JP.properties
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_ja_JP.properties?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_ja_JP.properties
(original)
+++
manifoldcf/trunk/connectors/filesystem/connector/src/main/native2ascii/org/apache/manifoldcf/crawler/connectors/filesystem/common_ja_JP.properties
Sun Jun 30 14:06:40 2013
@@ -16,6 +16,10 @@
FileConnector.Paths=ãªãã¸ããªãã¹
FileConnector.Paths2=ãªãã¸ããªãã¹ï¼
FileConnector.RootPath=ã«ã¼ããã¹
+FileConnector.ConvertToURI=Convert path to URI?
+FileConnector.ConvertToURIExample= ä¾) http/xyz/index.html =>
http://xyz/index.html
+FileConnector.Yes=Yes
+FileConnector.No=No
FileConnector.Rules=ã«ã¼ã«
FileConnector.Delete=åé¤
FileConnector.DeletePath=ãã¹ãåé¤ #
@@ -35,7 +39,3 @@ FileConnector.InsertNewMatchForPath=ã
FileConnector.DeletePath=ãã¹ãåé¤ï¼ #
FileConnector.AddNewMatchForPath=ãã¹ç¨ã«æ°ãããã¿ã¼ã³ã追å ï¼
#
FileConnector.AddNewPath=æ°ãããã¹ã追å
-
-FileConnector.FilePathToURITab=ãã¡ã¤ã«ãã¹ãURIã¸å¤æãã
-FileConnector.FilePathToURI=ãã¡ã¤ã«ãã¹ãURIã¸å¤æããï¼
-FileConnector.FilePathToURIExample=ä¾)
$REPOSITORY_PATH/http/localhost/index.html => http://localhost/index.html
Modified:
manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.html
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.html?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.html
(original)
+++
manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.html
Sun Jun 30 14:06:40 2013
@@ -18,6 +18,7 @@
#if($TABNAME == $ResourceBundle.getString('FileConnector.PathTabName'))
<table class="displaytable">
+ <tr><td class="separator" colspan="2"><hr/></td></tr>
<tr>
<td
class="description"><nobr>$Encoder.bodyEscape($ResourceBundle.getString('FileConnector.RootPath'))</nobr></td>
<td class="value"><input type="text" name="rootpath" size="64"
value="$Encoder.attributeEscape($ROOTPATH)" /></td>
Modified:
manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.js
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.js?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.js
(original)
+++
manifoldcf/trunk/connectors/filesystem/connector/src/main/resources/org/apache/manifoldcf/agents/output/filesystem/editSpecification.js
Sun Jun 30 14:06:40 2013
@@ -17,5 +17,16 @@
<script type="text/javascript">
<!--
+function checkOutputSpecificationForSave()
+{
+ if (editjob.rootpath.value == "")
+ {
+
alert("$Encoder.bodyJavascriptEscape($ResourceBundle.getString('FileConnector.RootPathCannotBeNull'))");
+
SelectTab("$Encoder.bodyJavascriptEscape($ResourceBundle.getString('FileConnector.PathTabName'))");
+ editjob.rootpath.focus();
+ return false;
+ }
+ return true;
+}
//-->
</script>
\ No newline at end of file
Modified:
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
---
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
(original)
+++
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
Sun Jun 30 14:06:40 2013
@@ -519,6 +519,19 @@
Solr, and click the "Add" button. Leaving the "target"
field blank will result in all metadata items of that name not being sent to
Solr.</p>
</section>
+ <section id="filesystemoutputconnector">
+ <title>File System Output Connection</title>
+ <p>The File System output connection type allows ManifoldCF to
store documents in a local filesystem, using the conventions established by the
+ Unix utility called <em>wget</em>. Documents stored by
this connection type will not include any metadata or security information, but
instead
+ consist solely of a binary file.</p>
+ <p>The connection configuration information for the File
System output connection type includes no additional tabs. There is an
additional job tab,
+ however, called "Output Path". The tab looks like
this:</p>
+ <br/><br/>
+ <figure src="images/en_US/filesystem-job-output-path.PNG"
alt="File System Specification, Output Path tab" width="80%"/>
+ <br/><br/>
+ <p>Fill in the path you want the connection type to use to
write the documents to. Then, click the "Save" button.</p>
+ </section>
+
<section id="opensearchserveroutputconnector">
<title>OpenSearchServer Output Connection</title>
<p>The OpenSearchServer Output Connection allow ManifoldCF to
submit documents to an OpenSearchServer instance, via the XML over HTTP API.
The connector has been designed
@@ -910,35 +923,35 @@ curl -XGET http://localhost:9200/index/_
<section id="genericauthority">
<title>Generic Authority</title>
<p>Generic authority is intended to be used with Generic
Connector and provide authentication tokens based on generic API. The idea is
that you can use it and implement only the API which is designed
- to be fine grained and as simple as it is possible to
handle all tasks.</p>
+ to be fine grained and as simple as it is possible to handle
all tasks.</p>
<p>API should be implemented as xml web page (entry point)
returning results based on provided GET params. It may be a simple server
script or part of the bigger application.
- API can be secured with HTTP basic authentication.</p>
+ API can be secured with HTTP basic authentication.</p>
<br/>
<p>There are 2 actions:</p>
- <ul>
- <li>check</li>
- <li>auth</li>
- </ul>
- <p>Action is passed as "action" GET param to the
entrypoint.</p>
+ <ul>
+ <li>check</li>
+ <li>auth</li>
+ </ul>
+ <p>Action is passed as "action" GET param to the entrypoint.</p>
<br/><br/>
- <p><b>[entrypoint]?action=check</b></p>
- <p>Should return HTTP status code 200 providing
information that entrypoint is working properly. Any content returned will be
ignored, only the status code matters.</p>
+ <p><b>[entrypoint]?action=check</b></p>
+ <p>Should return HTTP status code 200 providing information that
entrypoint is working properly. Any content returned will be ignored, only the
status code matters.</p>
<br/><br/>
-
<p><b>[entrypoint]?action=auth&username=UserName@Domain</b></p>
- <p>Parameters:</p>
- <ul>
- <li>username - name of the user we want to
resolve and fetch tokens.</li>
- </ul>
- <p>Result should be valid XML of form:</p>
- <source>
+
<p><b>[entrypoint]?action=auth&username=UserName@Domain</b></p>
+ <p>Parameters:</p>
+ <ul>
+ <li>username - name of the user we want to resolve and fetch
tokens.</li>
+ </ul>
+ <p>Result should be valid XML of form:</p>
+ <source>
<auth exists="true|false">
<token>token_1</token>;
<token>token_2</token>;
...
</auth>
- </source>
- <p><code>exists</code> attribute is required and it
carries information wheter user is valid or not.</p>
+ </source>
+ <p><code>exists</code> attribute is required and it carries
information whether user is valid or not.</p>
<br/><br/>
</section>
</section>
@@ -947,14 +960,16 @@ curl -XGET http://localhost:9200/index/_
<title>Repository Connection Types</title>
<section id="filesystemrepository">
- <title>Generic File System Repository Connection</title>
- <p>The generic file system repository connection type was
developed primarily as an example, demonstration, and testing tool, although it
can potentially be useful for indexing local
- files that exist on the same machine that ManifoldCF is
running on. Bear in mind that there is no support in this connection type for
any kind of
- security, and the options are somewhat limited.</p>
+ <title>Generic WGET-Sensitive File System Repository
Connection</title>
+ <p>The generic file system repository connection type was
developed in part as an example, demonstration, and testing tool, which reads
simple
+ files in directory paths, and partly as ManifoldCF
support for the Unix utility called <em>wget</em>. In the latter mode, the
File System Repository Connector
+ will parse file names that were created by
<em>wget</em>, or by the wget-compatible File System Output Connector, and turn
these back
+ into full URL's to external web content.</p>
+ <p>This connection type has no support for any kind of
document security.</p>
<p>The file system repository connection type provides no
configuration tabs beyond the standard ones. However, please consider setting
a "Maximum connections per
JVM" value on the "Throttling" tab to at least one per
worker thread, or 30, for best performance.</p>
<p>Jobs created using a file-system-type repository connection
- have two tabs in addition to the standard repertoire:
the "Hop Filters" tab, and the "Paths" tab.</p>
+ have two tabs in addition to the standard repertoire:
the "Hop Filters" tab, and the "Repository Paths" tab.</p>
<p>The "Hop Filters" tab allows you to restrict the document
set by the number of child hops from the path root. While this is not terribly
interesting in the case of a file
system, the same basic functionality is also used in
the Web connection type, where it is a more important feature. The file system
connection type gives you a way to see
how this feature works, in a more predictable
environment:</p>
@@ -969,12 +984,13 @@ curl -XGET http://localhost:9200/index/_
expensive bookkeeping, however, so you also have the
option of ignoring such changes. There are two varieties of this latter
option - you can ignore the changes
for now, with the option of turning back on the
aggressive bookkeeping at a later time, or you can decide not to ever allow
changes to propagate, in which case
the Framework will discard the necessary bookkeeping
information permanently.</p>
- <p>The "Paths" tab looks like this:</p>
+ <p>The "Repository Paths" tab looks like this:</p>
<br/><br/>
- <figure src="images/en_US/filesystem-job-paths.PNG" alt="File
System Connection, Paths tab" width="80%"/>
+ <figure src="images/en_US/filesystem-job-paths.PNG" alt="File
System Connection, Repository Paths tab" width="80%"/>
<br/><br/>
- <p>This tab allows you to type in a set of paths which
function as the roots of the crawl. For each desired path, type in the path
and click the "Add" button to add it to
- the list. The form of the path you type in obviously
needs to be meaningful for the operating system the Framework is running on.</p>
+ <p>This tab allows you to type in a set of paths which
function as the roots of the crawl. For each desired path, type in the path,
select whether the root should
+ behave as an WGET repository or not, and click the
"Add" button to add it to the list. The form of the path you type in obviously
needs to be meaningful
+ for the operating system the Framework is running
on.</p>
<p>Each root path has a set of rules which determines whether
a document is included or not in the set for the job. Once you have added the
root path to the list, you
may then add rules to it. Each rule has a match
expression, an indication of whether the rule is intended to match files or
directories, and an action (include or exclude).
Rules are evaluated from top to bottom, and the first
rule that matches the file name is the one that is chosen. To add a rule,
select the desired pulldowns, type in
Added:
manifoldcf/trunk/site/src/documentation/resources/images/en_US/filesystem-job-output-path.PNG
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/resources/images/en_US/filesystem-job-output-path.PNG?rev=1498126&view=auto
==============================================================================
Binary file - no diff available.
Propchange:
manifoldcf/trunk/site/src/documentation/resources/images/en_US/filesystem-job-output-path.PNG
------------------------------------------------------------------------------
svn:mime-type = application/octet-stream
Modified:
manifoldcf/trunk/site/src/documentation/resources/images/en_US/filesystem-job-paths.PNG
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/resources/images/en_US/filesystem-job-paths.PNG?rev=1498126&r1=1498125&r2=1498126&view=diff
==============================================================================
Binary files - no diff available.
