Author: kwright
Date: Mon Jun 24 06:55:17 2013
New Revision: 1495934
URL: http://svn.apache.org/r1495934
Log:
Add Jira connector. Finishes CONNECTORS-723.
Added:
manifoldcf/trunk/connectors/jira/ (props changed)
- copied from r1495933,
manifoldcf/branches/CONNECTORS-723/connectors/jira/
manifoldcf/trunk/framework/core/src/main/java/org/apache/manifoldcf/core/common/InterruptibleSocketFactory.java
- copied unchanged from r1495933,
manifoldcf/branches/CONNECTORS-723/framework/core/src/main/java/org/apache/manifoldcf/core/common/InterruptibleSocketFactory.java
manifoldcf/trunk/site/src/documentation/resources/images/en_US/jira-repository-connection-configuration-save.PNG
- copied unchanged from r1495933,
manifoldcf/branches/CONNECTORS-723/site/src/documentation/resources/images/en_US/jira-repository-connection-configuration-save.PNG
manifoldcf/trunk/site/src/documentation/resources/images/en_US/jira-repository-connection-configuration.PNG
- copied unchanged from r1495933,
manifoldcf/branches/CONNECTORS-723/site/src/documentation/resources/images/en_US/jira-repository-connection-configuration.PNG
manifoldcf/trunk/site/src/documentation/resources/images/en_US/jira-repository-connection-job-jira-seed-query.PNG
- copied unchanged from r1495933,
manifoldcf/branches/CONNECTORS-723/site/src/documentation/resources/images/en_US/jira-repository-connection-job-jira-seed-query.PNG
Modified:
manifoldcf/trunk/ (props changed)
manifoldcf/trunk/CHANGES.txt
manifoldcf/trunk/build.xml
manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java
manifoldcf/trunk/connectors/pom.xml
manifoldcf/trunk/connectors/rss/ (props changed)
manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java
manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java
manifoldcf/trunk/connectors/wiki/ (props changed)
manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
manifoldcf/trunk/site/src/documentation/skinconf.xml
Propchange: manifoldcf/trunk/
------------------------------------------------------------------------------
Merged /manifoldcf/branches/CONNECTORS-723:r1494650-1495933
Modified: manifoldcf/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/CHANGES.txt?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/CHANGES.txt (original)
+++ manifoldcf/trunk/CHANGES.txt Mon Jun 24 06:55:17 2013
@@ -3,6 +3,11 @@ $Id$
======================= 1.3-dev =====================
+CONNECTORS-723: Add a Jira connector. This connector does not yet
+handle security or comments, but may be extended in the future to do
+so.
+(Andrew Janowczyk, Karl Wright)
+
CONNECTORS-725: Update to SolrJ 4.3.1.
(Karl Wright)
Modified: manifoldcf/trunk/build.xml
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/build.xml?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/build.xml (original)
+++ manifoldcf/trunk/build.xml Mon Jun 24 06:55:17 2013
@@ -60,6 +60,7 @@
<ant dir="connectors/cmis" target="clean"/>
<ant dir="connectors/dropbox" target="clean"/>
<ant dir="connectors/googledrive" target="clean"/>
+ <ant dir="connectors/jira" target="clean"/>
<ant dir="connectors/activedirectory" target="clean"/>
<ant dir="connectors/ldap" target="clean"/>
<ant dir="connectors/documentum" target="clean"/>
@@ -112,6 +113,7 @@
<ant dir="connectors/cmis" target="clean"/>
<ant dir="connectors/dropbox" target="clean"/>
<ant dir="connectors/googledrive" target="clean"/>
+ <ant dir="connectors/jira" target="clean"/>
<ant dir="connectors/activedirectory" target="clean"/>
<ant dir="connectors/ldap" target="clean"/>
<ant dir="connectors/documentum" target="clean"/>
@@ -285,6 +287,7 @@
<target name="setup-dropbox-connector" depends="build-framework"
if="downloaded"/>
<target name="setup-googledrive-connector" depends="build-framework"
if="downloaded"/>
+ <target name="setup-jira-connector" depends="build-framework"
if="downloaded"/>
<target name="setup-alfresco-connector-tests"
depends="build-tests-framework" if="downloaded"/>
@@ -311,6 +314,10 @@
<ant dir="connectors/googledrive" target="build"/>
</target>
+ <target name="build-jira-connector" depends="setup-jira-connector"
if="downloaded">
+ <ant dir="connectors/jira" target="build"/>
+ </target>
+
<target name="doc-dropbox-connector" depends="setup-dropbox-connector"
if="downloaded">
<ant dir="connectors/dropbox" target="doc"/>
@@ -320,7 +327,10 @@
<ant dir="connectors/googledrive" target="doc"/>
</target>
-
+ <target name="doc-jira-connector" depends="setup-jira-connector"
if="downloaded">
+ <ant dir="connectors/googledrive" target="doc"/>
+ </target>
+
<target name="doc-alfresco-connector" depends="setup-alfresco-connector"
if="downloaded">
<ant dir="connectors/alfresco" target="doc"/>
</target>
@@ -1431,6 +1441,26 @@
</condition>
</target>
+<target name="calculate-jira-condition" depends="build-jira-connector">
+ <available file="connectors/jira/dist/lib" type="dir"
property="jira.exists"/>
+ <condition property="jira.include">
+ <and>
+ <isset property="jira.exists"/>
+ <isset property="downloaded"/>
+ </and>
+ </condition>
+ </target>
+
+ <target name="calculate-jira-doc-condition" depends="doc-jira-connector">
+ <available file="connectors/jira/dist/doc" type="dir"
property="jira-doc.exists"/>
+ <condition property="jira-doc.include">
+ <and>
+ <isset property="jira-doc.exists"/>
+ <isset property="downloaded"/>
+ </and>
+ </condition>
+ </target>
+
<target name="calculate-cmis-doc-condition" depends="doc-cmis-connector">
@@ -1493,7 +1523,24 @@
</antcall>
</target>
-
+ <target name="deliver-jira-connector" depends="calculate-jira-condition"
if="jira.include">
+ <antcall target="general-connector-delivery">
+ <param name="connector-name" value="jira"/>
+ </antcall>
+ <antcall target="general-add-repository-connector">
+ <param name="connector-name" value="jira"/>
+ <param name="connector-label" value="Jira"/>
+ <param name="connector-class"
value="org.apache.manifoldcf.crawler.connectors.jira.JiraRepositoryConnector"/>
+ </antcall>
+ </target>
+
+ <target name="deliver-jira-connector-doc"
depends="calculate-jira-doc-condition" if="jira-doc.include">
+ <antcall target="general-connector-doc-delivery">
+ <param name="connector-name" value="jira"/>
+ </antcall>
+ </target>
+
+
<target name="deliver-cmis-connector-doc"
depends="calculate-cmis-doc-condition" if="cmis-doc.include">
<antcall target="general-connector-doc-delivery">
<param name="connector-name" value="cmis"/>
@@ -2651,8 +2698,8 @@
<target name="end-to-end-loadtests-HSQLDB"
depends="run-filesystem-loadtests-HSQLDB,run-rss-loadtests-HSQLDB,run-wiki-loadtests-HSQLDB,run-alfresco-loadtests-HSQLDB,run-cmis-loadtests-HSQLDB,run-sharepoint-loadtests-HSQLDB"/>
- <target name="deliver-open-connectors"
depends="deliver-googledrive-connector,deliver-dropbox-connector,deliver-nullauthority-connector,deliver-activedirectory-connector,deliver-ldap-connector,deliver-alfresco-connector,deliver-cmis-connector,deliver-filesystem-connector,deliver-rss-connector,deliver-webcrawler-connector,deliver-wiki-connector,deliver-jdbc-connector"/>
- <target name="deliver-open-connectors-doc"
depends="deliver-googledrive-connector-doc,deliver-dropbox-connector-doc,deliver-nullauthority-connector-doc,deliver-activedirectory-connector-doc,deliver-ldap-connector-doc,deliver-alfresco-connector-doc,deliver-cmis-connector-doc,deliver-filesystem-connector-doc,deliver-rss-connector-doc,deliver-webcrawler-connector-doc,deliver-wiki-connector-doc,deliver-jdbc-connector-doc"/>
+ <target name="deliver-open-connectors"
depends="deliver-jira-connector,deliver-googledrive-connector,deliver-dropbox-connector,deliver-nullauthority-connector,deliver-activedirectory-connector,deliver-ldap-connector,deliver-alfresco-connector,deliver-cmis-connector,deliver-filesystem-connector,deliver-rss-connector,deliver-webcrawler-connector,deliver-wiki-connector,deliver-jdbc-connector"/>
+ <target name="deliver-open-connectors-doc"
depends="deliver-googledrive-connector-doc,deliver-jira-connector-doc,deliver-dropbox-connector-doc,deliver-nullauthority-connector-doc,deliver-activedirectory-connector-doc,deliver-ldap-connector-doc,deliver-alfresco-connector-doc,deliver-cmis-connector-doc,deliver-filesystem-connector-doc,deliver-rss-connector-doc,deliver-webcrawler-connector-doc,deliver-wiki-connector-doc,deliver-jdbc-connector-doc"/>
<target name="deliver-output-connectors"
depends="deliver-gts-connector,deliver-solr-connector,deliver-nulloutput-connector,deliver-opensearchserver-connector,deliver-elasticsearch-connector"/>
<target name="deliver-output-connectors-doc"
depends="deliver-gts-connector-doc,deliver-solr-connector-doc,deliver-nulloutput-connector-doc,deliver-opensearchserver-connector-doc,deliver-elasticsearch-connector-doc"/>
@@ -3660,6 +3707,24 @@ Use Apache Forrest version forrest-0.9-d
</antcall>
</target>
+
+ <target name="download-jira-client">
+ <mkdir dir="lib"/>
+ <antcall target="download-via-maven">
+ <param name="target" value="lib"/>
+ <param name="project-path" value="com/googlecode/json-simple"/>
+ <param name="artifact-version" value="1.1"/>
+ <param name="artifact-name" value="json-simple"/>
+ <param name="artifact-type" value="jar"/>
+ </antcall>
+ <antcall target="download-via-maven">
+ <param name="target" value="lib"/>
+ <param name="project-path" value="commons-codec"/>
+ <param name="artifact-version" value="1.8"/>
+ <param name="artifact-name" value="commons-codec"/>
+ <param name="artifact-type" value="jar"/>
+ </antcall>
+ </target>
<target name="download-google-api-client">
<mkdir dir="lib"/>
@@ -3761,7 +3826,7 @@ Use Apache Forrest version forrest-0.9-d
</antcall>
</target>
- <target name="make-core-deps"
depends="download-google-api-client,download-dropbox-client,download-solrj,download-httpcomponents,download-json,download-hsqldb,download-xerces,download-commons,download-elasticsearch-plugin,download-solr-plugins,download-sharepoint-plugins,download-jstl,download-xmlgraphics-commons,download-wstx-asl,download-xmlsec,download-xml-apis,download-wss4j,download-velocity,download-streambuffer,download-stax,download-servlet-api,download-xml-resolver,download-osgi,download-opensaml,download-mimepull,download-mail,download-log4j,download-junit,download-jaxws,download-glassfish,download-jaxb,download-tomcat,download-h2,download-h2-support,download-geronimo-specs,download-fop,download-derby,download-postgresql,download-axis,download-saaj,download-wsdl4j,download-castor,download-jetty,download-slf4j,download-xalan,download-activation,download-avalon-framework,download-poi,download-chemistry,download-ecj">
+ <target name="make-core-deps"
depends="download-jira-client,download-google-api-client,download-dropbox-client,download-solrj,download-httpcomponents,download-json,download-hsqldb,download-xerces,download-commons,download-elasticsearch-plugin,download-solr-plugins,download-sharepoint-plugins,download-jstl,download-xmlgraphics-commons,download-wstx-asl,download-xmlsec,download-xml-apis,download-wss4j,download-velocity,download-streambuffer,download-stax,download-servlet-api,download-xml-resolver,download-osgi,download-opensaml,download-mimepull,download-mail,download-log4j,download-junit,download-jaxws,download-glassfish,download-jaxb,download-tomcat,download-h2,download-h2-support,download-geronimo-specs,download-fop,download-derby,download-postgresql,download-axis,download-saaj,download-wsdl4j,download-castor,download-jetty,download-slf4j,download-xalan,download-activation,download-avalon-framework,download-poi,download-chemistry,download-ecj">
<copy todir="lib">
<fileset dir="lib-license" includes="*.txt"/>
</copy>
@@ -3792,6 +3857,7 @@ Use Apache Forrest version forrest-0.9-d
<ant dir="connectors/cmis" target="download-dependencies"/>
<ant dir="connectors/dropbox" target="download-dependencies"/>
<ant dir="connectors/googledrive" target="download-dependencies"/>
+ <ant dir="connectors/jira" target="download-dependencies"/>
<ant dir="connectors/activedirectory" target="download-dependencies"/>
<ant dir="connectors/ldap" target="download-dependencies"/>
<ant dir="connectors/documentum" target="download-dependencies"/>
@@ -3830,6 +3896,7 @@ Use Apache Forrest version forrest-0.9-d
<ant dir="connectors/cmis" target="download-cleanup"/>
<ant dir="connectors/dropbox" target="download-cleanup"/>
<ant dir="connectors/googledrive" target="download-cleanup"/>
+ <ant dir="connectors/jira" target="download-cleanup"/>
<ant dir="connectors/activedirectory" target="download-cleanup"/>
<ant dir="connectors/ldap" target="download-cleanup"/>
<ant dir="connectors/documentum" target="download-cleanup"/>
Propchange: manifoldcf/trunk/connectors/jira/
------------------------------------------------------------------------------
--- svn:ignore (added)
+++ svn:ignore Mon Jun 24 06:55:17 2013
@@ -0,0 +1,10 @@
+build
+dist
+doc
+target
+test-output
+test-derby-output
+test-postgresql-output
+test-HSQLDB-output
+test-HSQLDBext-output
+test-mysql-output
Modified:
manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java
(original)
+++
manifoldcf/trunk/connectors/livelink/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/livelink/LivelinkConnector.java
Mon Jun 24 06:55:17 2013
@@ -25,6 +25,7 @@ import org.apache.manifoldcf.crawler.sys
import org.apache.manifoldcf.crawler.system.ManifoldCF;
import org.apache.manifoldcf.core.common.XThreadInputStream;
import org.apache.manifoldcf.core.common.XThreadOutputStream;
+import org.apache.manifoldcf.core.common.InterruptibleSocketFactory;
import java.io.*;
import java.util.*;
@@ -473,6 +474,9 @@ public class LivelinkConnector extends o
getSessionParameters();
if (hasConnected == false)
{
+ int socketTimeout = 900000;
+ int connectionTimeout = 300000;
+
// Set up connection manager
PoolingClientConnectionManager localConnectionManager = new
PoolingClientConnectionManager();
localConnectionManager.setMaxTotal(1);
@@ -480,7 +484,7 @@ public class LivelinkConnector extends o
// Set up ingest ssl if indicated
if (ingestKeystoreManager != null)
{
- SSLSocketFactory myFactory = new
SSLSocketFactory(ingestKeystoreManager.getSecureSocketFactory(),
+ SSLSocketFactory myFactory = new SSLSocketFactory(new
InterruptibleSocketFactory(ingestKeystoreManager.getSecureSocketFactory(),
connectionTimeout),
new BrowserCompatHostnameVerifier());
Scheme myHttpsProtocol = new Scheme("https", 443, myFactory);
localConnectionManager.getSchemeRegistry().register(myHttpsProtocol);
Modified: manifoldcf/trunk/connectors/pom.xml
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/pom.xml?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/connectors/pom.xml (original)
+++ manifoldcf/trunk/connectors/pom.xml Mon Jun 24 06:55:17 2013
@@ -52,6 +52,7 @@
<module>elasticsearch</module>
<module>dropbox</module>
<module>googledrive</module>
+ <module>jira</module>
</modules>
</project>
Propchange: manifoldcf/trunk/connectors/rss/
------------------------------------------------------------------------------
Merged /manifoldcf/branches/CONNECTORS-723/connectors/rss:r1494650-1495933
Modified:
manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java
(original)
+++
manifoldcf/trunk/connectors/rss/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/rss/ThrottledFetcher.java
Mon Jun 24 06:55:17 2013
@@ -20,6 +20,7 @@ package org.apache.manifoldcf.crawler.co
import org.apache.manifoldcf.core.interfaces.*;
import org.apache.manifoldcf.core.common.XThreadInputStream;
+import org.apache.manifoldcf.core.common.InterruptibleSocketFactory;
import org.apache.manifoldcf.agents.interfaces.*;
import org.apache.manifoldcf.crawler.interfaces.*;
import org.apache.manifoldcf.crawler.system.Logging;
@@ -1485,176 +1486,5 @@ public class ThrottledFetcher
}
- /** SSL Socket factory which wraps another socket factory but allows timeout
on socket
- * creation.
- */
- protected static class InterruptibleSocketFactory extends
javax.net.ssl.SSLSocketFactory
- {
- protected final javax.net.ssl.SSLSocketFactory wrappedFactory;
- protected final long connectTimeoutMilliseconds;
-
- public InterruptibleSocketFactory(javax.net.ssl.SSLSocketFactory
wrappedFactory, long connectTimeoutMilliseconds)
- {
- this.wrappedFactory = wrappedFactory;
- this.connectTimeoutMilliseconds = connectTimeoutMilliseconds;
- }
-
- @Override
- public Socket createSocket()
- throws IOException
- {
- // Socket isn't open
- return wrappedFactory.createSocket();
- }
-
- @Override
- public Socket createSocket(String host, int port)
- throws IOException, UnknownHostException
- {
- return fireOffThread(InetAddress.getByName(host),port,null,-1);
- }
-
- @Override
- public Socket createSocket(InetAddress host, int port)
- throws IOException
- {
- return fireOffThread(host,port,null,-1);
- }
-
- @Override
- public Socket createSocket(String host, int port, InetAddress localHost,
int localPort)
- throws IOException, UnknownHostException
- {
- return
fireOffThread(InetAddress.getByName(host),port,localHost,localPort);
- }
-
- @Override
- public Socket createSocket(InetAddress address, int port, InetAddress
localAddress, int localPort)
- throws IOException
- {
- return fireOffThread(address,port,localAddress,localPort);
- }
-
- @Override
- public Socket createSocket(Socket s, String host, int port, boolean
autoClose)
- throws IOException
- {
- // Socket's already open
- return wrappedFactory.createSocket(s,host,port,autoClose);
- }
-
- @Override
- public String[] getDefaultCipherSuites()
- {
- return wrappedFactory.getDefaultCipherSuites();
- }
-
- @Override
- public String[] getSupportedCipherSuites()
- {
- return wrappedFactory.getSupportedCipherSuites();
- }
-
- protected Socket fireOffThread(InetAddress address, int port, InetAddress
localHost, int localPort)
- throws IOException
- {
- SocketCreateThread thread = new
SocketCreateThread(wrappedFactory,address,port,localHost,localPort);
- thread.start();
- try
- {
- // Wait for thread to complete for only a certain amount of time!
- thread.join(connectTimeoutMilliseconds);
- // If join() times out, then the thread is going to still be alive.
- if (thread.isAlive())
- {
- // Kill the thread - not that this will necessarily work, but we
need to try
- thread.interrupt();
- throw new ConnectTimeoutException("Secure connection timed out");
- }
- // The thread terminated. Throw an error if there is one, otherwise
return the result.
- Throwable t = thread.getException();
- if (t != null)
- {
- if (t instanceof java.net.SocketTimeoutException)
- throw (java.net.SocketTimeoutException)t;
- else if (t instanceof ConnectTimeoutException)
- throw (ConnectTimeoutException)t;
- else if (t instanceof InterruptedIOException)
- throw (InterruptedIOException)t;
- else if (t instanceof IOException)
- throw (IOException)t;
- else if (t instanceof Error)
- throw (Error)t;
- else if (t instanceof RuntimeException)
- throw (RuntimeException)t;
- throw new Error("Received an unexpected exception:
"+t.getMessage(),t);
- }
- return thread.getResult();
- }
- catch (InterruptedException e)
- {
- throw new InterruptedIOException("Interrupted: "+e.getMessage());
- }
-
- }
-
- }
-
- /** Create a secure socket in a thread, so that we can "give up" after a
while if the socket fails to connect.
- */
- protected static class SocketCreateThread extends Thread
- {
- // Socket factory
- protected javax.net.ssl.SSLSocketFactory socketFactory;
- protected InetAddress host;
- protected int port;
- protected InetAddress clientHost;
- protected int clientPort;
-
- // The return socket
- protected Socket rval = null;
- // The return error
- protected Throwable throwable = null;
-
- /** Create the thread */
- public SocketCreateThread(javax.net.ssl.SSLSocketFactory socketFactory,
- InetAddress host,
- int port,
- InetAddress clientHost,
- int clientPort)
- {
- this.socketFactory = socketFactory;
- this.host = host;
- this.port = port;
- this.clientHost = clientHost;
- this.clientPort = clientPort;
- setDaemon(true);
- }
-
- public void run()
- {
- try
- {
- if (clientHost == null)
- rval = socketFactory.createSocket(host,port);
- else
- rval = socketFactory.createSocket(host,port,clientHost,clientPort);
- }
- catch (Throwable e)
- {
- throwable = e;
- }
- }
-
- public Throwable getException()
- {
- return throwable;
- }
-
- public Socket getResult()
- {
- return rval;
- }
- }
}
Modified:
manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java
(original)
+++
manifoldcf/trunk/connectors/webcrawler/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/webcrawler/ThrottledFetcher.java
Mon Jun 24 06:55:17 2013
@@ -21,6 +21,7 @@ package org.apache.manifoldcf.crawler.co
import org.apache.manifoldcf.core.interfaces.*;
import org.apache.manifoldcf.core.common.DeflateInputStream;
import org.apache.manifoldcf.core.common.XThreadInputStream;
+import org.apache.manifoldcf.core.common.InterruptibleSocketFactory;
import org.apache.manifoldcf.agents.interfaces.*;
import org.apache.manifoldcf.crawler.interfaces.*;
import org.apache.manifoldcf.crawler.system.Logging;
@@ -2259,178 +2260,6 @@ public class ThrottledFetcher
}
}
- /** SSL Socket factory which wraps another socket factory but allows timeout
on socket
- * creation.
- */
- protected static class InterruptibleSocketFactory extends
javax.net.ssl.SSLSocketFactory
- {
- protected final javax.net.ssl.SSLSocketFactory wrappedFactory;
- protected final long connectTimeoutMilliseconds;
-
- public InterruptibleSocketFactory(javax.net.ssl.SSLSocketFactory
wrappedFactory, long connectTimeoutMilliseconds)
- {
- this.wrappedFactory = wrappedFactory;
- this.connectTimeoutMilliseconds = connectTimeoutMilliseconds;
- }
-
- @Override
- public Socket createSocket()
- throws IOException
- {
- // Socket isn't open
- return wrappedFactory.createSocket();
- }
-
- @Override
- public Socket createSocket(String host, int port)
- throws IOException, UnknownHostException
- {
- return fireOffThread(InetAddress.getByName(host),port,null,-1);
- }
-
- @Override
- public Socket createSocket(InetAddress host, int port)
- throws IOException
- {
- return fireOffThread(host,port,null,-1);
- }
-
- @Override
- public Socket createSocket(String host, int port, InetAddress localHost,
int localPort)
- throws IOException, UnknownHostException
- {
- return
fireOffThread(InetAddress.getByName(host),port,localHost,localPort);
- }
-
- @Override
- public Socket createSocket(InetAddress address, int port, InetAddress
localAddress, int localPort)
- throws IOException
- {
- return fireOffThread(address,port,localAddress,localPort);
- }
-
- @Override
- public Socket createSocket(Socket s, String host, int port, boolean
autoClose)
- throws IOException
- {
- // Socket's already open
- return wrappedFactory.createSocket(s,host,port,autoClose);
- }
-
- @Override
- public String[] getDefaultCipherSuites()
- {
- return wrappedFactory.getDefaultCipherSuites();
- }
-
- @Override
- public String[] getSupportedCipherSuites()
- {
- return wrappedFactory.getSupportedCipherSuites();
- }
-
- protected Socket fireOffThread(InetAddress address, int port, InetAddress
localHost, int localPort)
- throws IOException
- {
- SocketCreateThread thread = new
SocketCreateThread(wrappedFactory,address,port,localHost,localPort);
- thread.start();
- try
- {
- // Wait for thread to complete for only a certain amount of time!
- thread.join(connectTimeoutMilliseconds);
- // If join() times out, then the thread is going to still be alive.
- if (thread.isAlive())
- {
- // Kill the thread - not that this will necessarily work, but we
need to try
- thread.interrupt();
- throw new ConnectTimeoutException("Secure connection timed out");
- }
- // The thread terminated. Throw an error if there is one, otherwise
return the result.
- Throwable t = thread.getException();
- if (t != null)
- {
- if (t instanceof java.net.SocketTimeoutException)
- throw (java.net.SocketTimeoutException)t;
- else if (t instanceof ConnectTimeoutException)
- throw (ConnectTimeoutException)t;
- else if (t instanceof InterruptedIOException)
- throw (InterruptedIOException)t;
- else if (t instanceof IOException)
- throw (IOException)t;
- else if (t instanceof Error)
- throw (Error)t;
- else if (t instanceof RuntimeException)
- throw (RuntimeException)t;
- throw new Error("Received an unexpected exception:
"+t.getMessage(),t);
- }
- return thread.getResult();
- }
- catch (InterruptedException e)
- {
- throw new InterruptedIOException("Interrupted: "+e.getMessage());
- }
-
- }
-
- }
-
- /** Create a secure socket in a thread, so that we can "give up" after a
while if the socket fails to connect.
- */
- protected static class SocketCreateThread extends Thread
- {
- // Socket factory
- protected javax.net.ssl.SSLSocketFactory socketFactory;
- protected InetAddress host;
- protected int port;
- protected InetAddress clientHost;
- protected int clientPort;
-
- // The return socket
- protected Socket rval = null;
- // The return error
- protected Throwable throwable = null;
-
- /** Create the thread */
- public SocketCreateThread(javax.net.ssl.SSLSocketFactory socketFactory,
- InetAddress host,
- int port,
- InetAddress clientHost,
- int clientPort)
- {
- this.socketFactory = socketFactory;
- this.host = host;
- this.port = port;
- this.clientHost = clientHost;
- this.clientPort = clientPort;
- setDaemon(true);
- }
-
- public void run()
- {
- try
- {
- if (clientHost == null)
- rval = socketFactory.createSocket(host,port);
- else
- rval = socketFactory.createSocket(host,port,clientHost,clientPort);
- }
- catch (Throwable e)
- {
- throwable = e;
- }
- }
-
- public Throwable getException()
- {
- return throwable;
- }
-
- public Socket getResult()
- {
- return rval;
- }
- }
-
/** Class to override browser compatibility to make it not check cookie
paths. See CONNECTORS-97.
*/
protected static class LaxBrowserCompatSpec extends BrowserCompatSpec
Propchange: manifoldcf/trunk/connectors/wiki/
------------------------------------------------------------------------------
Merged /manifoldcf/branches/CONNECTORS-723/connectors/wiki:r1494650-1495933
Modified:
manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
---
manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java
(original)
+++
manifoldcf/trunk/connectors/wiki/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/wiki/WikiConnector.java
Mon Jun 24 06:55:17 2013
@@ -57,6 +57,9 @@ import org.apache.http.message.BasicHead
import org.apache.http.client.params.ClientPNames;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.protocol.HttpContext;
+import org.apache.http.conn.scheme.Scheme;
+import org.apache.http.conn.ssl.SSLSocketFactory;
+import org.apache.http.conn.ssl.AllowAllHostnameVerifier;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.client.CircularRedirectException;
@@ -181,13 +184,20 @@ public class WikiConnector extends org.a
baseURL = protocol + "://" + server + ((portString!=null)?":" +
portString:"") + path + "/api.php?format=xml&";
+ int socketTimeout = 900000;
+ int connectionTimeout = 300000;
+
+ javax.net.ssl.SSLSocketFactory httpsSocketFactory =
KeystoreManagerFactory.getTrustingSecureSocketFactory();
+ SSLSocketFactory myFactory = new SSLSocketFactory(new
InterruptibleSocketFactory(httpsSocketFactory,connectionTimeout),
+ new AllowAllHostnameVerifier());
+ Scheme myHttpsProtocol = new Scheme("https", 443, myFactory);
+
// Set up connection manager
PoolingClientConnectionManager localConnectionManager = new
PoolingClientConnectionManager();
localConnectionManager.setMaxTotal(1);
connectionManager = localConnectionManager;
-
- int socketTimeout = 900000;
- int connectionTimeout = 300000;
+ // Set up protocol registry
+ connectionManager.getSchemeRegistry().register(myHttpsProtocol);
BasicHttpParams params = new BasicHttpParams();
params.setBooleanParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE,true);
Modified:
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
---
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
(original)
+++
manifoldcf/trunk/site/src/documentation/content/xdocs/en_US/end-user-documentation.xml
Mon Jun 24 06:55:17 2013
@@ -1762,11 +1762,35 @@ curl -XGET http://localhost:9200/index/_
<figure
src="images/en_US/googledrive-repository-connection-job-googledrive-seed-query.PNG"
alt="googledrive Repository Connection, seed query" width="80%"/>
<br/><br/>
<p>This tab allows you to specify the query which will be used
to seed documents for the indexing process. The query language is
- specified on the <a
href="https://developers.google.com/drive/search-parameters";>Drive Search
Paramters</a> site. Directories
+ specified on the <a
href="https://developers.google.com/drive/search-parameters";>Drive Search
Parameters</a> site. Directories
which meet the seed query are fully crawled as the query on
applies to seeds. The default query indexes the entire drive. Lastly, native
Google
documents such as spreadsheets and word documents are exported
to PDF and then ingested.</p>
</section>
+ <section id="jirarepository">
+ <title>Jira Repository Connection</title>
+ <p>The Jira Repository Connection type allows you to index
tickets from <a href="http://www.atlassian.com/â";>Atlassian</a>'s <a
href="http://www.atlassian.com/software/jira";>Jira</a>.</p>
+ <p>A Jira connection has the following configuration parameters
on the repository connection editing screen:</p>
+ <br/><br/>
+ <figure
src="images/en_US/jira-repository-connection-configuration.PNG" alt="jira
Repository Connection, configuration parameters" width="80%"/>
+ <br/><br/>
+ <p>As we can see there are 3 pieces of information which are
needed to create a successful connection. The Client ID and Client Secret
+ are the username and password of a Jira account. The JiraUrl
is the base endpoint of the particular Jira Instance, for example:
+ https://searchbox.atlassian.net is the Jira version for
searchbox which is hosted at Atlassian.</p>
+ <br/><br/>
+ <p>After you click the "Save" button, you will see a connection
summary screen, which might look something like this:</p>
+ <br/><br/>
+ <figure
src="images/en_US/jira-repository-connection-configuration-save.PNG" alt="jira
Repository Connection, saving configuration" width="80%"/>
+ <br/><br/>
+ <p>When you configure a job to use the Jira repository
connection an additional tab is presented. This is the "Seed Query" tab:</p>
+ <br/><br/>
+ <figure
src="images/en_US/jira-repository-connection-job-jira-seed-query.PNG" alt="jira
Repository Connection, seed query" width="80%"/>
+ <br/><br/>
+ <p>This tab allows you to specify the query which will be used
to find documents for the indexing process. The query language is
+ specified on the <a
href="https://confluence.atlassian.com/display/JIRA/Advanced+Searching";>Jira
Advanced Searching</a> site. Directories
+ which meet the seed query are fully crawled as the query on
applies to seeds.</p>
+ </section>
+
<section id="livelinkrepository">
<title>OpenText LiveLink Repository Connection</title>
<p>The LiveLink connection type allows you to index content
from LiveLink repositories. LiveLink has a rich variety of different document
types and metadata,
Modified: manifoldcf/trunk/site/src/documentation/skinconf.xml
URL:
http://svn.apache.org/viewvc/manifoldcf/trunk/site/src/documentation/skinconf.xml?rev=1495934&r1=1495933&r2=1495934&view=diff
==============================================================================
--- manifoldcf/trunk/site/src/documentation/skinconf.xml (original)
+++ manifoldcf/trunk/site/src/documentation/skinconf.xml Mon Jun 24 06:55:17
2013
@@ -104,6 +104,7 @@ which will be used to configure the chos
QBase, MetaCarta, and GTS are trademarks of QBase, Inc.
Meridio and Autonomy are trademarks of Hewlett Packard, Inc.
Alfresco is a trademark of Alfresco Software, Inc.
+ Jira is a trademark of Atlassian, Inc.
</trademark-statement>
<!-- Some skins use this to form a 'breadcrumb trail' of links.
