[
https://issues.apache.org/jira/browse/CONNECTORS-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright resolved CONNECTORS-946.
------------------------------------
Resolution: Fixed
r1601529
> Add support for pipeline connector
> ----------------------------------
>
> Key: CONNECTORS-946
> URL: https://issues.apache.org/jira/browse/CONNECTORS-946
> Project: ManifoldCF
> Issue Type: New Feature
> Components: Framework crawler agent
> Affects Versions: ManifoldCF 1.7
> Reporter: Karl Wright
> Assignee: Karl Wright
> Fix For: ManifoldCF 1.7
>
>
> In the Amazon Search Connector, we finally found an example of an output
> connector that needed to do full document processing in order to work. This
> ticket represents work in the framework to create a concept of "pipeline
> connector". Pipeline connections would receive RepositoryDocument objects,
> and transform them to new RepositoryDocument objects. There would be a
> single important method:
> {code}
> public void transformDocument(RepositoryDocument rd,
> ITransformationActivities activities) throws ServiceInterruption,
> ManifoldCFException;
> {code}
> ... where ITransformationActivities would include a method that would send a
> RepositoryDocument object onward to either the output connection or to the
> next pipeline connection.
> Each pipeline connection would have:
> - A name
> - A description
> - Configuration data
> - An optional prerequisite pipeline connection
> Every output connection would have a new field, which is an optional
> prerequisite pipeline connection.
> This design is based loosely on how mapping connections and authority
> connections interrelate. An alternate design would involve having per-job
> specification information, but I think this would wind up being way too
> complex for very little benefit, since each pipeline connection/stage would
> be expected to do relatively simple/granular things, not usually involving
> interaction with an external system.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
