Questing regarding Tika text extraction and elasticsearch

[silvio . r . meier]

Hi Apache ManifoldCF user list

 

I’m experimenting with Apache ManifoldCF 2.3 which I use to index the network Windows shares of our company. I’m using Elasticsearch 1.7.4, Apache ManifoldCF 2.3 with MS Active Directory as authority source.  
I defined a job with the following connection configuration comprising the following chain of transformations (order in the list indicates the order of the transformations):

1.    Repository connection (MS Network Share)
2.    Allowed documents
3.    Tika extractor
4.    Metadata adjuster
5.    Elasticsearch

 

I do this because I don’t want to store the original document inside the elasticsearch index but only the extracted text of the document. This works so far. However, there are numerous documents which cause an exception of the following kind when being  analyzed and sent to the indexer by Apache ManifoldCF. Note that the exceptions happens in the Elastic search analyzer:

 

[2016-03-16 22:22:43,884][DEBUG][action.index             ] [Tefral the Surveyor] [shareindex][2], node[O2bWpnsKS8iAE7hwGEOpuA], [P], s[STARTED]: Failed to execute [index {[sharein
dex][attachment][file://///du-evs-01/AppDevData%24/0Repository/temp/indexingtestcorpus/M%C3%A4useTastaturen%202.3.16%20-%20Kopie.pdf], source[{"access_permission:extract_for_access
ibility" : "true","dcterms:created" : "2016-03-02T13:03:47Z","access_permission:can_modify" : "true","access_permission:modify_annotations" : "true","Creation-Date" : "2016-03-02T1
3:03:47Z","fileLastModified" : "2016-03-02T13:03:37.433Z","access_permission:fill_in_form" : "true","created" : "Wed Mar 02 14:03:47 CET 2016","stream_size" : "52067","dc:format" :
 "application\/pdf; version=1.4","access_permission:can_print" : "true","stream_name" : "M├ñuseTastaturen 2.3.16 - Kopie.pdf","xmp:CreatorTool" : "Canon iR-ADV C5250  PDF","resourc
eName" : "MäuseTastaturen 2.3.16 - Kopie.pdf","fileCreatedOn" : "2016-03-16T21:22:24.085Z","access_permission:assemble_document" : "true","meta:creation-date" : "2016-03-02T13:03:
47Z","lastModified" : "Wed Mar 02 14:03:37 CET 2016","pdf:PDFVersion" : "1.4","X-Parsed-By" : "org.apache.tika.parser.DefaultParser","shareName" : "AppDevData$","access_permission:
can_print_degraded" : "true","xmpTPg:NPages" : "1","createdOn" : "Wed Mar 16 22:22:24 CET 2016","pdf:encrypted" : "false","access_permission:extract_content" : "true","producer" :
"Adobe PSL 1.2e for Canon ","attributes" : "32","Content-Type" : "applica-tion\/pdf","allow_token_document" : ["LDAPConn:S-1-5-21-1751174259-1996115066-1435642685-16152","LDAPConn:S
-1-5-21-1751174259-1996115066-1435642685-16153","LDAPConn:S-1-5-21-1751174259-1996115066-1435642685-7894"],"deny_token_document" : "LDAPConn:DEAD_AUTHORITY","allow_token_share" : "
__nosecurity__","deny_token_share" : "__nosecurity__","allow_token_parent" : "__nosecurity__","deny_token_parent" : "__nosecurity__","content" : ""}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [_source]
        at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:411)
        at org.elasticsearch.index.mapper.internal.SourceFieldMapper.preParse(SourceFieldMapper.java:240)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:540)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493)
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:492)
        at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:192)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase.performOnPrimary(TransportShardReplicationOperationAction.java:574)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1.doRun(TransportShardReplicationOperationAction.java:440)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchParseException: Failed to parse content to map
        at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:130)
        at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:81)
        at org.elasticsearch.index.mapper.internal.SourceFieldMapper.parseCreateField(SourceFieldMapper.java:274)
        at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
        ... 11 more
Caused by: org.elasticsearch.common.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string va
lue
 at [Source: [B@5b774e8b; line: 1, column: 1145]
        at org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.java:1487)
        at org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:518)
        at org.elasticsearch.common.jackson.core.base.ParserMinimalBase._throwUnquotedSpace(ParserMinimalBase.java:482)
        at org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2357)
        at org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2287)
        at org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:286)
        at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:86)
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readValue(AbstractXContentParser.java:293)
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:275)
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readOrderedMap(AbstractXContentParser.java:258)
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrdered(AbstractXContentParser.java:213)
        at org.elasticsearch.common.xcontent.support.AbstractXContentParser.mapOrderedAndClose(AbstractXContentParser.java:228)
        at org.elasticsearch.common.xcontent.XContentHelper.convertToMap(XContentHelper.java:125)
        ... 14 more

 

This happens for documents of different types/extension, such as pdfs as well as xlsx, etc. It seems that Tika sometimes does not remove special characters as the null character 0x0000. The presence of the special characters causes Elasticsearch to omit the indexing of the document. Thus the document is not indexed at all, as  special characters need to be escaped when handed over as a JSON request. Is there a way to work around the problem with the existing functionality of Apache ManifoldCF?

 

Regards
Silvio