[
https://issues.apache.org/jira/browse/SOLR-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681629#action_12681629
]
Fergus McMenemie commented on SOLR-1060:
----------------------------------------
# Correct.
# Great idea I am fine with that; but the next patch I submit, will still do
both. I will work on seperating them once it is working.
# Sort of; it either needs a change or we need a new URLDataSource.
FileDataSource is fine for reading files from filesystems but it does not
handle the file:// syntax. HttpDataSource already half supports reading from
file:// locations (if the URL does not contain the protocol string http:// then
baseUrl can be used to add any protocol!). But after considering the input from
Otis Gospodnetic, Paul Libbrecht and thinking for a bit; It seemed reasonable
that the location of the ChangeList could be specified using file:/// of
http:// syntax as well as using plain old filesystem syntax. Likewise the lines
within the ChangeList could use either URL or filesystem syntax, this in turn
also applied to the baseDir.
# I have been playing about with ChangeListEntityProcessor feeding rows to
XPathEntityProcessor. ChangeListEntityProcessor currently supports use of
either FileDataSource or HttpDataSource by XPathEntityProcessor. However
depending which DataSource is specified within XPathEntityProcessor then
ChangeListEntityProcessor needs to be configured to return rows with the
approriatle syntax. (i Think!)
> a new DIH EnityProcessor allowing text file lists of files to be indexed
> ------------------------------------------------------------------------
>
> Key: SOLR-1060
> URL: https://issues.apache.org/jira/browse/SOLR-1060
> Project: Solr
> Issue Type: New Feature
> Components: contrib - DataImportHandler
> Affects Versions: 1.4
> Reporter: Fergus McMenemie
> Fix For: 1.4
>
> Attachments: SOLR-1060.patch
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> I have finished a new DIH EntityProcessor. It is designed around the idea
> that whatever demon is used to maintain your content store it is likely to
> drop a report or log file explaining what has changed within your content
> store. I wish to use this report file to control the indexing of the new or
> changed content and the removal of old content. The report files, perhaps
> from un-tar or un-zip, are likely to reference jpegs and directory stubs
> which need to be ignored. I assumed a file based content repository but this
> should be expanded to handle URI's as well
> I feel that the current FileListEntityProcessor is poorly named. It should be
> called the dirWalkEntityProcessor or dirCrawlEntityProcessor or such. And
> this new EntityProcessor should have the name FileListEntityProcessor.
> However what is done is done. I then came up with manifestEnityProcessor
> which I thought suited, manifest files are all over the content sets I deal
> with and the dictionary definition seemed close enough ("ships manifest").
> However how about ChangeListEntityProcessor
> {code}
> <entity name="jc"
> processor="ManifestEntityProcessor"
> baseDir="/Volumes/Techmore/ts/aaa/schema/data"
> rootEntity="false"
> dataSource="null"
> allowRegex="^.*\.xml$"
> blockRegex="usc2009"
> manifestFileName="/Volumes/ts/man-find.txt"
> docAddRegex=".*"
> >
> {code}
> The new entity fields are as follows.
>
> *manifestFileName* is the required location of the manifest file. If this
> value is relative, it assumed to be relative to baseDir.
> *allowRegex* is an optional attribute that if present discards any line
> which does not match the regExp
>
> *blockRegex* is an optional attribute that is applied after any allowRegex
> and discards any line which matches the regExp
> *docAddRegex* is a required regex to identify lines which when matched
> should cause docs to be added to the index. As well as matching the line it
> should also return the portion of the line which contains the filepath as
> group(1)
> *docDeleteRegex* is an optional value of a regex to identify documents
> which when matched should be deleted from the index. As well as matching the
> line it should also return the portion of the line which contains the
> filepath as group(1) **PLANNED**
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
