[
https://issues.apache.org/jira/browse/LUCENE-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511251#comment-13511251
]
Shai Erera commented on LUCENE-4590:
------------------------------------
Do you think perhaps that EnwikiContentSource should let the caller know
whether the returned DocData represents a content page or category page? In
general, for text indexing benchmarks, I don't think that indexing the category
pages adds much value, because they are very short and will often not come back
as a result to any query. Rather, their content seems to represent the
Wikipedia's taxonomy.
So maybe, if someone wants to generate a line file from the pages only,
EnwikiContentSource can support filtering out category pages entirely. That
will allow the flexibility that I think you are trying to achieve:
* If discardCategories = true, WriteLineDocTask will write just the content
pages. Otherwise, it will write both.
* WriteEnwikiLineDocTask will detect the page type from a specialized DocData
and know easily to which file to write it.
> WriteEnwikiLineDoc which writes Wikipedia category pages to a separate file
> ---------------------------------------------------------------------------
>
> Key: LUCENE-4590
> URL: https://issues.apache.org/jira/browse/LUCENE-4590
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/benchmark
> Reporter: Doron Cohen
> Assignee: Doron Cohen
> Priority: Minor
>
> It may be convenient to split Wikipedia's line file into two separate files:
> category-pages and non-category ones.
> It is possible to split the original line file with grep or such.
> It is more efficient to do it in advance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
