[
https://issues.apache.org/jira/browse/NUTCH-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma updated NUTCH-2229:
---------------------------------
Description:
CrawlDatum allows Jexl expressions on its metadata fields nicely, but it lacks
the opportunity to select on attributes like fetchTime and modifiedTime.
This includes a rudimentary date parser only supporting the
yyyy-MM-dd'T'HH:mm:ss'Z' format:
Dump everything with a modifiedTime higher than 2016-03-20T00:00:00Z
{code}
bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(modifiedTime >
2016-03-20T00:00:00Z)"
{code}
Dump everything that is an HTML file
{code}
bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(Content_Type ==
'text/html' || Content_Type == 'application/xhtml+xml')"
{code}
Keep in mind:
* Jexl doesn't allow a hyphen/minus in field identifier, they are transformed
to underscores
* string literals must be in quotes, only surrounding qoute needs to be escaped
by backslash
was:
CrawlDatum allows Jexl expressions on its metadata fields nicely, but it lacks
the opportunity to select on attributes like fetchTime and modifiedTime.
This includes a rudimentary date parser only supporting the
yyyy-MM-dd'T'HH:mm:ss'Z' format:
{code}
bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(modifiedTime >
2016-03-20T00:00:00Z)"
{code}
> Allow Jexl expressions on CrawlDatum's fixed attributes
> -------------------------------------------------------
>
> Key: NUTCH-2229
> URL: https://issues.apache.org/jira/browse/NUTCH-2229
> Project: Nutch
> Issue Type: Improvement
> Components: crawldb
> Affects Versions: 1.11
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.12
>
> Attachments: NUTCH-2229.patch
>
>
> CrawlDatum allows Jexl expressions on its metadata fields nicely, but it
> lacks the opportunity to select on attributes like fetchTime and modifiedTime.
> This includes a rudimentary date parser only supporting the
> yyyy-MM-dd'T'HH:mm:ss'Z' format:
> Dump everything with a modifiedTime higher than 2016-03-20T00:00:00Z
> {code}
> bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(modifiedTime >
> 2016-03-20T00:00:00Z)"
> {code}
> Dump everything that is an HTML file
> {code}
> bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(Content_Type ==
> 'text/html' || Content_Type == 'application/xhtml+xml')"
> {code}
> Keep in mind:
> * Jexl doesn't allow a hyphen/minus in field identifier, they are transformed
> to underscores
> * string literals must be in quotes, only surrounding qoute needs to be
> escaped by backslash
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
