[
https://issues.apache.org/jira/browse/NUTCH-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140800#comment-18140800
]
ASF GitHub Bot commented on NUTCH-3249:
---------------------------------------
lewismc opened a new pull request, #994:
URL: https://github.com/apache/nutch/pull/994
Tracked in [NUTCH-3249](https://issues.apache.org/jira/browse/NUTCH-3249).
Limit the crawled `date` metadata value used in `CommonCrawlDataDumper`
archive entry names to a single path segment. Normal HTTP date values are
unchanged, and a missing value keeps the existing filename.
Adds `testEpochGzipEntryNamesIgnoreUnsafeDateMetadata` to
`TestCommonCrawlDataDumper` for the gzip and epoch filename output.
Testing:
- `ant clean test-core -Dtestcase=TestCommonCrawlDataDumper`
- `ant clean test`
> CommonCrawlDataDumper should use safe names for archive entries built from
> crawled content
> ------------------------------------------------------------------------------------------
>
> Key: NUTCH-3249
> URL: https://issues.apache.org/jira/browse/NUTCH-3249
> Project: Nutch
> Issue Type: Bug
> Components: commoncrawl
> Affects Versions: 1.23
> Reporter: Lewis John McGibbney
> Priority: Major
> Fix For: 1.24
>
>
> CommonCrawlDataDumper builds archive entry names from values taken out of
> crawled content. Those values are not limited to a safe file name, so an
> entry name can point outside the output directory.
> Whether unpacking writes outside that directory depends on the program that
> unpacks the archive. The defect is that Nutch writes the unsafe name.
> Limit any crawled value used in an entry name to a single safe path segment.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)