[ 
https://issues.apache.org/jira/browse/NUTCH-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140800#comment-18140800
 ] 

ASF GitHub Bot commented on NUTCH-3249:
---------------------------------------

lewismc opened a new pull request, #994:
URL: https://github.com/apache/nutch/pull/994

   Tracked in [NUTCH-3249](https://issues.apache.org/jira/browse/NUTCH-3249).
   
   Limit the crawled `date` metadata value used in `CommonCrawlDataDumper` 
archive entry names to a single path segment. Normal HTTP date values are 
unchanged, and a missing value keeps the existing filename.
   
   Adds `testEpochGzipEntryNamesIgnoreUnsafeDateMetadata` to 
`TestCommonCrawlDataDumper` for the gzip and epoch filename output.
   
   Testing:
   - `ant clean test-core -Dtestcase=TestCommonCrawlDataDumper`
   - `ant clean test`




> CommonCrawlDataDumper should use safe names for archive entries built from 
> crawled content
> ------------------------------------------------------------------------------------------
>
>                 Key: NUTCH-3249
>                 URL: https://issues.apache.org/jira/browse/NUTCH-3249
>             Project: Nutch
>          Issue Type: Bug
>          Components: commoncrawl
>    Affects Versions: 1.23
>            Reporter: Lewis John McGibbney
>            Priority: Major
>             Fix For: 1.24
>
>
> CommonCrawlDataDumper builds archive entry names from values taken out of 
> crawled content. Those values are not limited to a safe file name, so an 
> entry name can point outside the output directory.
> Whether unpacking writes outside that directory depends on the program that 
> unpacks the archive. The defect is that Nutch writes the unsafe name.
> Limit any crawled value used in an entry name to a single safe path segment.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to