[ 
https://issues.apache.org/jira/browse/NUTCH-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18142294#comment-18142294
 ] 

ASF GitHub Bot commented on NUTCH-3249:
---------------------------------------

sebastian-nagel commented on code in PR #994:
URL: https://github.com/apache/nutch/pull/994#discussion_r4528801897


##########
src/java/org/apache/nutch/tools/CommonCrawlDataDumper.java:
##########
@@ -334,7 +334,8 @@ public void dump(File outputDir, File segmentRootDir, File 
linkdb, boolean gzip,
                         reverseKey, url, timestamp, extensionName, !gzip);
                 outputRelativePath = outputFullPath
                     .substring(0, outputFullPath.lastIndexOf(File.separator) - 
1);
-                filename = content.getMetadata().get(Metadata.DATE) + "."

Review Comment:
   But that's another issue. To be addressed in NUTCH-3251.



##########
src/java/org/apache/nutch/tools/CommonCrawlDataDumper.java:
##########


Review Comment:
   Hm. Either it's `"Date"`...



##########
src/java/org/apache/nutch/tools/CommonCrawlDataDumper.java:
##########
@@ -334,7 +334,8 @@ public void dump(File outputDir, File segmentRootDir, File 
linkdb, boolean gzip,
                         reverseKey, url, timestamp, extensionName, !gzip);
                 outputRelativePath = outputFullPath
                     .substring(0, outputFullPath.lastIndexOf(File.separator) - 
1);
-                filename = content.getMetadata().get(Metadata.DATE) + "."

Review Comment:
   ... or `"date"` which `Metadata.DATE` is set to.
   
   Because Metadata is case-sensitive it cannot work both ways.





> CommonCrawlDataDumper should use safe names for archive entries built from 
> crawled content
> ------------------------------------------------------------------------------------------
>
>                 Key: NUTCH-3249
>                 URL: https://issues.apache.org/jira/browse/NUTCH-3249
>             Project: Nutch
>          Issue Type: Bug
>          Components: commoncrawl
>    Affects Versions: 1.23
>            Reporter: Lewis John McGibbney
>            Priority: Major
>             Fix For: 1.24
>
>
> CommonCrawlDataDumper builds archive entry names from values taken out of 
> crawled content. Those values are not limited to a safe file name, so an 
> entry name can point outside the output directory.
> Whether unpacking writes outside that directory depends on the program that 
> unpacks the archive. The defect is that Nutch writes the unsafe name.
> Limit any crawled value used in an entry name to a single safe path segment.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to