Sebastian Nagel created NUTCH-2635:
--------------------------------------
Summary: Generator writes unneeded temporary output
Key: NUTCH-2635
URL: https://issues.apache.org/jira/browse/NUTCH-2635
Project: Nutch
Issue Type: Bug
Components: generator
Affects Versions: 1.15
Reporter: Sebastian Nagel
Fix For: 1.16
Generator writes the temporary output of the Selector job/step twice (see [line
516|https://github.com/apache/nutch/blob/branch-1.15/src/java/org/apache/nutch/crawl/Generator.java#L516]).
Not a big issue when generating small fetch lists but may be when working on
large data. The temporary output looks like:
{noformat}
% tree -h generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/
enerate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/
|-- [4.0K] fetchlist-1
| `-- [ 25M] part-r-00000
`-- [ 77M] part-r-00000
1 directory, 2 files
% file generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/part-r-00000
generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/part-r-00000: ASCII text
% file
generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/fetchlist-1/part-r-00000
generate-temp-fc27fe85-9ddc-4926-b6ba-dcd0066d5007/fetchlist-1/part-r-00000:
Apache Hadoop Sequence file version 6
{noformat}
The unneeded output is plain-text which explains its larger size compared to
the Hadoop Sequence file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
