[
https://issues.apache.org/jira/browse/GORA-20?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lewis John McGibbney closed GORA-20.
------------------------------------
Bulk close of issues. Preparation for Gora 0.2 release candidate.
> Flush datastore regularly
> -------------------------
>
> Key: GORA-20
> URL: https://issues.apache.org/jira/browse/GORA-20
> Project: Apache Gora
> Issue Type: New Feature
> Components: storage
> Affects Versions: 0.1-incubating
> Reporter: Alexis
> Fix For: 0.1-incubating
>
> Attachments: gora.patch, mapred-site.xml
>
>
> Right now you need to explicitly call the flush method to make the IO
> operation happen, or close the datastore.
> The issue is described here:
> http://techvineyard.blogspot.com/2010/12/build-nutch-20.html#Free_up_the_memory.
> Click on the image to see it in real size and look at the Heap utilization
> on the top right chart.
> Not everybody has infinite memory. In a Nutch fetch process, I usually run
> into trouble after around 20k urls downloaded because it takes up all the
> memory, the Java Heap space being set to 1G with a system that "only" has 1G
> RAM as well.
> The feature consists of allowing the datastore to be flushed regularly during
> the Hadoop job's reducer, org.apache.gora.mapreduce.GoraReducer. We would
> just add a maxBuffer parameter, which default value is 10000 for example and
> that you can override in org.apache.gora.mapreduce.GoraOutputFormat. It
> indicates the maximum number of records buffered in memory before the next
> flush operation occurs to actually write them in the datastore. This would
> actually be a member of the org.apache.hadoop.mapreduce.RecordWriter
> extension returned by getRecordWriter method.
> An idea of the fix is suggested in the above link.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
