[
https://issues.apache.org/jira/browse/GORA-20?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13696504#comment-13696504
]
Hudson commented on GORA-20:
----------------------------
Integrated in goraOracle #6 (See [https://builds.apache.org/job/goraOracle/6/])
GORA-20 Flush datastore regularly (Revision
cdacd2965c86067fb37a4e335d0663c9b5f769b4)
GORA-203 Bug in setting column field attribute qualifier in CassandraMapping
(Revision e7bb428cfa979ad49f355f8bc85ba98b50eb34f3)
Result = FAILURE
jnioche :
Files :
* gora-core/src/main/java/org/apache/gora/mapreduce/GoraRecordWriter.java
* gora-core/src/main/java/org/apache/gora/mapreduce/GoraOutputFormat.java
* CHANGES.txt
lewismc :
Files :
* CHANGES.txt
*
gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMapping.java
*
gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraMappingManager.java
> Flush datastore regularly
> -------------------------
>
> Key: GORA-20
> URL: https://issues.apache.org/jira/browse/GORA-20
> Project: Apache Gora
> Issue Type: New Feature
> Components: storage
> Affects Versions: 0.1-incubating
> Reporter: Alexis
> Fix For: 0.1-incubating
>
> Attachments: gora.patch, mapred-site.xml
>
>
> Right now you need to explicitly call the flush method to make the IO
> operation happen, or close the datastore.
> The issue is described here:
> http://techvineyard.blogspot.com/2010/12/build-nutch-20.html#Free_up_the_memory.
> Click on the image to see it in real size and look at the Heap utilization
> on the top right chart.
> Not everybody has infinite memory. In a Nutch fetch process, I usually run
> into trouble after around 20k urls downloaded because it takes up all the
> memory, the Java Heap space being set to 1G with a system that "only" has 1G
> RAM as well.
> The feature consists of allowing the datastore to be flushed regularly during
> the Hadoop job's reducer, org.apache.gora.mapreduce.GoraReducer. We would
> just add a maxBuffer parameter, which default value is 10000 for example and
> that you can override in org.apache.gora.mapreduce.GoraOutputFormat. It
> indicates the maximum number of records buffered in memory before the next
> flush operation occurs to actually write them in the datastore. This would
> actually be a member of the org.apache.hadoop.mapreduce.RecordWriter
> extension returned by getRecordWriter method.
> An idea of the fix is suggested in the above link.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
