[
https://issues.apache.org/jira/browse/HADOOP-9196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13555606#comment-13555606
]
Surenkumar Nihalani commented on HADOOP-9196:
---------------------------------------------
It does allocate a full array. I was looking at {{BitSet}}'s implementation.
Getting each bit at a time is too expensive. I am thinking extending {{BitSet}}
for {{BloomFilter}} to have a {{getInt(index)}} to return the internal integer
and write that each int at a time because allocation a huge array just for
writing to {{DataOutput}} seems overkill. It's not like it will be used and
kept around. It will be garbage collected. Passing an integer to {{DataOutput}}
at a time wouldn't be bad because it's buffered.
Thoughts?
> Modify BloomFilter.write() to address memory concerns
> -----------------------------------------------------
>
> Key: HADOOP-9196
> URL: https://issues.apache.org/jira/browse/HADOOP-9196
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: James
> Priority: Minor
>
> It appears that org.apache.hadoop.util.bloom.BloomFilter's write() method
> creates a byte array large enough to fit the entire bit vector into memory
> during serialization. This is unnecessary and may cause out of memory issues
> if the bit vector is sufficiently large and memory is tight.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
