[
https://issues.apache.org/jira/browse/GORA-210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13587207#comment-13587207
]
Roland commented on GORA-210:
-----------------------------
Hi Renato,
no, this just fixes this ConcurrentModificationException, the
NullPointerException is still there, thats my next point I'm trying to nail
down.
And yes, the original scope was the "InvalidRequestException(why:column name
must not be empty)" thing, but during adding debug info to the classes and
running inside jdb, the other exceptions where in the way of debugging this one
:)
So I hope to get back to the InvalidRequestException some time...
Regarding this patch: as long as you don't iterate over the buffer (what the
original code already tries to circumvent) it should be safe now.
The problem was: without the "Collections.synchronizedMap()", even simple code
like this:
{code}
Set<K> keys = this.buffer.keySet();
// this duplicates memory footprint
K[] keyArray = (K[]) keys.toArray();
{code}
might fail in a threaded environment.
> thread safety
> -------------
>
> Key: GORA-210
> URL: https://issues.apache.org/jira/browse/GORA-210
> Project: Apache Gora
> Issue Type: Bug
> Components: storage-cassandra
> Affects Versions: 0.2
> Environment: nutch 2.1 / cassandra 1.2.1 / gora-cassandra 0.2 /
> gora-core 0.2.1
> running fetch with parse=true
> fetcher.threads.per.queue>1
> Reporter: Roland
> Priority: Critical
> Labels: patch
> Attachments: GORA-210.patch
>
>
> This is the result of debugging one of my issues described in NUTCH-1534.
> I think there is a wrong assumpation about thread safety of LinkedHashMap, it
> is not enough to not iterate over the buffer (which is a LinkedHashMap).
> My patch fixes this error for me:
> java.util.ConcurrentModificationException
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
> at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:405)
> at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
> at
> org.apache.gora.cassandra.store.CassandraStore.flush(CassandraStore.java:200)
> at
> org.apache.gora.mapreduce.GoraRecordWriter.write(GoraRecordWriter.java:65)
> at
> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:587)
> at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at
> org.apache.nutch.fetcher.FetcherReducer$FetcherThread.output(FetcherReducer.java:664)
> at
> org.apache.nutch.fetcher.FetcherReducer$FetcherThread.run(FetcherReducer.java:534)
> It may not be perfect from a performance point of view...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
