An,
The BatchScanner will query multiple servers at once, up to the
numThreads argument you specified when invoking createBatchScanner from
a Connector.
Each of these connections will write data into a shared buffer in
TabletServerBatchReaderIterator. Each server the BatchScanner is talking
to will return up to the configured table property
`table.scan.max.memory` (default 512k) amount of Key-Value pairs.
However, you'll probably not see each thread always returning the full
amount as the tabletserver will time-slice your query to ensure that
your one query thread does not consume all resources.
In short, make you have sufficient JVM heap overhead for the value of
`table.scan.max.memory` of the table you're querying (`config -t
'yourTableName'` in the Accumulo shell) multiplied by the number of
threads you're using with the BatchScanner multiplied by two. Remember
while you are processing the Key-Value pairs that you queried from
Accumulo, it will already be fetching the next batch.
- Josh
On 11/19/2013 07:10 PM, Tran, An - 0553 - MITLL wrote:
Hi,
I am currently getting TabletServerBatchReaderIterator Java heap space errors.
Below are the log outputs. Do any of you know what is going on and what is
the proper solution to fix this issue?
A little detail on the use case is I have an application which uses a thread
pool that does concurrent queries on Accumulo. Any insight on this issue is
greatly appreciated .
Thanks.
19 Nov 2013 14:13:06,845 52142 [batch scanner 7-7 looking up 4 ranges at
192.168.1.66:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,845 52142 [batch scanner 7-6 looking up 5 ranges at
192.168.1.68:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,844 52141 [batch scanner 7-3 looking up 3 ranges at
192.168.1.69:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,844 52141 [batch scanner 7-2 looking up 3 ranges at
192.168.1.63:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,843 52140 [batch scanner 6-1 looking up 5 ranges at
192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,846 52143 [batch scanner 7-4 looking up 7 ranges at
192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,846 52143 [batch scanner 6-3 looking up 5 ranges at
192.168.1.64:9997] WARN TabletServerBatchReaderIterator run
Java heap space
ava.lang.OutOfMemoryError: Java heap space
at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLookup(TabletServerBatchReaderIterator.java:564)
at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:338)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
