No, the shared buffer on the BatchScanner is on the instance, not a
static member across all instances.
On 11/19/2013 07:41 PM, Tran, An - 0553 - MITLL wrote:
Hi Josh,
That was very helpful. If I have each of my thread in my thread pool
create an Accumulo connector prior to creating a batch scanner will each
of those thread still share the same buffer?
Thanks,
-An
On 11/19/13 7:37 PM, "Josh Elser" <[email protected]> wrote:
An,
The BatchScanner will query multiple servers at once, up to the
numThreads argument you specified when invoking createBatchScanner from
a Connector.
Each of these connections will write data into a shared buffer in
TabletServerBatchReaderIterator. Each server the BatchScanner is talking
to will return up to the configured table property
`table.scan.max.memory` (default 512k) amount of Key-Value pairs.
However, you'll probably not see each thread always returning the full
amount as the tabletserver will time-slice your query to ensure that
your one query thread does not consume all resources.
In short, make you have sufficient JVM heap overhead for the value of
`table.scan.max.memory` of the table you're querying (`config -t
'yourTableName'` in the Accumulo shell) multiplied by the number of
threads you're using with the BatchScanner multiplied by two. Remember
while you are processing the Key-Value pairs that you queried from
Accumulo, it will already be fetching the next batch.
- Josh
On 11/19/2013 07:10 PM, Tran, An - 0553 - MITLL wrote:
Hi,
I am currently getting TabletServerBatchReaderIterator Java heap space
errors. Below are the log outputs. Do any of you know what is going on
and what is the proper solution to fix this issue?
A little detail on the use case is I have an application which uses a
thread pool that does concurrent queries on Accumulo. Any insight on
this issue is greatly appreciated .
Thanks.
19 Nov 2013 14:13:06,845 52142 [batch scanner 7-7 looking up 4 ranges
at 192.168.1.66:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,845 52142 [batch scanner 7-6 looking up 5 ranges
at 192.168.1.68:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,844 52141 [batch scanner 7-3 looking up 3 ranges
at 192.168.1.69:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,844 52141 [batch scanner 7-2 looking up 3 ranges
at 192.168.1.63:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,843 52140 [batch scanner 6-1 looking up 5 ranges
at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,846 52143 [batch scanner 7-4 looking up 7 ranges
at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run
Java heap space
java.lang.OutOfMemoryError: Java heap space
19 Nov 2013 14:13:06,846 52143 [batch scanner 6-3 looking up 5 ranges
at 192.168.1.64:9997] WARN TabletServerBatchReaderIterator run
Java heap space
ava.lang.OutOfMemoryError: Java heap space
at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLo
okup(TabletServerBatchReaderIterator.java:564)
at
org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$Quer
yTask.run(TabletServerBatchReaderIterator.java:338)
at
org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable
.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
a:615)
at java.lang.Thread.run(Thread.java:724)
