Hi Josh, That was very helpful. If I have each of my thread in my thread pool create an Accumulo connector prior to creating a batch scanner will each of those thread still share the same buffer?
Thanks, -An On 11/19/13 7:37 PM, "Josh Elser" <[email protected]> wrote: >An, > >The BatchScanner will query multiple servers at once, up to the >numThreads argument you specified when invoking createBatchScanner from >a Connector. > >Each of these connections will write data into a shared buffer in >TabletServerBatchReaderIterator. Each server the BatchScanner is talking >to will return up to the configured table property >`table.scan.max.memory` (default 512k) amount of Key-Value pairs. >However, you'll probably not see each thread always returning the full >amount as the tabletserver will time-slice your query to ensure that >your one query thread does not consume all resources. > >In short, make you have sufficient JVM heap overhead for the value of >`table.scan.max.memory` of the table you're querying (`config -t >'yourTableName'` in the Accumulo shell) multiplied by the number of >threads you're using with the BatchScanner multiplied by two. Remember >while you are processing the Key-Value pairs that you queried from >Accumulo, it will already be fetching the next batch. > >- Josh > >On 11/19/2013 07:10 PM, Tran, An - 0553 - MITLL wrote: >> Hi, >> >> I am currently getting TabletServerBatchReaderIterator Java heap space >>errors. Below are the log outputs. Do any of you know what is going on >>and what is the proper solution to fix this issue? >> A little detail on the use case is I have an application which uses a >>thread pool that does concurrent queries on Accumulo. Any insight on >>this issue is greatly appreciated . >> >> Thanks. >> >> 19 Nov 2013 14:13:06,845 52142 [batch scanner 7-7 looking up 4 ranges >>at 192.168.1.66:9997] WARN TabletServerBatchReaderIterator run >> Java heap space >> java.lang.OutOfMemoryError: Java heap space >> 19 Nov 2013 14:13:06,845 52142 [batch scanner 7-6 looking up 5 ranges >>at 192.168.1.68:9997] WARN TabletServerBatchReaderIterator run >> Java heap space >> java.lang.OutOfMemoryError: Java heap space >> 19 Nov 2013 14:13:06,844 52141 [batch scanner 7-3 looking up 3 ranges >>at 192.168.1.69:9997] WARN TabletServerBatchReaderIterator run >> Java heap space >> java.lang.OutOfMemoryError: Java heap space >> 19 Nov 2013 14:13:06,844 52141 [batch scanner 7-2 looking up 3 ranges >>at 192.168.1.63:9997] WARN TabletServerBatchReaderIterator run >> Java heap space >> java.lang.OutOfMemoryError: Java heap space >> 19 Nov 2013 14:13:06,843 52140 [batch scanner 6-1 looking up 5 ranges >>at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run >> Java heap space >> java.lang.OutOfMemoryError: Java heap space >> 19 Nov 2013 14:13:06,846 52143 [batch scanner 7-4 looking up 7 ranges >>at 192.168.1.67:9997] WARN TabletServerBatchReaderIterator run >> Java heap space >> java.lang.OutOfMemoryError: Java heap space >> 19 Nov 2013 14:13:06,846 52143 [batch scanner 6-3 looking up 5 ranges >>at 192.168.1.64:9997] WARN TabletServerBatchReaderIterator run >> Java heap space >> ava.lang.OutOfMemoryError: Java heap space >> at >>org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator.doLo >>okup(TabletServerBatchReaderIterator.java:564) >> at >>org.apache.accumulo.core.client.impl.TabletServerBatchReaderIterator$Quer >>yTask.run(TabletServerBatchReaderIterator.java:338) >> at >>org.apache.accumulo.cloudtrace.instrument.TraceRunnable.run(TraceRunnable >>.java:47) >> at >>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java >>:1145) >> at >>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav >>a:615) >> at java.lang.Thread.run(Thread.java:724) >> >> >
