> > I realized this could be due to an inability by the JVM to create > additional native threads
You may need to increase the nproc limit on your systems. -Eric On Wed, Oct 1, 2014 at 11:12 AM, Geoffry Roberts <[email protected]> wrote: > Thanks for the response. > > The only reason I was creating q new BatchWriter periodically was to > determine if BatchWriter was holding on to memory even after a flush. I > had memory on my BatchWriterConfig set to 1M already. I am reading my RDB > tables in pages of 10K rows. > > Bumping up the JVM size didn't help, > > I tried setting -XX:+HeapDumpOnOutOfMemoryError and when it did not > produce any output (hprof file). I realized this could be due to an > inability by the JVM to create additional native threads. > > What I now think is the problem is not with Acc directly but hiding out on > the JDBC side. > > Perhaps this is not an Acc issue at all but merely masquerading as one. > We'll see. > > > On Tue, Sep 30, 2014 at 12:17 PM, Josh Elser <[email protected]> wrote: > >> You shouldn't have to create a new BatchWriter -- have you tried reducing >> the amount of memory the BatchWriter will use? It keeps a cache internally >> to try to do an amortization of Mutations to send to a given tabletserver. >> >> To limit this memory, use the BatchWriterConfig#setMaxMemory(long) >> method. By default, the maxMemory value is set to 50MB. Reducing this may >> be enough to hold less data in your client and give you some more head room. >> >> Alternatively, you could give your client JVM some more heap :) >> >> >> Geoffry Roberts wrote: >> >>> I am try to pump some data into Accumulo but I keep encountering >>> >>> Exception in thread "Thrift Connection Pool Checker" >>> java.lang.OutOfMemoryError: Java heap space >>> >>> at java.util.HashMap.newValueIterator(HashMap.java:971) >>> >>> at java.util.HashMap$Values.iterator(HashMap.java:1038) >>> >>> at >>> org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer. >>> closeConnections(ThriftTransportPool.java:103) >>> >>> at >>> org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer. >>> run(ThriftTransportPool.java:147) >>> >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> I tried, as a work around, creating a new BatchWriter and closing the >>> old one every ten thousand rows, but to no avail. Data gets written up >>> to the 200kth row, then the error. >>> >>> I have a table of 8M rows in a RDB that I am pumping into Acc via a >>> groovy script. The rows are narrow, a short text field and four floats. >>> >>> I googled of course but nothing was helpful. What can be done? >>> >>> Thanks so much. >>> >>> -- >>> There are ways and there are ways, >>> >>> Geoffry Roberts >>> >> > > > -- > There are ways and there are ways, > > Geoffry Roberts >
