Oh, this memory size: tserver.memory.maps.max 1G -> 20G (looks like this is an overkill, is it?)
tserver.cache.data.size 128M? -> 1024M tserver.cache.index.size 128M? -> 1024M On Thu, Jun 19, 2014 at 12:48 AM, Josh Elser <[email protected]> wrote: > Which memory size? :) > > JVM max heap size? In-memory map size? Either (really, both) would be good > to increase. > > If the situation is as Eric posited, it's very likely that increasing the > resources that Accumulo has available would make it more responsive and > able to keep up with your ingest load. > > - Josh > > > On 6/18/14, 9:44 AM, Jianshi Huang wrote: > >> Does memory size important to reduce this sort of errors? I used the >> default settings (1GB/per tserver), this might be too small. >> >> I increased it to 20GB, and I saw no errors after that. >> >> Jianshi >> >> >> On Thu, Jun 19, 2014 at 12:39 AM, Eric Newton <[email protected] >> <mailto:[email protected]>> wrote: >> >> This error is often a result of overwhelming your server resources. >> >> It basically says "an update came in that was so old, the id used to >> identify the sender has already aged off." >> >> What is your expected ingest rate during the job? What sort of >> resources does accumulo have? >> >> >> On Wed, Jun 18, 2014 at 7:09 AM, Jianshi Huang >> <[email protected] <mailto:[email protected]>> wrote: >> >> Here's the error message I got from the tserver_xxx.log >> >> 2014-06-18 01:06:06,816 [tserver.TabletServer] INFO : Adding 1 >> logs for extent g;cust:2072821;cust:20700111 as alias 37 >> 2014-06-18 01:06:16,286 [thrift.ProcessFunction] ERROR: Internal >> error processing applyUpdates >> java.lang.RuntimeException: No Such SessionID >> at >> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler. >> applyUpdates(TabletServer.java:1522) >> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown >> Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke( >> DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.accumulo.trace.instrument.thrift.TraceWrap$1. >> invoke(TraceWrap.java:63) >> at com.sun.proxy.$Proxy23.applyUpdates(Unknown Source) >> at >> org.apache.accumulo.core.tabletserver.thrift. >> TabletClientService$Processor$applyUpdates.getResult( >> TabletClientService.java:2347\ >> ) >> at >> org.apache.accumulo.core.tabletserver.thrift. >> TabletClientService$Processor$applyUpdates.getResult( >> TabletClientService.java:2333\ >> ) >> at >> org.apache.thrift.ProcessFunction.process( >> ProcessFunction.java:39) >> at >> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) >> at >> org.apache.accumulo.server.util.TServerUtils$ >> TimedProcessor.process(TServerUtils.java:171) >> at >> org.apache.thrift.server.AbstractNonblockingServer$ >> FrameBuffer.invoke(AbstractNonblockingServer.java:478) >> at >> org.apache.accumulo.server.util.TServerUtils$THsHaServer$ >> Invocation.run(TServerUtils.java:231) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:615) >> at >> org.apache.accumulo.trace.instrument.TraceRunnable.run( >> TraceRunnable.java:47) >> at >> org.apache.accumulo.core.util.LoggingRunnable.run( >> LoggingRunnable.java:34) >> at java.lang.Thread.run(Thread.java:724) >> 2014-06-18 01:06:16,287 [thrift.ProcessFunction] ERROR: Internal >> error processing applyUpdates >> java.lang.RuntimeException: No Such SessionID >> at >> org.apache.accumulo.tserver.TabletServer$ThriftClientHandler. >> applyUpdates(TabletServer.java:1522) >> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown >> Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke( >> DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.accumulo.trace.instrument.thrift.TraceWrap$1. >> invoke(TraceWrap.java:63) >> at com.sun.proxy.$Proxy23.applyUpdates(Unknown Source) >> at >> org.apache.accumulo.core.tabletserver.thrift. >> TabletClientService$Processor$applyUpdates.getResult( >> TabletClientService.java:2347\ >> ) >> at >> org.apache.accumulo.core.tabletserver.thrift. >> TabletClientService$Processor$applyUpdates.getResult( >> TabletClientService.java:2333\ >> ) >> at >> org.apache.thrift.ProcessFunction.process( >> ProcessFunction.java:39) >> at >> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) >> at >> org.apache.accumulo.server.util.TServerUtils$ >> TimedProcessor.process(TServerUtils.java:171) >> at >> org.apache.thrift.server.AbstractNonblockingServer$ >> FrameBuffer.invoke(AbstractNonblockingServer.java:478) >> at >> org.apache.accumulo.server.util.TServerUtils$THsHaServer$ >> Invocation.run(TServerUtils.java:231) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:615) >> at >> org.apache.accumulo.trace.instrument.TraceRunnable.run( >> TraceRunnable.java:47) >> at >> org.apache.accumulo.core.util.LoggingRunnable.run( >> LoggingRunnable.java:34) >> at java.lang.Thread.run(Thread.java:724) >> 2014-06-18 01:06:16,287 [util.TServerUtils$THsHaServer] WARN : >> Got an IOException during write! >> java.io.IOException: Connection reset by peer >> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >> at >> sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil. >> java:93) >> at sun.nio.ch.IOUtil.write(IOUtil.java:65) >> at >> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) >> at >> org.apache.thrift.transport.TNonblockingSocket.write( >> TNonblockingSocket.java:164) >> at >> org.apache.thrift.server.AbstractNonblockingServer$ >> FrameBuffer.write(AbstractNonblockingServer.java:381) >> at >> org.apache.thrift.server.AbstractNonblockingServer$ >> AbstractSelectThread.handleWrite(AbstractNonblockingServer.java:220) >> at >> org.apache.thrift.server.TNonblockingServer$ >> SelectAcceptThread.select(TNonblockingServer.java:201) >> >> >> Jianshi >> >> >> On Wed, Jun 18, 2014 at 2:54 PM, Jianshi Huang >> <[email protected] <mailto:[email protected]>> wrote: >> >> I see. I'll check the tablet server log and paste the error >> message in later thread. >> >> BTW, looks like the AccumuloOutputFormat is the cause, I'm >> currently using BatchWriter and it works well. >> >> My code looks like this (it's in Scala as I'm using Spark): >> >> AccumuloOutputFormat.setZooKeeperInstance(job, >> Conf.getString("accumulo.instance"), >> Conf.getString("accumulo.zookeeper.servers")) >> AccumuloOutputFormat.setConnectorInfo(job, >> Conf.getString("accumulo.user"), new >> PasswordToken(Conf.getString("accumulo.password"))) >> AccumuloOutputFormat.setDefaultTableName(job, >> Conf.getString("accumulo.table")) >> >> val paymentRDD: RDD[(Text, Mutation)] = payment.flatMap >> { payment => >> // val key = new Text(Conf.getString("accumulo.table")) >> paymentMutations(payment).map((null, _)) >> } >> >> >> paymentRDD.saveAsNewAPIHadoopFile(Conf. >> getString("accumulo.instance"), >> classOf[Void], classOf[Mutation], >> classOf[AccumuloOutputFormat], job.getConfiguration) >> >> >> It's also possible that saveAsNewAPIHadoopFile doesn't work >> well with AccumuloOutputFormat. >> >> >> >> Jianshi >> >> >> >> >> On Wed, Jun 18, 2014 at 12:45 PM, Josh Elser >> <[email protected] <mailto:[email protected]>> wrote: >> >> Check the TabletServer logs. This Exception is telling >> you that there was an error on the server. You should >> look there for what the real problem was. You can do >> this one of two ways. >> >> 1) Use the "Recent Logs" page on the Accumulo monitor >> (http://accumulo_monitor_host:__50095 >> <http://accumulo_monitor_host:50095>). Unless you >> >> cleared the logs, or restarted the monitor process since >> you got this error, you should be able to see a nice >> HTML view of any errors >> >> 2) Check the debug log, e.g. >> $ACCUMULO_HOME/logs/tserver_$__host.debug.log. If you're >> >> running tservers on more than one node, be sure that you >> check the log files on all nodes. >> >> - Josh >> >> >> On 6/17/14, 9:33 PM, Jianshi Huang wrote: >> >> Hi, >> >> I got the following errors during MapReduce >> ingestion, are they serious >> errors? >> >> java.io.IOException: >> org.apache.accumulo.core.__client.__ >> MutationsRejectedException: >> >> # constraint >> violations : 0 security codes: {} # server\ errors >> 1 # exceptions 0 >> at >> org.apache.accumulo.core.__client.mapreduce.__ >> AccumuloOutputFormat$__AccumuloRecordWriter.write(__ >> AccumuloOutputFormat.java:437) >> at >> org.apache.accumulo.core.__client.mapreduce.__ >> AccumuloOutputFormat$__AccumuloRecordWriter.write(__ >> AccumuloOutputFormat.java:373) >> at >> org.apache.spark.rdd.__PairRDDFunctions.org >> <http://org.apache.spark.rdd.PairRDDFunctions.org> >> <http://org.apache.spark.rdd.__PairRDDFunctions.org >> <http://org.apache.spark.rdd.PairRDDFunctions.org >> >>$apache$__spark$rdd$PairRDDFunctions$$__writeShard$1( >> PairRDDFunctions.__scala:716) >> >> at >> org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__ >> saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730) >> at >> org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__ >> saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730) >> at >> org.apache.spark.scheduler.__ >> ResultTask.runTask(ResultTask.__scala:111) >> at >> org.apache.spark.scheduler.__Task.run(Task.scala:51) >> at >> org.apache.spark.executor.__ >> Executor$TaskRunner.run(__Executor.scala:187) >> at >> java.util.concurrent.__ThreadPoolExecutor.runWorker(_ >> _ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.__ThreadPoolExecutor$Worker.run( >> __ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.__java:724) >> >> >> And >> >> java.io.IOException: >> org.apache.accumulo.core.__client.AccumuloException: >> org.apache.thrift.__TApplicationException: Internal >> error processing\ >> applyUpdates >> at >> org.apache.accumulo.core.__client.mapreduce.__ >> AccumuloOutputFormat.__getRecordWriter(__AccumuloOutputFormat.java:558) >> at >> org.apache.spark.rdd.__PairRDDFunctions.org >> <http://org.apache.spark.rdd.PairRDDFunctions.org> >> <http://org.apache.spark.rdd.__PairRDDFunctions.org >> <http://org.apache.spark.rdd.PairRDDFunctions.org >> >>$apache$__spark$rdd$PairRDDFunctions$$__writeShard$1( >> PairRDDFunctions.__scala:712) >> >> at >> org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__ >> saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730) >> at >> org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__ >> saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730) >> at >> org.apache.spark.scheduler.__ >> ResultTask.runTask(ResultTask.__scala:111) >> at >> org.apache.spark.scheduler.__Task.run(Task.scala:51) >> at >> org.apache.spark.executor.__ >> Executor$TaskRunner.run(__Executor.scala:187) >> at >> java.util.concurrent.__ThreadPoolExecutor.runWorker(_ >> _ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.__ThreadPoolExecutor$Worker.run( >> __ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.__java:724) >> >> >> >> Cheers, >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> >> >> >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> >> >> >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> >> >> >> >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
