Which memory size? :)
JVM max heap size? In-memory map size? Either (really, both) would be
good to increase.
If the situation is as Eric posited, it's very likely that increasing
the resources that Accumulo has available would make it more responsive
and able to keep up with your ingest load.
- Josh
On 6/18/14, 9:44 AM, Jianshi Huang wrote:
Does memory size important to reduce this sort of errors? I used the
default settings (1GB/per tserver), this might be too small.
I increased it to 20GB, and I saw no errors after that.
Jianshi
On Thu, Jun 19, 2014 at 12:39 AM, Eric Newton <[email protected]
<mailto:[email protected]>> wrote:
This error is often a result of overwhelming your server resources.
It basically says "an update came in that was so old, the id used to
identify the sender has already aged off."
What is your expected ingest rate during the job? What sort of
resources does accumulo have?
On Wed, Jun 18, 2014 at 7:09 AM, Jianshi Huang
<[email protected] <mailto:[email protected]>> wrote:
Here's the error message I got from the tserver_xxx.log
2014-06-18 01:06:06,816 [tserver.TabletServer] INFO : Adding 1
logs for extent g;cust:2072821;cust:20700111 as alias 37
2014-06-18 01:06:16,286 [thrift.ProcessFunction] ERROR: Internal
error processing applyUpdates
java.lang.RuntimeException: No Such SessionID
at
org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.applyUpdates(TabletServer.java:1522)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
at com.sun.proxy.$Proxy23.applyUpdates(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$applyUpdates.getResult(TabletClientService.java:2347\
)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$applyUpdates.getResult(TabletClientService.java:2333\
)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:171)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:231)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:724)
2014-06-18 01:06:16,287 [thrift.ProcessFunction] ERROR: Internal
error processing applyUpdates
java.lang.RuntimeException: No Such SessionID
at
org.apache.accumulo.tserver.TabletServer$ThriftClientHandler.applyUpdates(TabletServer.java:1522)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.accumulo.trace.instrument.thrift.TraceWrap$1.invoke(TraceWrap.java:63)
at com.sun.proxy.$Proxy23.applyUpdates(Unknown Source)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$applyUpdates.getResult(TabletClientService.java:2347\
)
at
org.apache.accumulo.core.tabletserver.thrift.TabletClientService$Processor$applyUpdates.getResult(TabletClientService.java:2333\
)
at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.accumulo.server.util.TServerUtils$TimedProcessor.process(TServerUtils.java:171)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
at
org.apache.accumulo.server.util.TServerUtils$THsHaServer$Invocation.run(TServerUtils.java:231)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at
org.apache.accumulo.trace.instrument.TraceRunnable.run(TraceRunnable.java:47)
at
org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34)
at java.lang.Thread.run(Thread.java:724)
2014-06-18 01:06:16,287 [util.TServerUtils$THsHaServer] WARN :
Got an IOException during write!
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at
org.apache.thrift.transport.TNonblockingSocket.write(TNonblockingSocket.java:164)
at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.write(AbstractNonblockingServer.java:381)
at
org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleWrite(AbstractNonblockingServer.java:220)
at
org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:201)
Jianshi
On Wed, Jun 18, 2014 at 2:54 PM, Jianshi Huang
<[email protected] <mailto:[email protected]>> wrote:
I see. I'll check the tablet server log and paste the error
message in later thread.
BTW, looks like the AccumuloOutputFormat is the cause, I'm
currently using BatchWriter and it works well.
My code looks like this (it's in Scala as I'm using Spark):
AccumuloOutputFormat.setZooKeeperInstance(job,
Conf.getString("accumulo.instance"),
Conf.getString("accumulo.zookeeper.servers"))
AccumuloOutputFormat.setConnectorInfo(job,
Conf.getString("accumulo.user"), new
PasswordToken(Conf.getString("accumulo.password")))
AccumuloOutputFormat.setDefaultTableName(job,
Conf.getString("accumulo.table"))
val paymentRDD: RDD[(Text, Mutation)] = payment.flatMap
{ payment =>
// val key = new Text(Conf.getString("accumulo.table"))
paymentMutations(payment).map((null, _))
}
paymentRDD.saveAsNewAPIHadoopFile(Conf.getString("accumulo.instance"),
classOf[Void], classOf[Mutation],
classOf[AccumuloOutputFormat], job.getConfiguration)
It's also possible that saveAsNewAPIHadoopFile doesn't work
well with AccumuloOutputFormat.
Jianshi
On Wed, Jun 18, 2014 at 12:45 PM, Josh Elser
<[email protected] <mailto:[email protected]>> wrote:
Check the TabletServer logs. This Exception is telling
you that there was an error on the server. You should
look there for what the real problem was. You can do
this one of two ways.
1) Use the "Recent Logs" page on the Accumulo monitor
(http://accumulo_monitor_host:__50095
<http://accumulo_monitor_host:50095>). Unless you
cleared the logs, or restarted the monitor process since
you got this error, you should be able to see a nice
HTML view of any errors
2) Check the debug log, e.g.
$ACCUMULO_HOME/logs/tserver_$__host.debug.log. If you're
running tservers on more than one node, be sure that you
check the log files on all nodes.
- Josh
On 6/17/14, 9:33 PM, Jianshi Huang wrote:
Hi,
I got the following errors during MapReduce
ingestion, are they serious
errors?
java.io.IOException:
org.apache.accumulo.core.__client.__MutationsRejectedException:
# constraint
violations : 0 security codes: {} # server\ errors
1 # exceptions 0
at
org.apache.accumulo.core.__client.mapreduce.__AccumuloOutputFormat$__AccumuloRecordWriter.write(__AccumuloOutputFormat.java:437)
at
org.apache.accumulo.core.__client.mapreduce.__AccumuloOutputFormat$__AccumuloRecordWriter.write(__AccumuloOutputFormat.java:373)
at
org.apache.spark.rdd.__PairRDDFunctions.org
<http://org.apache.spark.rdd.PairRDDFunctions.org>
<http://org.apache.spark.rdd.__PairRDDFunctions.org
<http://org.apache.spark.rdd.PairRDDFunctions.org>>$apache$__spark$rdd$PairRDDFunctions$$__writeShard$1(PairRDDFunctions.__scala:716)
at
org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
at
org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
at
org.apache.spark.scheduler.__ResultTask.runTask(ResultTask.__scala:111)
at
org.apache.spark.scheduler.__Task.run(Task.scala:51)
at
org.apache.spark.executor.__Executor$TaskRunner.run(__Executor.scala:187)
at
java.util.concurrent.__ThreadPoolExecutor.runWorker(__ThreadPoolExecutor.java:1145)
at
java.util.concurrent.__ThreadPoolExecutor$Worker.run(__ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.__java:724)
And
java.io.IOException:
org.apache.accumulo.core.__client.AccumuloException:
org.apache.thrift.__TApplicationException: Internal
error processing\
applyUpdates
at
org.apache.accumulo.core.__client.mapreduce.__AccumuloOutputFormat.__getRecordWriter(__AccumuloOutputFormat.java:558)
at
org.apache.spark.rdd.__PairRDDFunctions.org
<http://org.apache.spark.rdd.PairRDDFunctions.org>
<http://org.apache.spark.rdd.__PairRDDFunctions.org
<http://org.apache.spark.rdd.PairRDDFunctions.org>>$apache$__spark$rdd$PairRDDFunctions$$__writeShard$1(PairRDDFunctions.__scala:712)
at
org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
at
org.apache.spark.rdd.__PairRDDFunctions$$anonfun$__saveAsNewAPIHadoopDataset$1.__apply(PairRDDFunctions.scala:__730)
at
org.apache.spark.scheduler.__ResultTask.runTask(ResultTask.__scala:111)
at
org.apache.spark.scheduler.__Task.run(Task.scala:51)
at
org.apache.spark.executor.__Executor$TaskRunner.run(__Executor.scala:187)
at
java.util.concurrent.__ThreadPoolExecutor.runWorker(__ThreadPoolExecutor.java:1145)
at
java.util.concurrent.__ThreadPoolExecutor$Worker.run(__ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.__java:724)
Cheers,
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
