The bytecode manipulation appears to be a promising lead. We have had NewRelic and Jolokia in place for a long time, both of which use Java agents to perform bytecode instrumentation. I can't explain why this problem suddenly appeared (I suspect but have not confirmed it is somehow related to the newer JVM version). The error looks to have cleared up when we remove these.
Thank you! On Tuesday, June 20, 2017 at 10:28:29 AM UTC-4, Ionuț G. Stan wrote: > > Do you use any tool that performs bytecode instrumentation? From your > error message it appears there's try/catch block in there, but there's > no such thing in the source code for akka-actor_2.11:2.4.10: > > > https://github.com/akka/akka/blob/v2.4.10/akka-actor/src/main/scala/akka/dispatch/AbstractDispatcher.scala#L27 > > > If yes, that tool might be the culprit. > > On 20/06/2017 16:53, Ionuț G. Stan wrote: > > What's the actual akka-actor version being used there? I've investigated > > a bit and it doesn't seem to be akka-actor_2.11:2.4.10. The stackmap > > frames are different I'm seeing with 2.4.10 are different than what's > > show in your stack trace. > > > > Once a colleague bumped into a similar issue with Scala and it solved it > > by upgrading the JVM. Not sure what's the problem in your case. > > > > For a bit of background. The JVM is performing some bytecode security > > checks when it loads .class files — to ensure there's no possibility for > > stack underflows or overflows in the generated bytecode, or that the > > arguments on stack have the types expected by the method receiving them, > > etc. > > > > Before JVM 1.7 this was a thing done entirely by the JVM. After 1.7, > > compilers are sometimes required to emit "proofs" that the code they > > generated is behaving correctly. These proofs are the StackMapTables in > > the .class files. > > > > Apparently, your JVM verified whether the emitted proofs are valid and > > figured out they're not. This can happen when there's a mismatch between > > the JVM algorithm that performs the verification and the compiler > > algorithm that generates the proofs. Which one is in your case I have no > > idea. > > > > > > On 19/06/2017 23:36, Matlik wrote: > >> I'm encountering this issue which I've never seen before, and was > >> wondering if someone could point me in the right direction. I have > some > >> Akka streams 2.4.10 code (with Scala 2.11.8) that is embedded into a > >> GlassFish 4.0 server running Oracle Java 1.8.0_131-b11. All this is > >> wrapped into a Docker container for deployment that uses the Ubuntu as > >> the base image. Things were all working great for a while, but > something > >> appears to have changed that is causing us some grief. > >> > >> The docker image is built with the latest Oracle Java 8 JDK and base OS > >> (Ubuntu 14.04). When we needed to rebuild the image with a minor > >> configuration change, we are now consistently getting the following > error: > >> > >> [#|2017-06-19T19:15:09.770+0000|WARNING|glassfish > >> > 4.0|javax.enterprise.web|_ThreadID=45;_ThreadName=http-listener-1(1);_TimeMillis=1497899709770;_LevelValue=900;| > > > >> StandardWrapperValve[TimeSeriesServlet]: Servlet.service() for > >> servlet TimeSeriesServlet threw exception > >> java.lang.VerifyError: Stack map does not match the one at exception > >> handler 98 > >> Exception Details: > >> Location: > >> > >> > akka/dispatch/Envelope.<init>(Ljava/lang/Object;Lakka/actor/ActorRef;)V > @98: athrow > >> Reason: > >> Current frame's flags are not assignable to stack map frame's. > >> Current Frame: > >> bci: @0 > >> flags: { flagThisUninit } > >> locals: { uninitializedThis, 'java/lang/Object', > >> 'akka/actor/ActorRef' } > >> stack: { 'java/lang/Throwable' } > >> Stackmap Frame: > >> bci: @98 > >> flags: { } > >> locals: { top, 'java/lang/Object', 'akka/actor/ActorRef' } > >> stack: { 'java/lang/Throwable' } > >> Bytecode: > >> 0x0000000: 2a2b b500 322a 2cb5 0037 2ab7 0085 014e > >> 0x0000010: b200 8703 3201 0105 bd00 0859 032a 5359 > >> 0x0000020: 0405 bd00 0859 032b 5359 042c 5353 b900 > >> 0x0000030: 8d04 004e a700 053a 052a b800 932d c600 > >> 0x0000040: 232d 1295 0105 bd00 0859 0311 00b1 bb00 > >> 0x0000050: 975a 5fb7 009a 5359 0401 53b9 008d 0400 > >> 0x0000060: 57b1 bf > >> Exception Handler Table: > >> bci [16, 52] => handler: 55 > >> bci [0, 98] => handler: 98 > >> Stackmap Table: > >> > >> > > full_frame(@55,{Object[#6],Object[#8],Object[#120],Object[#8]},{Object[#130]}) > > > >> same_frame(@57) > >> same_frame(@97) > >> full_frame(@98,{Top,Object[#8],Object[#120]},{Object[#130]}) > >> at akka.dispatch.Envelope$.apply(AbstractDispatcher.scala:27) > >> at akka.actor.Cell$class.sendMessage(ActorCell.scala:295) > >> at akka.actor.ActorCell.sendMessage(ActorCell.scala:374) > >> at > >> akka.actor.RepointableActorRef.$bang(RepointableActorRef.scala:171) > >> at akka.actor.ActorRef.tell(ActorRef.scala:128) > >> at > >> > akka.pattern.AskableActorRef$.internalAsk$extension(AskSupport.scala:295) > >> at > >> akka.pattern.AskableActorRef$.$qmark$extension1(AskSupport.scala:281) > >> at > >> > akka.event.LoggingBus$class.akka$event$LoggingBus$$addLogger(Logging.scala:181) > > > >> at > >> > akka.event.LoggingBus$$anonfun$4$$anonfun$apply$4.apply(Logging.scala:116) > >> at > >> > akka.event.LoggingBus$$anonfun$4$$anonfun$apply$4.apply(Logging.scala:115) > >> at scala.util.Success$$anonfun$map$1.apply(Try.scala:237) > >> at scala.util.Try$.apply(Try.scala:192) > >> at scala.util.Success.map(Try.scala:237) > >> at akka.event.LoggingBus$$anonfun$4.apply(Logging.scala:115) > >> at akka.event.LoggingBus$$anonfun$4.apply(Logging.scala:112) > >> at > >> > scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683) > > > >> at > scala.collection.Iterator$class.foreach(Iterator.scala:893) > >> at > scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > >> at > >> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > >> at > scala.collection.AbstractIterable.foreach(Iterable.scala:54) > >> at > >> > scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682) > >> at > >> akka.event.LoggingBus$class.startDefaultLoggers(Logging.scala:112) > >> at > >> > scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:683) > > > >> at > scala.collection.Iterator$class.foreach(Iterator.scala:893) > >> at > scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > >> at > >> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > >> at > scala.collection.AbstractIterable.foreach(Iterable.scala:54) > >> at > >> > scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:682) > >> at > >> akka.event.LoggingBus$class.startDefaultLoggers(Logging.scala:112) > >> at > akka.event.EventStream.startDefaultLoggers(EventStream.scala:28) > >> at > >> akka.actor.LocalActorRefProvider.init(ActorRefProvider.scala:667) > >> at > akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:648) > >> at > >> akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:645) > >> at akka.actor.ActorSystemImpl._start(ActorSystem.scala:645) > >> at akka.actor.ActorSystemImpl.start(ActorSystem.scala:661) > >> at akka.actor.ActorSystem$.apply(ActorSystem.scala:142) > >> at akka.actor.ActorSystem$.apply(ActorSystem.scala:109) > >> at > >> > com.vnomicscorp.timeseries.tsfs.TsfsApplicationContext.<init>(TsfsApplicationContext.scala:42) > > > >> > >> And this is all kicked off just by calling this line of code: > >> > >> implicit val actorSystem = ActorSystem("tsfs") > >> > >> This far, I tried various versions of the base OS and tried updating > >> Akka with zero effect. Any suggestions would be greatly appreciated! > >> Thank you, > >> James > >> > >> -- > >> >>>>>>>>>> Read the docs: http://akka.io/docs/ > >> >>>>>>>>>> Check the FAQ: > >> http://doc.akka.io/docs/akka/current/additional/faq.html > >> >>>>>>>>>> Search the archives: > https://groups.google.com/group/akka-user > >> --- > >> You received this message because you are subscribed to the Google > >> Groups "Akka User List" group. > >> To unsubscribe from this group and stop receiving emails from it, send > >> an email to [email protected] <javascript:> > >> <mailto:[email protected] <javascript:>>. > >> To post to this group, send email to [email protected] > <javascript:> > >> <mailto:[email protected] <javascript:>>. > >> Visit this group at https://groups.google.com/group/akka-user. > >> For more options, visit https://groups.google.com/d/optout. > > > > > > > -- > Ionuț G. Stan | http://igstan.ro | http://bucharestfp.ro > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
