Hi, >From your stacktrace it seems like you are trying to consume an entity's body (the stream of bytes) twice.
> The stack trace does not point to any of my classes/namespace, rather show akka-stream classes.. This does not mean anything, since you might made your error when you wired up your graph using only Akka Streams components. Can you give us a small reproducer that demonstrates this problem? -Endre On Fri, Aug 28, 2015 at 11:43 PM, IR <[email protected]> wrote: > akka version: 2.3.12 > akka-http version: 1.0 > > Hi all, > I am running a service which is receiving small to medium payloads and > uploading them to S3 type cloud storage. The load I am applying is about > 200 requests/sec. > Occasionally, I see the below stack trace and the service is pausing, > temporarily not able to process any further requests. It will eventually > resume. > The stack trace does not point to any of my classes/namespace, rather show > akka-stream classes.. > Has anyone seen this exception and/or can help pin point the source of the > error? > Thanks. > > === > > Aug 28 04:57:33: 2015-08-28T04:57:33.773+0000 level=ERROR > logger=a.a.ActorSystemImpl Outgoing request stream error > sourceThread=casper-akka.actor.default-dispatcher-206, > akkaTimestamp=04:57:33.767UTC, akkaSource=ActorSystem(casper) > Aug 28 04:57:33: java.lang.IllegalStateException: oneHundredContinueRef > unavailable > Aug 28 04:57:33: at > akka.http.impl.engine.parsing.HttpRequestParser$$anonfun$expect100continueHandling$1$1$$anonfun$apply$1$$anon$1$$anonfun$1.apply(HttpRequestParser.scala:148) > ~[com.typesafe.akka.akka-http-core-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.http.impl.engine.parsing.HttpRequestParser$$anonfun$expect100continueHandling$1$1$$anonfun$apply$1$$anon$1$$anonfun$1.apply(HttpRequestParser.scala:148) > ~[com.typesafe.akka.akka-http-core-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at scala.Option.getOrElse(Option.scala:121) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > akka.http.impl.engine.parsing.HttpRequestParser$$anonfun$expect100continueHandling$1$1$$anonfun$apply$1$$anon$1.onPull(HttpRequestParser.scala:148) > ~[com.typesafe.akka.akka-http-core-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.http.impl.engine.parsing.HttpRequestParser$$anonfun$expect100continueHandling$1$1$$anonfun$apply$1$$anon$1.onPull(HttpRequestParser.scala:143) > ~[com.typesafe.akka.akka-http-core-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.fusing.OneBoundedInterpreter$$anon$3.run(Interpreter.scala:473) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.fusing.OneBoundedInterpreter$State$class.progress(Interpreter.scala:245) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.fusing.OneBoundedInterpreter$$anon$3.progress(Interpreter.scala:467) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.fusing.OneBoundedInterpreter.akka$stream$impl$fusing$OneBoundedInterpreter$$execute(Interpreter.scala:580) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.fusing.OneBoundedInterpreter$State$class.execute(Interpreter.scala:241) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.fusing.OneBoundedInterpreter$EntryState.execute(Interpreter.scala:666) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.stage.AbstractStage.enterAndPull(Stage.scala:76) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.fusing.ActorOutputBoundary.akka$stream$impl$fusing$ActorOutputBoundary$$tryPutBallIn(ActorInterpreter.scala:217) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.fusing.ActorOutputBoundary$$anonfun$downstreamRunning$1.applyOrElse(ActorInterpreter.scala:305) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at akka.stream.impl.SubReceive.apply(Transfer.scala:16) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at akka.stream.impl.SubReceive.apply(Transfer.scala:12) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.SubReceive.applyOrElse(Transfer.scala:12) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at akka.actor.Actor$class.aroundReceive(Actor.scala:467) > ~[com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.fusing.ActorInterpreter.aroundReceive(ActorInterpreter.scala:366) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at akka.actor.ActorCell.invoke(ActorCell.scala:487) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at > akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at akka.dispatch.Mailbox.run(Mailbox.scala:220) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > [org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > [org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > [org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: 2015-08-28T04:57:33.789+0000 level=ERROR > logger=a.a.ActorSystemImpl Outgoing request stream error > sourceThread=casper-akka.actor.default-dispatcher-222, > akkaTimestamp=04:57:33.772UTC, akkaSource=ActorSystem(casper) > Aug 28 04:57:33: java.lang.IllegalStateException: Substream publisher only > supports one subscriber (which is allowed, see reactive-streams > specification, rule 1.12) > Aug 28 04:57:33: at > akka.stream.impl.ReactiveStreamsCompliance$.rejectAdditionalSubscriber(ReactiveStreamsCompliance.scala:57) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MultiStreamOutputProcessor$SubstreamOutput.subscribe(StreamOfStreamProcessors.scala:104) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.assignPort(StreamLayout.scala:721) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.ActorMaterializerImpl$$anon$3.materializeAtomic(ActorMaterializerImpl.scala:99) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:688) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.collection.immutable.Set$Set4.foreach(Set.scala:181) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.materializeComposite(StreamLayout.scala:704) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:694) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.collection.immutable.Set$Set1.foreach(Set.scala:79) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:691) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.collection.immutable.Set$Set3.foreach(Set.scala:145) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.materializeComposite(StreamLayout.scala:704) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:694) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.collection.immutable.Set$Set1.foreach(Set.scala:79) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.materializeComposite(StreamLayout.scala:704) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:694) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.collection.immutable.Set$Set2.foreach(Set.scala:111) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.MaterializerSession.materialize(StreamLayout.scala:655) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.ActorMaterializerImpl.materialize(ActorMaterializerImpl.scala:201) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:365) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at akka.stream.scaladsl.Source.runWith(Source.scala:93) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.ConcatAllImpl$$anonfun$1.apply$mcV$sp(ConcatAllImpl.scala:28) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at akka.stream.impl.Pump$class.pump(Transfer.scala:199) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.ActorProcessorImpl.pump(ActorProcessor.scala:250) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.BatchingInputBuffer.enqueueInputElement(ActorProcessor.scala:91) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.BatchingInputBuffer$$anonfun$upstreamRunning$1.applyOrElse(ActorProcessor.scala:142) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at akka.stream.impl.SubReceive.apply(Transfer.scala:16) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at akka.stream.impl.SubReceive.apply(Transfer.scala:12) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.SubReceive.applyOrElse(Transfer.scala:12) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > ~[org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at akka.actor.Actor$class.aroundReceive(Actor.scala:467) > ~[com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at > akka.stream.impl.ActorProcessorImpl.aroundReceive(ActorProcessor.scala:250) > ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] > Aug 28 04:57:33: at > akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at akka.actor.ActorCell.invoke(ActorCell.scala:487) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at > akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at akka.dispatch.Mailbox.run(Mailbox.scala:220) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] > Aug 28 04:57:33: at > scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > [org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > [org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > [org.scala-lang.scala-library-2.11.6.jar:na] > Aug 28 04:57:33: at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > [org.scala-lang.scala-library-2.11.6.jar:na] > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Akka Team Typesafe - Reactive apps on the JVM Blog: letitcrash.com Twitter: @akkateam -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
