The exception might be a bug. Please open an issue in https://github.com/akka/reactive-kafka
You should use mapAsync for the commits. Otherwise you loose the backpressure. commit returns a Future. You don't need to use a graph for that simple flow, see the examples. /Patrik sön 3 apr. 2016 kl. 01:52 skrev tigerfoot <[email protected]>: > Ok, I've made some progress but I'm seeing crashes now. > > I can easily populate a set number of messages into a topic. No issues, > and offsets are properly set. > > My consume code with 0.11-M1: > > count = 0 > val consumerSettings = ConsumerSettings(as, new ByteArrayDeserializer, > new StringDeserializer, Set("lowercaseStrings")) > .withBootstrapServers("192.168.99.100:9092") > .withGroupId("group1") > .withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest") > > val graph = RunnableGraph.fromGraph(GraphDSL.create() { implicit > builder: GraphDSL.Builder[akka.NotUsed] => > import GraphDSL.Implicits._ > type In = CommittableMessage[Array[Byte],String] > val src = > Consumer.committableSource(consumerSettings.withClientId("client1")) > val commit = Flow[In].map{ msg => msg.committableOffset.commit() } > val show = Flow[In].map{ i => > count += 1 > if(i.value=="done") println(s"Time ($count): > "+(System.currentTimeMillis() - now)) > i > } > > src ~> show ~> commit ~> Sink.ignore > ClosedShape > }) > now = System.currentTimeMillis() > graph.run() > > Ok, timings aside for a moment--I still have concerns there, but let's > focus on the crashing. > I restart Kafka each run so there's no left-over messages/offsets from > previous run. > I'm using a single topic with just one partition for this test. > If I run this for a small number of messages pre-populated, maybe up to > 10000, it works. > If I run it with 100,000 I get this error: > > [ReactiveKafka-akka.kafka.default-dispatcher-11] ERROR > akka.kafka.internal.CommittableConsumerStage$$anon$1 - Error in poll > java.lang.IllegalArgumentException: requirement failed > at scala.Predef$.require(Predef.scala:207) > at > akka.kafka.internal.ConsumerStageLogic.handleResult$1(ConsumerStage.scala:372) > at akka.kafka.internal.ConsumerStageLogic.poll(ConsumerStage.scala:384) > at > akka.kafka.internal.CommittableConsumerStage$$anon$1.poll(ConsumerStage.scala:145) > at > akka.kafka.internal.CommittableConsumerStage$$anon$1.commitAsync(ConsumerStage.scala:257) > at > akka.kafka.internal.CommittableConsumerStage$$anon$1.akka$kafka$internal$CommittableConsumerStage$$anon$$commitSingleInternal(ConsumerStage.scala:223) > at > akka.kafka.internal.CommittableConsumerStage$$anon$1$$anonfun$6.apply(ConsumerStage.scala:131) > at > akka.kafka.internal.CommittableConsumerStage$$anon$1$$anonfun$6.apply(ConsumerStage.scala:130) > at > akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:569) > at > akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:417) > at > akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:593) > at > akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:604) > at akka.actor.Actor$class.aroundReceive(Actor.scala:482) > at > akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:519) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) > at akka.actor.ActorCell.invoke(ActorCell.scala:495) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) > at akka.dispatch.Mailbox.run(Mailbox.scala:224) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
