Ok, I've made some progress but I'm seeing crashes now.
I can easily populate a set number of messages into a topic. No issues,
and offsets are properly set.
My consume code with 0.11-M1:
count = 0
val consumerSettings = ConsumerSettings(as, new ByteArrayDeserializer,
new StringDeserializer, Set("lowercaseStrings"))
.withBootstrapServers("192.168.99.100:9092")
.withGroupId("group1")
.withProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
val graph = RunnableGraph.fromGraph(GraphDSL.create() { implicit
builder: GraphDSL.Builder[akka.NotUsed] =>
import GraphDSL.Implicits._
type In = CommittableMessage[Array[Byte],String]
val src =
Consumer.committableSource(consumerSettings.withClientId("client1"))
val commit = Flow[In].map{ msg => msg.committableOffset.commit() }
val show = Flow[In].map{ i =>
count += 1
if(i.value=="done") println(s"Time ($count):
"+(System.currentTimeMillis() - now))
i
}
src ~> show ~> commit ~> Sink.ignore
ClosedShape
})
now = System.currentTimeMillis()
graph.run()
Ok, timings aside for a moment--I still have concerns there, but let's
focus on the crashing.
I restart Kafka each run so there's no left-over messages/offsets from
previous run.
I'm using a single topic with just one partition for this test.
If I run this for a small number of messages pre-populated, maybe up to
10000, it works.
If I run it with 100,000 I get this error:
[ReactiveKafka-akka.kafka.default-dispatcher-11] ERROR
akka.kafka.internal.CommittableConsumerStage$$anon$1 - Error in poll
java.lang.IllegalArgumentException: requirement failed
at scala.Predef$.require(Predef.scala:207)
at
akka.kafka.internal.ConsumerStageLogic.handleResult$1(ConsumerStage.scala:372)
at akka.kafka.internal.ConsumerStageLogic.poll(ConsumerStage.scala:384)
at
akka.kafka.internal.CommittableConsumerStage$$anon$1.poll(ConsumerStage.scala:145)
at
akka.kafka.internal.CommittableConsumerStage$$anon$1.commitAsync(ConsumerStage.scala:257)
at
akka.kafka.internal.CommittableConsumerStage$$anon$1.akka$kafka$internal$CommittableConsumerStage$$anon$$commitSingleInternal(ConsumerStage.scala:223)
at
akka.kafka.internal.CommittableConsumerStage$$anon$1$$anonfun$6.apply(ConsumerStage.scala:131)
at
akka.kafka.internal.CommittableConsumerStage$$anon$1$$anonfun$6.apply(ConsumerStage.scala:130)
at
akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:569)
at
akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:417)
at
akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:593)
at
akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:604)
at akka.actor.Actor$class.aroundReceive(Actor.scala:482)
at
akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:519)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
