Re: [akka-user] Question about Reactive Kafka performances

I am actually using scala 2.12. My build is a bit more complex, but here is an extract of it: scalaVersion := "2.12.2", val akkaV = "2.4.17" val akkaHttpV = "10.0.4" val scalaTestV = "3.0.3" val scalaCheckV = "1.13.5" val kafkaV = "0.10.0.1" val reactiveKafkaV = "0.13" val logbackV = "1.2.3"

Re: [akka-user] Question about Reactive Kafka performances

Sometimes I get : [WARN] [06/23/2017 21:56:44.500] [toto-akka.kafka.default-dispatcher-14] [ akka://toto/system/kafka-consumer-1] Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds Java HotSpot(TM)

Re: [akka-user] Question about Reactive Kafka performances

Pretty weird, I have the same build.sbt than you : scalaVersion := "2.11.7" val kafkaV = "0.10.0.1" val reactiveKafkaV = "0.13" libraryDependencies ++= Seq( "org.apache.kafka" % "kafka-clients" % kafkaV, "org.apache.kafka" % "kafka_2.11" % kafkaV intransitive, "com.typesafe.akka" %%

Re: [akka-user] Question about Reactive Kafka performances

Thank you Alex, I will give a try :) Le vendredi 23 juin 2017 23:30:54 UTC+2, Kilic Ali-Firat a écrit : > > Did you do something like this ? > > libraryDependencies ++= Seq( > "com.typesafe.akka" %% "akka-stream-kafka" % "0.16" > ) > > dependencyOverrides += ( > "org.apache.kafka" %

Re: [akka-user] Question about Reactive Kafka performances

Did you do something like this ? libraryDependencies ++= Seq( "com.typesafe.akka" %% "akka-stream-kafka" % "0.16" ) dependencyOverrides += ( "org.apache.kafka" % "kafka-clients" % "0.10.0.1" ) Using dependencyOverrides on kafka-clients should not be enough to get the same speedup .. Le

Re: [akka-user] Question about Reactive Kafka performances

essentially you put this in your build.sbt: val kafkaV = "0.10.0.1" val reactiveKafkaV = "0.13" libraryDependencies ++= Seq( "org.apache.kafka" % "kafka-clients" % kafkaV, "org.apache.kafka" % "kafka_2.11" % kafkaV intransitive, "com.typesafe.akka" %% "akka-stream-kafka" % reactiveKafkaV,

Re: [akka-user] Question about Reactive Kafka performances

yes, in our dependencies we do not only include reactive kafka, but also explicitly the dependencies on kafka libs so that we can force 0.10.0.1. On Friday, June 23, 2017 at 1:50:33 PM UTC-7, Kilic Ali-Firat wrote: > > Oh so I'm not the only one to observe a such throughput. > > Maybe a stupid

Re: [akka-user] Question about Reactive Kafka performances

Oh so I'm not the only one to observe a such throughput. Maybe a stupid question but how did you force the compilation against the 0.10.0.1 release of kafka libs ? Le vendredi 23 juin 2017 22:24:21 UTC+2, Alex Cozzi a écrit : > > We observed a similar slowdown when i use reactive kafka (which

Re: [akka-user] Question about Reactive Kafka performances

We observed a similar slowdown when i use reactive kafka (which uses kafka 0.10.2.0 as dependency) connecting to a 0.10.0.0 server, but we solved by forcing compiling against the 0.10.0.1 release of the kafka libraries (kafka-clients and kafka_2.11) we get the same speedup. I think it has to

Re: [akka-user] Question about Reactive Kafka performances

*kafka broker is kafka_2.11-0.10.0.0 -- >> Read the docs: http://akka.io/docs/ >> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> Search the archives: https://groups.google.com/group/akka-user --- You received this

Re: [akka-user] Question about Reactive Kafka performances

It's me again ! I reached a better throughput (not as the producer). The version of my kafka version is kafka_211-0.10.0.0 and using reactive-kafka (0.12), I can reach a throughput between 3000 and 4000 messages / sec. It seems that the reactive kafka version should be aligned with (or

Re: [akka-user] Question about Reactive Kafka performances

Hi Patrick, Running in a local kafka, I can have the same throughput than the kafka producer. But running in a hosted kafka, I cannot get the same throughput even if I use the same parameter than in local. The main difference between the hosted and local is that I'm using a secured

Re: [akka-user] Question about Reactive Kafka performances

Hi Patrick, I attached a tarball with only the reactive kafka lib and a Main class with how I'm using the reactive API. There is a README explaining how to reproduce the throughput issue that I have in my code. I can reproduce the issue on a local kafka too. Thanking you in advance ! Le

Re: [akka-user] Question about Reactive Kafka performances

I'd recommend that you create a minimized example without addition external dependencies. Then I or someone can run it and perhaps see what is the culprit. /Patrik fre 23 juni 2017 kl. 00:15 skrev Kilic Ali-Firat : > Like you said, 2000 msg / sec is really relly

Re: [akka-user] Question about Reactive Kafka performances

Like you said, 2000 msg / sec is really relly slow. The reactive Kafka consumer should consume at the same speed than the producer. Reading the benchmark examples in GitHub plus the benchmarks result in your link didn't help me to catch what I am doigt wrong. I respect the way to use the