Re: Deleting topics in windows - fix estimate or workaround

2019-12-01 Thread Sachin Mittal
Hi Edward, I hear you. So basically we are saying that unless major sponsors of Kafka project get actual requirement for support on windows this issue will linger on. Anyway my couple of takes from this issue is: 1. One should not overlook the potential opportunity for windows based systems. Ther

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 2.2.2

2019-12-01 Thread Vahid Hashemian
Awesome. Thanks for managing this release Randall! Regards, --Vahid On Sun, Dec 1, 2019 at 5:45 PM Randall Hauch wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 2.2.2 > > This is a bugfix release for Apache Kafka 2.2. > All of the changes in this releas

Reducing streams startup bandwidth usage

2019-12-01 Thread Alessandro Tagliapietra
Hello everyone, we're having a problem with bandwidth usage on streams application startup, our current setup does this: ... .groupByKey() .windowedBy(TimeWindows.of(Duration.ofMinutes(1))) .aggregate( { MetricSequenceList(ArrayList()) }, { key, value, aggregate -> agg

[ANNOUNCE] Apache Kafka 2.2.2

2019-12-01 Thread Randall Hauch
The Apache Kafka community is pleased to announce the release for Apache Kafka 2.2.2 This is a bugfix release for Apache Kafka 2.2. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/2.2.2/RELEASE_NOTES.html You can download the source and bina

Re: More partitions => less throughput?

2019-12-01 Thread Craig Pastro
Dear Tom, Peter and Eric, Thank you very much for your answers! What I think is that I need to play around with more configurations. Actually I had not thought that 64 partitions on a single box was very large. Thank you! I had thought that when I set log.flush.interval.message = 1, that I could

Re: Deleting topics in windows - fix estimate or workaround

2019-12-01 Thread Edward Capriolo
My development environment is windows. I am actually able to run integration tests launching Kafka and Zookeeper. However I have noticed that the test warn likely about deleting a file that is locked and I tried deleting a topic at one point and saw some of the issues you mentioned. I personally t

Re: [EXTERNAL] Re: UNKNOWN_PRODUCER_ID

2019-12-01 Thread Matthias J. Sax
Correct. As long as there are only LOG statements about it, you don't need to worry. If the producer cannot recover from this error internally, it would throw an `UnknowdProducerIdException`; for this case, the write was _not_ successful. Your application could handle this case by creating a new `

Re: Deleting topics in windows - fix estimate or workaround

2019-12-01 Thread Sachin Mittal
Hi, Firstly development environment is windows and I have been able to run almost any big data framework on windows (or cygwin) be it HBase, Cassandra or anything else. Now just for running Kafka switching to linux is bit difficult. Further Kafka site does mention that windows is supported and I f

Re: [EXTERNAL] Re: UNKNOWN_PRODUCER_ID

2019-12-01 Thread Jose Manuel Vega Monroy
Hi Matthias, So looking will be fixed in Kafka 2.5.0, as per https://issues.apache.org/jira/browse/KAFKA-8710, so we will take into account to upgrade properly to that version when ready and stable. We see this kind of errors in broker logs: [2019-12-01 15:27:39,790] ERROR [ReplicaManager brok

Re: Deleting topics in windows - fix estimate or workaround

2019-12-01 Thread M. Manna
Hi both, It’s been going around for a long time, but Kafka is officially not fully tested and verified for Windows. The disclaimer is on the official site. Windows servers are easy choice because a lot of infrastructures are on Windows and a lot of businesses are dependent on those infrastructure.

Re: Deleting topics in windows - fix estimate or workaround

2019-12-01 Thread Israel Ekpo
Sachin Are you running production Kafka on Windows? Is it possible to migrate to an alternate UNIX based environment? It would be great to learn why you chose to run it on Windows On Sun, Dec 1, 2019 at 2:22 AM Sachin Mittal wrote: > Hi All, > I hope we are well aware of the critical bug in