Re: sliding ktable?

There is a problem with tombstoning old entries based on a new entry, that, the keys which have no new entries will remain there forever. On Mon, Nov 7, 2016 at 9:38 AM, Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > John, > > your thinking

RE: is there a way to make sure two consumers receive the same message from the broker?

Generally Cassandra itself is not consistent enough, even with quorum read-writes, say one of the writes fail, the nodes who received the data won't roll back and it might lead to dirty reads which in turn makes roll back logic tricky (if not impossible). You can use linearizable writes but if

RE: SASL error when tring to connect kafka to kerberos server

Hi, Thanks for your cue. But where to configure kafka subject and how to add Kerberos subject permission? Searched docs and thought for a few days but couldn't figure out. Thanks, Ye -Original Message- From: Martin Gainty [mailto:mgai...@hotmail.com] Sent: Friday, November 04, 2016

Re: Deadlock using latest 0.10.1 Kafka release

Thanks Becket. I was working on that today. I have a working jar, created from the 0.10.1.0 branch, and that specific KAFKA-3994 patch applied to it. I've left it running in one test broker today, will try tomorrow to trigger the issue, and try it with both the patched and un-patched versions.

Re: Kafka Connect Hdfs Sink not sinking

And you're using the same version of the broker and clients, and there are no other error messages in the log? I see you're using AvroConverter, do you also have schema registry listening on the appropriate port? Normally I'd expect to at least see some sort of exception or error message within

Re: Reopen KAFKA-4344 ?

Hi Srinivas I raised the issue and the way I got around this was to let kafka streams run on POJO way rather than some of the dependent instances being spring managed bean instances. If you create the instance of riakService and counterService in processor class instead of passing the spring

Re: consumer client pause/resume/rebalance

I think the current behavior is fairly reasonable. Following a rebalance the entire state of the consumer changes - you may get an entirely new set of partitions. A common use-case for pause is to allow a consumer to keep polling and avoid getting new events while it is retrying to process

Re: HDFS Connector Compression?

Documentation patches are welcome :) On Mon, Nov 7, 2016 at 9:06 AM, Henry Kim wrote: > Nevermind, it seems snappy is supported by default for ParquetFormat. It was > not obvious with the fixed extension. > > > private final static String EXTENSION = ".parquet"; > >

Re: Added to Wiki please

Gwen, Makes total sense! Sorry for the wide distribution then, my apologies to the list. Kenny Gorman Founder www.eventador.io > On Nov 5, 2016, at 9:30 PM, Gwen Shapira wrote: > > Hi Kenny, > > First, thank you for letting the community know about your valuable service.

Re: Deadlock using latest 0.10.1 Kafka release

Hi Marcos, Is it possible for you to apply the patch of KAFKA-3994 and see if the issue is still there. The current patch of KAFKA-3994 should work, the only reason we haven't checked that in was because when we ran stress test it shows noticeable performance impact when producers are producing

Re: Reopen KAFKA-4344 ?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 KAFKA-4344 was not a bug. The issues was as wrong initialization order of Kafka Streams by the user. Please double check your initialization order (and maybe read the old email thread and JIRA comments -- it might have some relevant information for

consumer client pause/resume/rebalance

Using the v0.9.0.1 consumer API, I recently learned that paused partitions can unexpectedly become become unpaused during a rebalance. I also found an old thread from the mailing list which corroborates this behavior:

Re: is there a way to make sure two consumers receive the same message from the broker?

And there is this https://github.com/vroyer/elassandra which is still under active development and not sure how they plan to keep up with Apache Cassandra moving forward. On Mon, Nov 7, 2016 at 9:36 AM, kant kodali wrote: > Fixing typo's > > Hi Tauzell, > > Yeah our users

zookeeper restart fatal error

I observed this as well when zookeeper runs out of file handles. Increase file handles to 5 times the number of expected topics. This communication and all information contained in or attached to it (including, but not limited to market prices/levels and market commentary) (the "Information")

Re: Deadlock using latest 0.10.1 Kafka release

We ran into this issue several more times over the weekend. Basically, FDs are exhausted so fast now, we can't even get to the server in time, the JVM goes down in less than 5 minutes. I can send the whole thread dumps if needed, but for brevity's sake, I just copied over the relevant deadlock

Re: sliding ktable?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 John, your thinking is on the right track! About infinitely growing KTable: It seems you are extending each lane with a list of all txnId -- so your view needs infinite memory as you expend your values... A quick fix might be, to delete older

Re: is there a way to make sure two consumers receive the same message from the broker?

Fixing typo's Hi Tauzell, Yeah our users want to query, do aggregations on Elastic Search directly and we cannot have inconsistent data because say the writes didn't make it into Cassandra but made it to Elastic search then a simple aggregations like count will lead to a wrong answer but again

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi Tauzell, Yeah our users want to query, do aggregations on Elastic Search directly and we cannot have inconsistent data because say the writes didn't make it into Cassandra but made it to Elastic search then a simple aggregations like count will lead to a wrong answer but again as @Hans

Re: HDFS Connector Compression?

Nevermind, it seems snappy is supported by default for ParquetFormat. It was not obvious with the fixed extension. private final static String EXTENSION = ".parquet"; final Schema avroSchema = avroData.fromConnectSchema(record.valueSchema()); CompressionCodecName compressionCodecName =

RE: is there a way to make sure two consumers receive the same message from the broker?

Here is a scenario where this could be useful: Add the kafka offset as a field on the record in both Cassandra and Elasticsearch Now when you get search results from Elastic search and look up details in Cassandra you can know if they come from the same kafka record. If you can use the

Re: Checking the consumer lag when using manual partition assignment with the KafkaConsumer

Mattias, > kafka-consumer-offet-checker works for older Kakfa versions for which > offsets are stored in ZK. It does not work for v0.9+ when offsets are > stored in topic __consumer_offsets That is not true. The offset checker works with both ZK based and broker based offset commit. It query

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi AmitHossein, I still don't see how that guarantees consistency at any given time. other words how do I know at time X the data in Cassandra and ES are the same. Thanks On Mon, Nov 7, 2016 at 3:26 AM, AmirHossein Roozbahany wrote: > Hi > > Can you use elasticsearch

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi Hans, We currently do #2 and thats is quite slow so yeah In thoery #1 is probably a better choice although its not quite what we want since it doesn't guarantee consistency at any given time as you have already pointed out. Thanks a lot for the response! kant On Mon, Nov 7, 2016 at 6:31 AM,

RE: sliding ktable?

Thanks for the reply. I really appreciate the insight. Again newbie here. I want to expand on what I am struggling with. It may be that I just need to get my mind thinking more in a streaming mode. Please let me know you thoughts. Just having problem ‘getting it’ on my own. Below is a

replica fetch error and shuabing

Hi, when I move __consumer_offsets from old broker to new broker, we encounter error as follow and it always shuabing. server.log.2016-11-07-19:[2016-11-07 19:17:15,392] ERROR Found invalid messages during fetch for partition [__consumer_offsets,10] offset 13973569 error Message found with

Reopen KAFKA-4344 ?

Hi, I'm still seeing the same issue with spring boot. Code is below, sorry code is in groovy and not fully baked. Just have single processor. It worked well with single partition. But when i increased the partitions, started seeing the error as in this kafka-4344. import

Re: is there a way to make sure two consumers receive the same message from the broker?

Hi Can you use elasticsearch _version field as cassandra's writetime?(_version is strictly increasing, cassandra uses writetime for applying LWW, so last write in elasticsearch will always win) It needs no transaction and makes databases convergent. From:

Re: is there a way to make sure two consumers receive the same message from the broker?

I don't believe that either of your two storage systems support distributed atomic transactions. You are just going to have to do one of the following: 1) update them separately (in parallel) and be aware that their committed offsets may be slightly different at certain points in time 2) update

Re: Re-consume messages

Thanks for your reply. Will give it a shot. On Sun, Nov 6, 2016 at 9:38 PM, Becket Qin wrote: > Hi Amit, > > In Kafka 0.9, the closest approach would be use > SimpleConsumer.getOffsetBefore() to search the offset by timestamp. And > then you can consume from the returned

Broker outage sent controller into infinite loop

I'm managing a 21 node Kafka cluster, running 0.8.2.1. Recently one broker crashed during software RAID rebuild after one of the OS disks failed (Kafka logs are stored on separate RAID1+0 drive). At first the controller behaved as expected, marking the node as down and sending out messages to

Force delete tombstone markers in the log compacted topics in kafka 0.9.0.1

Hi everybody, I'm newbie in this mail list , hi everybody! I'm trying to improve the documentation for my operator guys and it could be useful to know some method to force the deletion of tombstone markers in the log compacted topics. I only know the properties