Re: Kafka Streams And Partitioning

Hi Sophie, Thanks very much for the response! So if I understand correctly it will be impossible to avoid the repartition topic? e.g. my original message may have key = A...and will be partitioned on A. But in my Kafka Streams app, I will want to aggregate on A:B or A:C or A:D (B, C or D come

Re: Kafka Streams And Partitioning

Hey Gareth, Kafka Streams state store partitioning is based on the partitioning of the upstream input topics. If you want your RocksDB stores to be partitioned based on the prefix of a key, then you should make sure the input topic feeding into it uses whatever partitioning strategy you had in

Re: MirrorMaker 2 and Negative Lag

OK. I follow now. Let me try to re-test to see if it makes a difference. Thanks. ... Alan On Wed, Mar 17, 2021 at 5:46 PM Samuel Cantero wrote: > I've found that bug the hard way. FWIW I've migrated several clusters from > kafka 0.10 to kafka 2.x using mm2. So offsets sync work fine for kafka

Re: MirrorMaker 2 and Negative Lag

I've found that bug the hard way. FWIW I've migrated several clusters from kafka 0.10 to kafka 2.x using mm2. So offsets sync work fine for kafka 0.10. Best, On Wed, Mar 17, 2021 at 6:43 PM Samuel Cantero wrote: > No, what I meant is that offsets sync won't work if >

Re: MirrorMaker 2 and Negative Lag

No, what I meant is that offsets sync won't work if `consumer.auto.offset.reset:latest` (it was not talking about that particular bug). Try setting `consumer.auto.offset.reset:earliest` and do verify if offsets are sync'd correctly. Best, On Wed, Mar 17, 2021 at 6:42 PM Alan Ning wrote: > Hey

Re: MirrorMaker 2 and Negative Lag

Hey Ning, My source cluster is a very old cluster running 0.10. I don't think I can change the version. In fact, my whole effort is to migrate out of the legacy cluster to 2.6+. Any insight would be greatly appreciated. Thank you. ... Alan On Wed, Mar 17, 2021 at 4:27 PM Ning Zhang wrote: >

Re: MirrorMaker 2 and Negative Lag

Hey Samuel, I am aware of that `consumer.auto.offset.reset:latest` problem. It was because this PR never made it to trunk. I patched MM2 locally for 2.7 so that `latest` offset will work. ... Alan On Wed, Mar 17, 2021 at 4:50 PM

Re: Right Use Case For Kafka Streams?

Hello Gareth, A common practice for rolling up aggregations with Kafka Streams is to do the finest granularity at processor (5 days in your case), and to coarse-grained rolling up upon query serving through the interactive query API -- i.e. whenever a query is issued for a 30 day aggregate you do

Re: SAP PI/PO -> Kafka integration options

Kafka doesn’t have a rest proxy. Confluent does. Also, Instaclustr offers a Kafka REST proxy. Also, SAP has 100s of products including SAP Cloud Platform. So not sure what this PI/PO means for your case. Unless there’s something I’m unaware of, you referring to non-Apache offering here. You

SAP PI/PO -> Kafka integration options

Hi, My name is Arian. I work at Scania as an architect. We have SAP in our landscape and are using PI/PO as our integration platform. We are interested to know what type of integration options there is to integrate to Kafka from a SAP PI/PO perspective. We know that Kafka has an API proxy

Re: MirrorMaker 2 and Negative Lag

I've seen this before. I've found that consumer offsets sync does not work with `consumer.auto.offset.reset:latest`. If you set this to earliest, then it should work. One way to workaround the need to start from earliest is by starting with latest and once mirroring is ongoing swap to earliest.

Re: Mirrormaker 2.0 - duplicates with idempotence enabled

Hello Vangelis, By default, current MM 2.0 guarantees "at-least" once delivery guarantee, meaning there will be duplicate messages under some failure scenarios. If you prefer to no-message loss, there is a pending PR about MM 2.0 https://issues.apache.org/jira/browse/KAFKA-10339 On 2021/03/10

Re: MirrorMaker 2 and Negative Lag

Hello Alan, I may probably see the similar case. One quick validation that could be run is to test on the source cluster with higher Kafka version. If still not working, please email me and I could introduce you to person who may have similar case before. On 2021/03/15 21:59:03, Alan Ning

Re: Issue writing to Google BigQuery

On the sideline I also validated the Json format {"schema": { "type": "struct", "fields": [ { "field": "rowkey", "type": "string", "optional": true}],"optional": false,"name": "BQ"}, "payload": {"rowkey": "c0224abd-a6c4-4743-ac01-55e7b6980062"}} [image: image.png] Thanks LinkedIn *

SAP PI/PO -> Kafka integration options

Hi, My name is Arian. I work at Scania as an architect. We have SAP in our landscape and are using PI/PO as our integration platform. We are interested to know what type of integration options there is to integrate to Kafka from a SAP PI/PO perspective. We know that Kafka has an API proxy

Issue writing to Google BigQuery

This is what is termed as fun and game. Trying to write a single column (for the sake of test) in this case to BigQuery from Kafka. I am sending the schema and payload as per docs This is message sent that I can get it from console $KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server

Re: Emit events that are NOT joined

Maybe it’s not what you are looking for, but I think that functionality can be implemented in two steps. First, you perform a LEFT join, and then you filter by “null” on the joined field to identify those whose joins didn’t succeed. HTH, Rubén El El mar, 16 mar 2021 a las 5:34, Ross Black