Perf on history reprocessing

[Mathieu D]

Hello there

Sometimes we need to reprocess a large amount of history data.
I find the performance in that case quite disappointing. More precisely
throughput is quite low (which is not surprising for a system optimized for
low latency).

Is there any knob to turn to get a much higher throughput in such cases ?

Thanks
Mathieu

Re: kafka-streams: do not output anything while state is not stable

[Mathieu D]

Ok. Thanks ;-)

Le mar. 20 oct. 2020 à 19:12, Matthias J. Sax  a écrit :

> It's highly use-case dependent, but applying a filter at the end does
> sound like a good solution to me.
>
> -Matthias
>
> On 10/19/20 12:40 PM, Mathieu D wrote:
> > Hello there,
> >
> > Let's say I need to restart my streams app from a blank state (whether by
> > changing app.id or using application-reset-tool).
> > My app is designed on "at least once" paradigm, and outputs are upserts.
> > The input topics have a few days worth of data, and the app will restart
> > from there.
> >
> > If I restart it, states will be empty, meaning most outputs will be
> wrong.
> > So it should NOT upsert anything downstream for a certain period of time
> > (something like my biggest time window or something)
> >
> > I'm thinking of a filter at the end of the topology, removing any record
> > whose timestamp < topic retention + biggest window size.
> >
> > How do you guys manage this usually ?
> >
> > Thanks for your insights
> >
> > Mathieu
> >
>

Re: Client session timed out

[Praveen Kumar K S]

Hello,

Can someone please help me to understand what is the issue ?

Regards,
Praveen Kumar K S
+91-9986855625


On Thu, Oct 22, 2020 at 6:52 AM Praveen Kumar K S 
wrote:

> Hello Experts,
>
> Any help to debug and resolve this issue is highly appreciated.
>
> Regards,
> Praveen
>
> On Wed, 21 Oct, 2020, 11:26 Sabina Marx,  wrote:
>
>> Hi Praveen,
>>
>> it seems to be the same problem, your log looks quite similar to mine.
>> But I have no solution until now.
>>
>> Regards
>> Sabina
>>
>> Von: Praveen Kumar K S 
>> Antworten an: "users@kafka.apache.org" 
>> Datum: Dienstag, 20. Oktober 2020 um 20:07
>> An: "users@kafka.apache.org" 
>> Betreff: Re: Client session timed out
>>
>> Hello,
>>
>> I'm not sure if I can add my issue in this thread. But it seems like I'm
>> facing the same problem.
>>
>> KAFKA_VERSION=2.5.1
>> ZK_VERSION=3.5.8
>>
>> I run 3 node zookeeper cluster and 3 node kafka cluster as docker
>> containers in docker swarm environment. When I install it for first time,
>> everything goes well. Zookeeper and Kafka are able to form the cluster.
>> Services are healthy.
>>
>> But when I issue docker update command, kafka is not coming up though the
>> zookeeper cluster is healthy. Below is the sequence of steps.
>>
>> docker service update one_zookeeper --image x.x.x/v1/zookeeper:latest
>> --force
>> docker service update one_zookeeper1 --image x.x.x/v1/zookeeper:latest
>> --force
>> docker service update one_zookeeper2 --image x.x.x/v1/zookeeper:latest
>> --force
>>
>> Zookeeper is healthy now. I'm able to query leader and follower.
>>
>> Now, I'm updating kafka and it doesn't work.
>> docker service update one_kafka --image x.x.com/v1/kafka:latest<
>> http://x.x.com/v1/kafka:latest> --force
>>
>> PFA Kafka log.
>>
>> While kafka update has failed, I see that kafka1 and kafka2 are running
>> and healthy.
>>
>> docker service ls | grep kafka
>> one_kafkareplicated  0/1
>> one_kafka1   replicated  1/1
>> one_kafka2   replicated  1/1
>>
>> To cross verify, I have just brought down the services zookeeper and
>> kafka without data loss. I preserve
>> zookeeperdata,zookeeperlogs,zookeepertxns and kafkadata,kafkalogs.
>>
>> docker stack remove one
>> docker stack deploy -c cluster-zookeeper.yml one
>> docker stack deploy -c cluster-kafka.yml one
>>
>> Now, all the services are healthy.
>>
>> I'm not sure why kafka deployment is failing only during update. There is
>> no change in the configuration in either zookeeper or kafka.
>>
>> Please help me resolve this issue and let me know if you need any
>> additional details.
>>
>> Regards,
>> Praveen Kumar K S
>> +91-9986855625
>>
>>
>> On Tue, Oct 20, 2020 at 3:54 PM Sabina Marx > sabina.m...@sneo.io>> wrote:
>> Yes, it's the same problem.
>>
>> Am 19.10.20, 19:50 schrieb "Mich Talebzadeh" > >:
>>
>> can you try to disable automatic start and on the node just booted,
>> start
>> zookeeper first, check the log that it is connected and then start
>> Kafka?
>>
>> I assume everything is set-up OK including in
>> $KAFKA_HOME/config/server.properties values for broker.id<
>> http://broker.id>, hostname,
>> zookeeper.connect=:2181,,server2>:2181, :2181  and
>> also
>> zookeeper.connection.timeout.ms<
>> http://zookeeper.connection.timeout.ms>=6000 (default)
>>
>> HTH
>>
>>
>>
>>
>>
>> LinkedIn *
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> >*
>>
>>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any
>> loss, damage or destruction of data or any other property which may
>> arise
>> from relying on this email's technical content is explicitly
>> disclaimed.
>> The author will in no case be liable for any monetary damages arising
>> from
>> such loss, damage or destruction.
>>
>>
>>
>>
>> On Mon, 19 Oct 2020 at 18:17, Sabina Marx > > wrote:
>>
>> > Yes, you have it
>> >
>> > Holen Sie sich Outlook für iOS
>> > 
>> > Von: Mich Talebzadeh > mich.talebza...@gmail.com>>
>> > Gesendet: Monday, October 19, 2020 7:09:53 PM
>> > An: users@kafka.apache.org <
>> users@kafka.apache.org>
>> > Betreff: Re: Client session timed out
>> >
>> > Ok I think it is clearer now.
>> >
>> > As I understand all your Zookeepers and Kafkas are running. (5 nodes
>> > meaning 5 physical hosts?). Then you have to reboot one physical
>> host. You
>> > still have the redundancy. But when the physical host comes up and
>> your
>> > zookeeper and then Kafka come up, you have Kafka timing out and not
>> > connecting to the existing Kafka cluster?
>> >
>> > Does that make sense?

How to get Date format value before 1970 in Kafka connect

[vishnu murali]

Hi guys

I am using JDBC SOURCE CONNECTOR to take data from AWS Redshift to Kafka

There I am having a field with Datatype as Date.

So while performing the value which is greater than 1970 works fine.

But if value is before 1970 it provide 00:00:00:Z

May I know how to solve this problem?

Does anyone have experienced this scenario?