Re: Kafka Streams Failed to rebalance error

[João Peixoto]

I am now able to consistently reproduce this issue with a dummy project.

1. Set "max.poll.interval.ms" to a low value
2. Have the pipeline take longer than the interval above
3. Profit

This happens every single time and never recovers.
I simulated the delay by adding a breakpoint on my IDE on a sink "foreach"
step and then proceeding after the above interval had elapsed.

Any advice on how to work around this using 0.10.2.1 would be greatly
appreciated.
Hope it helps

On Wed, Jun 7, 2017 at 10:19 PM João Peixoto 
wrote:

> But my stream definition does not have a state store at all, Rocksdb or in
> memory... That's the most concerning part...
> On Wed, Jun 7, 2017 at 9:48 PM Sachin Mittal  wrote:
>
>> One instance with 10 threads may cause rocksdb issues.
>> What is the RAM you have?
>>
>> Also check CPU wait time. Many rocks db instances on one machine (depends
>> upon number of partitions) may cause lot of disk i/o causing wait times to
>> increase and hence slowing down the message processing causing frequent
>> rebalance's.
>>
>> Also what is your topic partitions. My experience is having one thread per
>> partition is ideal.
>>
>> Thanks
>> Sachin
>>
>>
>> On Thu, Jun 8, 2017 at 9:58 AM, João Peixoto 
>> wrote:
>>
>> > There is one instance with 10 threads.
>> >
>> > On Wed, Jun 7, 2017 at 9:07 PM Guozhang Wang 
>> wrote:
>> >
>> > > João,
>> > >
>> > > Do you also have multiple running instances in parallel, and how many
>> > > threads are your running within each instance?
>> > >
>> > > Guozhang
>> > >
>> > >
>> > > On Wed, Jun 7, 2017 at 3:18 PM, João Peixoto > >
>> > > wrote:
>> > >
>> > > > Eno before I do so I just want to be sure this would not be a
>> > duplicate.
>> > > I
>> > > > just found the following issues:
>> > > >
>> > > > * https://issues.apache.org/jira/browse/KAFKA-5167. Marked as being
>> > > fixed
>> > > > on 0.11.0.0/0.10.2.2 (both not released afaik)
>> > > > * https://issues.apache.org/jira/browse/KAFKA-5070. Currently in
>> > > progress
>> > > >
>> > > > On Wed, Jun 7, 2017 at 2:24 PM Eno Thereska > >
>> > > > wrote:
>> > > >
>> > > > > Hi there,
>> > > > >
>> > > > > This might be a bug, would you mind opening a JIRA (copy-pasting
>> > below
>> > > is
>> > > > > sufficient).
>> > > > >
>> > > > > Thanks
>> > > > > Eno
>> > > > > > On 7 Jun 2017, at 21:38, João Peixoto 
>> > > wrote:
>> > > > > >
>> > > > > > I'm using Kafka Streams 0.10.2.1 and I still see this error
>> > > > > >
>> > > > > > 2017-06-07 20:28:37.211  WARN 73 --- [ StreamThread-1]
>> > > > > > o.a.k.s.p.internals.StreamThread : Could not create task
>> > > 0_31.
>> > > > > Will
>> > > > > > retry.
>> > > > > >
>> > > > > > org.apache.kafka.streams.errors.LockException: task [0_31]
>> Failed
>> > to
>> > > > lock
>> > > > > > the state directory for task 0_31
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.streams.processor.internals.
>> > > > ProcessorStateManager.(ProcessorStateManager.java:100)
>> > > > > > ~[kafka-streams-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.streams.processor.internals.AbstractTask.(
>> > > > AbstractTask.java:73)
>> > > > > > ~[kafka-streams-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.streams.processor.internals.
>> > > > StreamTask.(StreamTask.java:108)
>> > > > > > ~[kafka-streams-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.streams.processor.internals.
>> > > > StreamThread.createStreamTask(StreamThread.java:864)
>> > > > > > [kafka-streams-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.streams.processor.internals.
>> > StreamThread$TaskCreator.
>> > > > createTask(StreamThread.java:1237)
>> > > > > > ~[kafka-streams-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.streams.processor.internals.StreamThread$
>> > > > AbstractTaskCreator.retryWithBackoff(StreamThread.java:1210)
>> > > > > > ~[kafka-streams-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.streams.processor.internals.
>> > > > StreamThread.addStreamTasks(StreamThread.java:967)
>> > > > > > [kafka-streams-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.streams.processor.internals.
>> > StreamThread.access$600(
>> > > > StreamThread.java:69)
>> > > > > > [kafka-streams-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.streams.processor.internals.StreamThread$1.
>> > > > onPartitionsAssigned(StreamThread.java:234)
>> > > > > > [kafka-streams-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.
>> > > > onJoinComplete(ConsumerCoordinator.java:259)
>> > > > > > [kafka-clients-0.10.2.1.jar!/:na]
>> > > > > > at
>> > > > > >
>> > > > > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
>> > > > joinGroupIfNeeded(AbstractCoordinator.java:352)
>> > > > > > [kafka-clients-0.10.2.1.jar!/:na]
>> > >

Re: kafka streams docs about topology picture only one source node

[john cheng]

Tks matthias, I did't consider about the situation you mentioned. you're
right.
one or more input topic can add to one single source node:
builder.addSource("source-1", "topic-1", "topic-2");

Or they can add to different source node:
builder.addSource("source-1", "topic-1");
builder.addSource("source-2", "topic-2");

in KStream world, one input topic to one source node:
KStream kstream1 = builder.stream("topic-1")
KStream kstream2 = builder.stream("topic-2")

many input topic to one source node:
KStream kstream1 = builder.stream("topic-1", "topic-2")

The key and value type by Processor API is configured by
StreamsConfig.KEY_SERDE_CLASS_CONFIG and StreamsConfig.
VALUE_SERDE_CLASS_CONFIG
While the key value type by DSL is strong type.

By now, the picture is good for me...

2017-06-09 12:37 GMT+08:00 Matthias J. Sax :

> Well. You can also ready multiple topics as a single KStream.
>
> > builder.stream("topic-1", "topic-2")
>
> Of course both topics must contain data with same key and value type.
> For this pattern, there is only one source node.
>
> There is no 1-to-1 relationship between input topics and source node,
> and thus, the picture is not wrong...
>
> Do you find that the picture is confusion/miss-leading?
>
>
> -Matthias
>
> On 6/8/17 5:58 PM, john cheng wrote:
> > Kafka streams topology can define one or many SourceNode.
> > The picture on official document <
> > http://kafka.apache.org/0102/documentation/streams#streams_
> architecture_tasks
> >>
> > only draw one source node in some place:
> > 1. Stream Partitions and Tasks
> > 2. Threading Model
> > 3. Local StateStore
> >
> > And the topology has two source topics: topicA and topicB
> > Should we change those picture above to two source node like the
> > Architecture part?
> >
>
>

Re: kafka streams docs about topology picture only one source node

[Matthias J. Sax]

Well. You can also ready multiple topics as a single KStream.

> builder.stream("topic-1", "topic-2")

Of course both topics must contain data with same key and value type.
For this pattern, there is only one source node.

There is no 1-to-1 relationship between input topics and source node,
and thus, the picture is not wrong...

Do you find that the picture is confusion/miss-leading?


-Matthias

On 6/8/17 5:58 PM, john cheng wrote:
> Kafka streams topology can define one or many SourceNode.
> The picture on official document <
> http://kafka.apache.org/0102/documentation/streams#streams_architecture_tasks
>>
> only draw one source node in some place:
> 1. Stream Partitions and Tasks
> 2. Threading Model
> 3. Local StateStore
> 
> And the topology has two source topics: topicA and topicB
> Should we change those picture above to two source node like the
> Architecture part?
> 



signature.asc
Description: OpenPGP digital signature

kafka streams docs about topology picture only one source node

[john cheng]

Kafka streams topology can define one or many SourceNode.
The picture on official document <
http://kafka.apache.org/0102/documentation/streams#streams_architecture_tasks
>
only draw one source node in some place:
1. Stream Partitions and Tasks
2. Threading Model
3. Local StateStore

And the topology has two source topics: topicA and topicB
Should we change those picture above to two source node like the
Architecture part?

Info regarding kafka topic

[BigData dev]

Hi,
I have a 3 node Kafka Broker cluster.
I have created a topic and the leader for the topic is broker 1(1001). And
the broker got died.
But when I see the information in zookeeper for the topic, I see the leader
is still set to broker 1 (1001) and isr is set to 1001. Is this a bug in
kafka, as now leader is died, the leader should have set to none.

*[zk: localhost:2181(CONNECTED) 7] get
/brokers/topics/t3/partitions/0/state*

*{"controller_epoch":1,"leader":1001,"version":1,"leader_epoch":1,"isr":[1001]}*

*cZxid = 0x10078*

*ctime = Thu Jun 08 14:50:07 PDT 2017*

*mZxid = 0x1008c*

*mtime = Thu Jun 08 14:51:09 PDT 2017*

*pZxid = 0x10078*

*cversion = 0*

*dataVersion = 1*

*aclVersion = 0*

*ephemeralOwner = 0x0*

*dataLength = 78*

*numChildren = 0*

*[zk: localhost:2181(CONNECTED) 8] *


And when I use describe command the output is

*[root@meets2 kafka-broker]# bin/kafka-topics.sh --describe --topic t3
--zookeeper localhost:2181*

*Topic:t3 PartitionCount:1 ReplicationFactor:2 Configs:*

*Topic: t3 Partition: 0 Leader: 1001 Replicas: 1001,1003 Isr: 1001*


When I use unavailable-partition option, I can know correctly.

*[root@meets2 kafka-broker]# bin/kafka-topics.sh --describe --topic t3
--zookeeper localhost:2181 --unavailable-partitions*

* Topic: t3 Partition: 0 Leader: 1001 Replicas: 1001,1003 Isr: 1001*


But in zookeeper topic state, the leader should have been set to none, not
the actual leader when the broker has died. Is this according to design or
is it a bug in Kafka. Could you please provide any information on this?


*Thanks,*

*Bharat*

Re: Kafka MirrorMaker error

[karan alang]

Thanks, that fixed the issue !


On Thu, Jun 8, 2017 at 11:58 AM, Mostafa Zarifyar 
wrote:

> maybe this will help.
>
> https://community.hortonworks.com/articles/24599/kafka-mirrormaker.html
>
> -M
>
> On Thu, Jun 8, 2017 at 11:42 AM, karan alang 
> wrote:
>
> > Hi All,
> >
> > I'm trying to transfer data between kafka clusters using Kafka
> MirrorMaker
> > & running into issues.
> >
> > I've created a consumer.config & producer.config files & using the
> command
> > shown below.
> > The error indicates - equirement failed: Missing required property
> > 'zookeeper.connect'
> >
> > --
> > $KAFKA10_HOME/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> > --consumer.config $KAFKA10_HOME/config/mmConsumer.config --num.streams 2
> > --producer.config $KAFKA10_HOME/config/mmProducer.config
> > --whitelist="mmtopic"
> > [2017-06-08 11:32:55,962] ERROR Exception when starting mirror maker.
> > (kafka.tools.MirrorMaker$)
> > java.lang.IllegalArgumentException: requirement failed: Missing required
> > property 'zookeeper.connect'
> > at scala.Predef$.require(Predef.scala:224)
> > at
> > kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:
> 177)
> > at kafka.utils.ZKConfig.(ZkUtils.scala:902)
> > at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:101)
> > at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:105)
> > at kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:306)
> > at kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:304)
> > at
> > scala.collection.TraversableLike$$anonfun$map$
> > 1.apply(TraversableLike.scala:234)
> > at
> > scala.collection.TraversableLike$$anonfun$map$
> > 1.apply(TraversableLike.scala:234)
> > at scala.collection.immutable.Range.foreach(Range.scala:160)
> > at scala.collection.TraversableLike$class.map(
> > TraversableLike.scala:234)
> > at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> > at kafka.tools.MirrorMaker$.createOldConsumers(
> MirrorMaker.scala:304)
> > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:233)
> > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > Exception in thread "main" java.lang.NullPointerException
> > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> >
> >
> > I tried adding the option (-zookeeper.connect = localhost:21810), it
> gives
> > error -> zookeeper.connect is not a recognized option
> >
> > -
> >
> > [2017-06-08 11:40:11,033] ERROR Exception when starting mirror maker.
> > (kafka.tools.MirrorMaker$)
> > joptsimple.UnrecognizedOptionException: zookeeper.connect is not a
> > recognized option
> > at
> > joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
> > at joptsimple.OptionParser.handleLongOptionToken(
> > OptionParser.java:449)
> > at
> > joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
> > at joptsimple.OptionParser.parse(OptionParser.java:381)
> > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:167)
> > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > Exception in thread "main" java.lang.NullPointerException
> > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> >
> >
> > Any ideas on what needs to be done ?
> >
>
> --
>
>
> Legal Disclaimer: The information contained in this message may be
> privileged and confidential. It is intended to be read only by the
> individual or entity to whom it is addressed or by their designee. If the
> reader of this message is not the intended recipient, you are on notice
> that any distribution of this message, in any form, is strictly prohibited.
> If you have received this message in error, please immediately notify the
> sender and delete or destroy any copy of this message
>

Re: Losing messages in Kafka Streams after upgrading

[Guozhang Wang]

We currently do not have a KIP for it yet.

On Wed, Jun 7, 2017 at 3:21 AM, Frank Lyaruu  wrote:

> I tried to use a TimestampExtractor that uses our timestamps from the
> messages, and use a 'map' operation on the KTable to set it to current, to
> have a precise point where I discard our original timestamps. That does not
> work, (I verified by writing a separate java Kafka Consumer and spit out
> the timestamps) as the TimestampExtractor only gets called once, and it
> will stick with that time. I did not really have a good reason not to
> simply use the WallclockTimeExtractor, and that one seems to do exactly
> what I wanted.
>
> So, I'm good! I am interested in the community discussion Guozhang
> mentions. Is there a KIP for that?
>
> regards, Frank
>
>
> On Mon, Jun 5, 2017 at 8:25 PM, Matthias J. Sax 
> wrote:
>
> > Frank,
> >
> > If you use "now", I assume you are calling System.currentTimeMillis().
> > If yes, you can also use predefined WallclockTimestampExtractor that
> > ships with Streams (no need to write your own one).
> >
> > > I thought that the Timestamp extractor would then also use
> > >> that updated timestamp as 'stream time', but I don't really see that
> > >> happening, so that assumption was wrong.
> >
> > Yes, this should happen. Not sure why you don't observe this. And thus,
> > the producer should use this timestamp to write the records.
> >
> > How did you verify the timestamps that are set for your output records?
> >
> >
> > -Matthias
> >
> >
> > On 6/5/17 6:15 AM, Frank Lyaruu wrote:
> > > Thanks Guozhang,
> > >
> > > I figured I could use a custom timestamp extractor, and set that
> > timestamp
> > > to 'now' when reading a source topic, as the original timestamp is
> pretty
> > > much irrelevant. I thought that the Timestamp extractor would then also
> > use
> > > that updated timestamp as 'stream time', but I don't really see that
> > > happening, so that assumption was wrong.
> > >
> > > If I could configure a timestamp extractor that would also be used by
> the
> > > producer I think I'd be in business, but right now I don't see an
> elegant
> > > way forward, so any ideas for work arounds are welcome.
> > >
> > > regards, Frank
> > >
> > > On Mon, Jun 5, 2017 at 7:01 AM, Guozhang Wang 
> > wrote:
> > >
> > >> Frank, thanks for sharing with your findings.
> > >>
> > >> I think this is a general issue to consider in Streams, and the
> > community
> > >> has been thinking about it: we write intermediate topics with the
> stream
> > >> time that is inherited from the source topic's timestamps, however
> that
> > >> timestamp is used for log rolling / retention etc as well, and these
> two
> > >> purposes (use timestamps in processing for out-of-ordering and late
> > >> arrivals, and operations on the Kafka topics) could rely on different
> > >> timestamp semantics. We need to revisit on timestamps can be
> maintained
> > >> across the topology in Streams.
> > >>
> > >> Guozhang
> > >>
> > >> On Sat, Jun 3, 2017 at 10:54 AM, Frank Lyaruu 
> > wrote:
> > >>
> > >>> Hi Matthias,
> > >>>
> > >>> Ok, that clarifies quite a bit. I never really went into the
> timestamp
> > >>> aspects, as time does not really play a role in my application (aside
> > >> from
> > >>> the repartition topics, I have no KStreams or Windowed operation,
> just
> > >>> different kind of KTable join).
> > >>>
> > >>> I do think that the fail case I see (With this version joining two
> > 'old'
> > >>> KTables causes a small percentage of records to vanish) is far from
> > >>> intuitive, and it somehow worked fine until a few weeks ago.
> > >>>
> > >>> I think your option 3 should work. I'll make a custom timestamp
> > extractor
> > >>> (I actually do have a timestamp in my messages), and I'll set it to
> the
> > >>> current time as they enter the streams application.
> > >>>
> > >>> Thanks, that helped, regards, Frank
> > >>>
> > >>> On Fri, Jun 2, 2017 at 9:17 PM, Matthias J. Sax <
> matth...@confluent.io
> > >
> > >>> wrote:
> > >>>
> >  Hi Frank,
> > 
> >  yes, retention policy is based on the embedded record timestamps and
> > >> not
> >  on system time. Thus, if you send messages with an old timestamp,
> they
> >  can trigger log/segment rolling.
> > 
> > >> I see that the repartition topics have timestamp.type =
> CreateTime,
> > >>> does
> > >> that mean it uses the timestamp of the
> > >> original message?
> > 
> >  Yes. That's the default setting on the broker side. For Streams, we
> >  maintain a so-called "stream time" that is computed based on the
> input
> >  record timestamps. This "stream time" is used to set the timestamp
> for
> >  records that are written by Stream. (so it's more or less the
> > timestamp
> >  of the input records).
> > 
> > >> Shouldn't that be LogAppendTime for repartition topics?
> > 
> >  No. Streams needs to preserve the original timestamp to guaranteed
> >  correct semantics for downstre

Parsing Kafka logs using Logstash

[IT Consultant]

Hi All,

Has anybody tried to parse Kafka logs using Logstash ?

If yes , can you please share patterns used to parse .

Thanks in advance.

RE: Larger payload size

[Tauzell, Dave]

Lots of large messages will slow down throughput.  From the client side you 
might want to have a client for large messages and one for the others so that 
they each have their own queue.

-Dave

-Original Message-
From: Ghosh, Achintya (Contractor) [mailto:achintya_gh...@comcast.com]
Sent: Thursday, June 8, 2017 2:48 PM
To: users@kafka.apache.org
Subject: Larger payload size

Hi there,

We observed if our payload size is larger we see "Failed to send; nested 
exception is org.apache.kafka.common.errors.RecordTooLargeException" execption 
so we changed the settings from 1 MB to 5 MB for both Producer and Consumer end.

Server.properties:

message.max.bytes=5242880
replica.fetch.max.bytes=5242880

Producer.properties

max.request.size=5242880

and at application level we are passing

max.partitions_fetch_bytes=5242880

As we made 1 MB to 5 MB, will be any performance impact or any other issue?

Thanks


This e-mail and any files transmitted with it are confidential, may contain 
sensitive information, and are intended solely for the use of the individual or 
entity to whom they are addressed. If you have received this e-mail in error, 
please notify the sender by reply e-mail immediately and destroy all copies of 
the e-mail and any attachments.

Larger payload size

[Ghosh, Achintya (Contractor)]

Hi there,

We observed if our payload size is larger we see "Failed to send; nested 
exception is org.apache.kafka.common.errors.RecordTooLargeException" execption 
so we changed the settings from 1 MB to 5 MB for both Producer and Consumer end.

Server.properties:

message.max.bytes=5242880
replica.fetch.max.bytes=5242880

Producer.properties

max.request.size=5242880

and at application level we are passing

max.partitions_fetch_bytes=5242880

As we made 1 MB to 5 MB, will be any performance impact or any other issue?

Thanks

Re: Kafka MirrorMaker error

[Mostafa Zarifyar]

maybe this will help.

https://community.hortonworks.com/articles/24599/kafka-mirrormaker.html

-M

On Thu, Jun 8, 2017 at 11:42 AM, karan alang  wrote:

> Hi All,
>
> I'm trying to transfer data between kafka clusters using Kafka MirrorMaker
> & running into issues.
>
> I've created a consumer.config & producer.config files & using the  command
> shown below.
> The error indicates - equirement failed: Missing required property
> 'zookeeper.connect'
>
> --
> $KAFKA10_HOME/bin/kafka-run-class.sh kafka.tools.MirrorMaker
> --consumer.config $KAFKA10_HOME/config/mmConsumer.config --num.streams 2
> --producer.config $KAFKA10_HOME/config/mmProducer.config
> --whitelist="mmtopic"
> [2017-06-08 11:32:55,962] ERROR Exception when starting mirror maker.
> (kafka.tools.MirrorMaker$)
> java.lang.IllegalArgumentException: requirement failed: Missing required
> property 'zookeeper.connect'
> at scala.Predef$.require(Predef.scala:224)
> at
> kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:177)
> at kafka.utils.ZKConfig.(ZkUtils.scala:902)
> at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:101)
> at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:105)
> at kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:306)
> at kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:304)
> at
> scala.collection.TraversableLike$$anonfun$map$
> 1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$
> 1.apply(TraversableLike.scala:234)
> at scala.collection.immutable.Range.foreach(Range.scala:160)
> at scala.collection.TraversableLike$class.map(
> TraversableLike.scala:234)
> at scala.collection.AbstractTraversable.map(Traversable.scala:104)
> at kafka.tools.MirrorMaker$.createOldConsumers(MirrorMaker.scala:304)
> at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:233)
> at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> Exception in thread "main" java.lang.NullPointerException
> at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
>
>
> I tried adding the option (-zookeeper.connect = localhost:21810), it gives
> error -> zookeeper.connect is not a recognized option
>
> -
>
> [2017-06-08 11:40:11,033] ERROR Exception when starting mirror maker.
> (kafka.tools.MirrorMaker$)
> joptsimple.UnrecognizedOptionException: zookeeper.connect is not a
> recognized option
> at
> joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
> at joptsimple.OptionParser.handleLongOptionToken(
> OptionParser.java:449)
> at
> joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
> at joptsimple.OptionParser.parse(OptionParser.java:381)
> at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:167)
> at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> Exception in thread "main" java.lang.NullPointerException
> at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
>
>
> Any ideas on what needs to be done ?
>

-- 
 

Legal Disclaimer: The information contained in this message may be 
privileged and confidential. It is intended to be read only by the 
individual or entity to whom it is addressed or by their designee. If the 
reader of this message is not the intended recipient, you are on notice 
that any distribution of this message, in any form, is strictly prohibited. 
If you have received this message in error, please immediately notify the 
sender and delete or destroy any copy of this message 

Kafka MirrorMaker error

[karan alang]

Hi All,

I'm trying to transfer data between kafka clusters using Kafka MirrorMaker
& running into issues.

I've created a consumer.config & producer.config files & using the  command
shown below.
The error indicates - equirement failed: Missing required property
'zookeeper.connect'

--
$KAFKA10_HOME/bin/kafka-run-class.sh kafka.tools.MirrorMaker
--consumer.config $KAFKA10_HOME/config/mmConsumer.config --num.streams 2
--producer.config $KAFKA10_HOME/config/mmProducer.config
--whitelist="mmtopic"
[2017-06-08 11:32:55,962] ERROR Exception when starting mirror maker.
(kafka.tools.MirrorMaker$)
java.lang.IllegalArgumentException: requirement failed: Missing required
property 'zookeeper.connect'
at scala.Predef$.require(Predef.scala:224)
at
kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:177)
at kafka.utils.ZKConfig.(ZkUtils.scala:902)
at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:101)
at kafka.consumer.ConsumerConfig.(ConsumerConfig.scala:105)
at kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:306)
at kafka.tools.MirrorMaker$$anonfun$3.apply(MirrorMaker.scala:304)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.tools.MirrorMaker$.createOldConsumers(MirrorMaker.scala:304)
at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:233)
at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
Exception in thread "main" java.lang.NullPointerException
at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
at kafka.tools.MirrorMaker.main(MirrorMaker.scala)


I tried adding the option (-zookeeper.connect = localhost:21810), it gives
error -> zookeeper.connect is not a recognized option

-

[2017-06-08 11:40:11,033] ERROR Exception when starting mirror maker.
(kafka.tools.MirrorMaker$)
joptsimple.UnrecognizedOptionException: zookeeper.connect is not a
recognized option
at
joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:449)
at
joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:381)
at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:167)
at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
Exception in thread "main" java.lang.NullPointerException
at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
at kafka.tools.MirrorMaker.main(MirrorMaker.scala)


Any ideas on what needs to be done ?

[DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

[Vahid S Hashemian]

Hi all,

I'm resending my earlier note hoping it would spark some conversation this 
time around :)

Thanks.
--Vahid




From:   "Vahid S Hashemian" 
To: dev , "Kafka User" 
Date:   05/30/2017 08:33 AM
Subject:KIP-163: Lower the Minimum Required ACL Permission of 
OffsetFetch



Hi,

I started a new KIP to improve the minimum required ACL permissions of 
some of the APIs: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-163%3A+Lower+the+Minimum+Required+ACL+Permission+of+OffsetFetch

The KIP is to address KAFKA-4585.

Feedback and suggestions are welcome!

Thanks.
--Vahid

Re: ConnectStandalone : not found JsonConverter running in IntelliJ

[Paolo Patierno]

So you are saying that in order to debug the ConnectStandalone from an IDE I 
have to put the "json" module on the classpath in a run configuration ? or the 
way I'm following attaching a remote debugger is the right one ?


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience



From: Randall Hauch 
Sent: Thursday, June 8, 2017 4:23 PM
To: users@kafka.apache.org
Subject: Re: ConnectStandalone : not found JsonConverter running in IntelliJ

I expect that this is because the Gradle file does defines the JSON
converter module as "test". See
https://github.com/apache/kafka/blob/trunk/build.gradle#L1082

This is likely because this ensures that the runtime source code will not
directly depend upon the JSON converter module. Any custom software
component (such as custom test modules or applications) that relies upon
these modules and tries to execute Connect workers is thus acting as its
own runtime and should declare the dependency on the JSON converter module
as well as any other modules needed by that environment.

On Thu, Jun 8, 2017 at 11:09 AM, Paolo Patierno  wrote:

> Yes exactly ... I see that for the Connect "runtime" module, the "json"
> module is a dependency as "Test". I'm able to avoid the exception if I
> change the dependency to be "Compile" so that it will be available even in
> test and at runtime.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Randall Hauch 
> Sent: Thursday, June 8, 2017 4:05 PM
> To: users@kafka.apache.org
> Subject: Re: ConnectStandalone : not found JsonConverter running in
> IntelliJ
>
> Are you talking about importing the Apache Kafka project into IntelliJ? If
> so, IntelliJ should have the JSON converter's source code in the IDE's
> project. Is that correct? And you're still getting the ConfigException for
> the "org.apache.kafka.connect.json.JsonConverter"?
>
> On Thu, Jun 8, 2017 at 10:57 AM, Paolo Patierno 
> wrote:
>
> > Hi Randall,
> >
> >
> > after running the "./gradlew idea" command I imported the project into
> > IntelliJ thanks to the Gradle support.
> >
> > I did no changes on that.
> >
> >
> > For now the way I'm debugging is just building the kafka binaries, then
> > running the connect example on the command line with debugging enabled
> > (DEBUG_SUSPEND_FLAG=y and KAFKA_DEBUG=y) and then attaching a remote
> > session from IntelliJ.
> >
> > I hope there is a simpler way to do that ;)
> >
> >
> > Thanks,
> >
> > Paolo.
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: Randall Hauch 
> > Sent: Thursday, June 8, 2017 3:14 PM
> > To: users@kafka.apache.org
> > Subject: Re: ConnectStandalone : not found JsonConverter running in
> > IntelliJ
> >
> > Hi, Paolo.
> >
> > How are the Kafka Connect libraries loaded into your IntelliJ project? If
> > they are loaded as external libraries, where in the order of the external
> > libraries does the "org.apache.kafka:connector:connect-json" Maven
> module
> > appear? Or, is that module loaded as source?
> >
> > Best regards,
> >
> > Randall
> >
> > On Thu, Jun 8, 2017 at 8:17 AM, Paolo Patierno 
> wrote:
> >
> > > Hi,
> > >
> > >
> > > I'm trying to run the ConnectStandalone application inside IntelliJ
> > > providing the worker and console source properties files from the
> config
> > > dir but I receive the following exception :
> > >
> > >
> > > Exception in thread "main" org.apache.kafka.common.
> > config.ConfigException:
> > > Invalid value org.apache.kafka.connect.json.JsonConverter for
> > > configuration internal.key.converter: Class
> > org.apache.kafka.connect.json.JsonConverter
> > > could not be found.
> > > at org.apache.kafka.common.config.ConfigDef.parseType(
> > ConfigDef.java:711)
> > > at org.apache.kafka.common.config.ConfigDef.parseValue(
> > ConfigDef.java:457)
> > > at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
> > > at org.apache.kafka.common.config.AbstractConfig.(
> > > AbstractConfig.java:62)
> > > at org.apache.kafka.common.config.AbstractConfig.(
> > > AbstractConfig.java:75)
> > > at org.apache.kafka.connect.runtime.WorkerConfig.(
> > > WorkerConfig.java:19

Re: ConnectStandalone : not found JsonConverter running in IntelliJ

[Randall Hauch]

I expect that this is because the Gradle file does defines the JSON
converter module as "test". See
https://github.com/apache/kafka/blob/trunk/build.gradle#L1082

This is likely because this ensures that the runtime source code will not
directly depend upon the JSON converter module. Any custom software
component (such as custom test modules or applications) that relies upon
these modules and tries to execute Connect workers is thus acting as its
own runtime and should declare the dependency on the JSON converter module
as well as any other modules needed by that environment.

On Thu, Jun 8, 2017 at 11:09 AM, Paolo Patierno  wrote:

> Yes exactly ... I see that for the Connect "runtime" module, the "json"
> module is a dependency as "Test". I'm able to avoid the exception if I
> change the dependency to be "Compile" so that it will be available even in
> test and at runtime.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Randall Hauch 
> Sent: Thursday, June 8, 2017 4:05 PM
> To: users@kafka.apache.org
> Subject: Re: ConnectStandalone : not found JsonConverter running in
> IntelliJ
>
> Are you talking about importing the Apache Kafka project into IntelliJ? If
> so, IntelliJ should have the JSON converter's source code in the IDE's
> project. Is that correct? And you're still getting the ConfigException for
> the "org.apache.kafka.connect.json.JsonConverter"?
>
> On Thu, Jun 8, 2017 at 10:57 AM, Paolo Patierno 
> wrote:
>
> > Hi Randall,
> >
> >
> > after running the "./gradlew idea" command I imported the project into
> > IntelliJ thanks to the Gradle support.
> >
> > I did no changes on that.
> >
> >
> > For now the way I'm debugging is just building the kafka binaries, then
> > running the connect example on the command line with debugging enabled
> > (DEBUG_SUSPEND_FLAG=y and KAFKA_DEBUG=y) and then attaching a remote
> > session from IntelliJ.
> >
> > I hope there is a simpler way to do that ;)
> >
> >
> > Thanks,
> >
> > Paolo.
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
> >
> > 
> > From: Randall Hauch 
> > Sent: Thursday, June 8, 2017 3:14 PM
> > To: users@kafka.apache.org
> > Subject: Re: ConnectStandalone : not found JsonConverter running in
> > IntelliJ
> >
> > Hi, Paolo.
> >
> > How are the Kafka Connect libraries loaded into your IntelliJ project? If
> > they are loaded as external libraries, where in the order of the external
> > libraries does the "org.apache.kafka:connector:connect-json" Maven
> module
> > appear? Or, is that module loaded as source?
> >
> > Best regards,
> >
> > Randall
> >
> > On Thu, Jun 8, 2017 at 8:17 AM, Paolo Patierno 
> wrote:
> >
> > > Hi,
> > >
> > >
> > > I'm trying to run the ConnectStandalone application inside IntelliJ
> > > providing the worker and console source properties files from the
> config
> > > dir but I receive the following exception :
> > >
> > >
> > > Exception in thread "main" org.apache.kafka.common.
> > config.ConfigException:
> > > Invalid value org.apache.kafka.connect.json.JsonConverter for
> > > configuration internal.key.converter: Class
> > org.apache.kafka.connect.json.JsonConverter
> > > could not be found.
> > > at org.apache.kafka.common.config.ConfigDef.parseType(
> > ConfigDef.java:711)
> > > at org.apache.kafka.common.config.ConfigDef.parseValue(
> > ConfigDef.java:457)
> > > at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
> > > at org.apache.kafka.common.config.AbstractConfig.(
> > > AbstractConfig.java:62)
> > > at org.apache.kafka.common.config.AbstractConfig.(
> > > AbstractConfig.java:75)
> > > at org.apache.kafka.connect.runtime.WorkerConfig.(
> > > WorkerConfig.java:197)
> > > at org.apache.kafka.connect.runtime.standalone.
> StandaloneConfig.(
> > > StandaloneConfig.java:42)
> > > at org.apache.kafka.connect.cli.ConnectStandalone.main(
> > > ConnectStandalone.java:70)
> > >
> > >
> > > The converter is available in the JSON module of course.
> > >
> > >
> > > Any idea ?
> > >
> > >
> > > Thanks,
> > >
> > > Paolo
> > >
> > >
> > > Paolo Patierno
> > > Senior Software Engineer (IoT) @ Red Hat
> > > Microsoft MVP on Windows Embedded & IoT
> > > Microsoft Azure Advisor
> > >
> > > Twitter : @ppatierno
> > > Linkedin : paolopatierno
> > > Blog : DevExperience
> > >
> >
>

Re: ConnectStandalone : not found JsonConverter running in IntelliJ

[Paolo Patierno]

Yes exactly ... I see that for the Connect "runtime" module, the "json" module 
is a dependency as "Test". I'm able to avoid the exception if I change the 
dependency to be "Compile" so that it will be available even in test and at 
runtime.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience



From: Randall Hauch 
Sent: Thursday, June 8, 2017 4:05 PM
To: users@kafka.apache.org
Subject: Re: ConnectStandalone : not found JsonConverter running in IntelliJ

Are you talking about importing the Apache Kafka project into IntelliJ? If
so, IntelliJ should have the JSON converter's source code in the IDE's
project. Is that correct? And you're still getting the ConfigException for
the "org.apache.kafka.connect.json.JsonConverter"?

On Thu, Jun 8, 2017 at 10:57 AM, Paolo Patierno  wrote:

> Hi Randall,
>
>
> after running the "./gradlew idea" command I imported the project into
> IntelliJ thanks to the Gradle support.
>
> I did no changes on that.
>
>
> For now the way I'm debugging is just building the kafka binaries, then
> running the connect example on the command line with debugging enabled
> (DEBUG_SUSPEND_FLAG=y and KAFKA_DEBUG=y) and then attaching a remote
> session from IntelliJ.
>
> I hope there is a simpler way to do that ;)
>
>
> Thanks,
>
> Paolo.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Randall Hauch 
> Sent: Thursday, June 8, 2017 3:14 PM
> To: users@kafka.apache.org
> Subject: Re: ConnectStandalone : not found JsonConverter running in
> IntelliJ
>
> Hi, Paolo.
>
> How are the Kafka Connect libraries loaded into your IntelliJ project? If
> they are loaded as external libraries, where in the order of the external
> libraries does the "org.apache.kafka:connector:connect-json" Maven module
> appear? Or, is that module loaded as source?
>
> Best regards,
>
> Randall
>
> On Thu, Jun 8, 2017 at 8:17 AM, Paolo Patierno  wrote:
>
> > Hi,
> >
> >
> > I'm trying to run the ConnectStandalone application inside IntelliJ
> > providing the worker and console source properties files from the config
> > dir but I receive the following exception :
> >
> >
> > Exception in thread "main" org.apache.kafka.common.
> config.ConfigException:
> > Invalid value org.apache.kafka.connect.json.JsonConverter for
> > configuration internal.key.converter: Class
> org.apache.kafka.connect.json.JsonConverter
> > could not be found.
> > at org.apache.kafka.common.config.ConfigDef.parseType(
> ConfigDef.java:711)
> > at org.apache.kafka.common.config.ConfigDef.parseValue(
> ConfigDef.java:457)
> > at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
> > at org.apache.kafka.common.config.AbstractConfig.(
> > AbstractConfig.java:62)
> > at org.apache.kafka.common.config.AbstractConfig.(
> > AbstractConfig.java:75)
> > at org.apache.kafka.connect.runtime.WorkerConfig.(
> > WorkerConfig.java:197)
> > at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.(
> > StandaloneConfig.java:42)
> > at org.apache.kafka.connect.cli.ConnectStandalone.main(
> > ConnectStandalone.java:70)
> >
> >
> > The converter is available in the JSON module of course.
> >
> >
> > Any idea ?
> >
> >
> > Thanks,
> >
> > Paolo
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
>

Re: ConnectStandalone : not found JsonConverter running in IntelliJ

[Randall Hauch]

Are you talking about importing the Apache Kafka project into IntelliJ? If
so, IntelliJ should have the JSON converter's source code in the IDE's
project. Is that correct? And you're still getting the ConfigException for
the "org.apache.kafka.connect.json.JsonConverter"?

On Thu, Jun 8, 2017 at 10:57 AM, Paolo Patierno  wrote:

> Hi Randall,
>
>
> after running the "./gradlew idea" command I imported the project into
> IntelliJ thanks to the Gradle support.
>
> I did no changes on that.
>
>
> For now the way I'm debugging is just building the kafka binaries, then
> running the connect example on the command line with debugging enabled
> (DEBUG_SUSPEND_FLAG=y and KAFKA_DEBUG=y) and then attaching a remote
> session from IntelliJ.
>
> I hope there is a simpler way to do that ;)
>
>
> Thanks,
>
> Paolo.
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>
>
> 
> From: Randall Hauch 
> Sent: Thursday, June 8, 2017 3:14 PM
> To: users@kafka.apache.org
> Subject: Re: ConnectStandalone : not found JsonConverter running in
> IntelliJ
>
> Hi, Paolo.
>
> How are the Kafka Connect libraries loaded into your IntelliJ project? If
> they are loaded as external libraries, where in the order of the external
> libraries does the "org.apache.kafka:connector:connect-json" Maven module
> appear? Or, is that module loaded as source?
>
> Best regards,
>
> Randall
>
> On Thu, Jun 8, 2017 at 8:17 AM, Paolo Patierno  wrote:
>
> > Hi,
> >
> >
> > I'm trying to run the ConnectStandalone application inside IntelliJ
> > providing the worker and console source properties files from the config
> > dir but I receive the following exception :
> >
> >
> > Exception in thread "main" org.apache.kafka.common.
> config.ConfigException:
> > Invalid value org.apache.kafka.connect.json.JsonConverter for
> > configuration internal.key.converter: Class
> org.apache.kafka.connect.json.JsonConverter
> > could not be found.
> > at org.apache.kafka.common.config.ConfigDef.parseType(
> ConfigDef.java:711)
> > at org.apache.kafka.common.config.ConfigDef.parseValue(
> ConfigDef.java:457)
> > at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
> > at org.apache.kafka.common.config.AbstractConfig.(
> > AbstractConfig.java:62)
> > at org.apache.kafka.common.config.AbstractConfig.(
> > AbstractConfig.java:75)
> > at org.apache.kafka.connect.runtime.WorkerConfig.(
> > WorkerConfig.java:197)
> > at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.(
> > StandaloneConfig.java:42)
> > at org.apache.kafka.connect.cli.ConnectStandalone.main(
> > ConnectStandalone.java:70)
> >
> >
> > The converter is available in the JSON module of course.
> >
> >
> > Any idea ?
> >
> >
> > Thanks,
> >
> > Paolo
> >
> >
> > Paolo Patierno
> > Senior Software Engineer (IoT) @ Red Hat
> > Microsoft MVP on Windows Embedded & IoT
> > Microsoft Azure Advisor
> >
> > Twitter : @ppatierno
> > Linkedin : paolopatierno
> > Blog : DevExperience
> >
>

Re: ConnectStandalone : not found JsonConverter running in IntelliJ

[Paolo Patierno]

Hi Randall,


after running the "./gradlew idea" command I imported the project into IntelliJ 
thanks to the Gradle support.

I did no changes on that.


For now the way I'm debugging is just building the kafka binaries, then running 
the connect example on the command line with debugging enabled 
(DEBUG_SUSPEND_FLAG=y and KAFKA_DEBUG=y) and then attaching a remote session 
from IntelliJ.

I hope there is a simpler way to do that ;)


Thanks,

Paolo.


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience



From: Randall Hauch 
Sent: Thursday, June 8, 2017 3:14 PM
To: users@kafka.apache.org
Subject: Re: ConnectStandalone : not found JsonConverter running in IntelliJ

Hi, Paolo.

How are the Kafka Connect libraries loaded into your IntelliJ project? If
they are loaded as external libraries, where in the order of the external
libraries does the "org.apache.kafka:connector:connect-json" Maven module
appear? Or, is that module loaded as source?

Best regards,

Randall

On Thu, Jun 8, 2017 at 8:17 AM, Paolo Patierno  wrote:

> Hi,
>
>
> I'm trying to run the ConnectStandalone application inside IntelliJ
> providing the worker and console source properties files from the config
> dir but I receive the following exception :
>
>
> Exception in thread "main" org.apache.kafka.common.config.ConfigException:
> Invalid value org.apache.kafka.connect.json.JsonConverter for
> configuration internal.key.converter: Class 
> org.apache.kafka.connect.json.JsonConverter
> could not be found.
> at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:711)
> at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:457)
> at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
> at org.apache.kafka.common.config.AbstractConfig.(
> AbstractConfig.java:62)
> at org.apache.kafka.common.config.AbstractConfig.(
> AbstractConfig.java:75)
> at org.apache.kafka.connect.runtime.WorkerConfig.(
> WorkerConfig.java:197)
> at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.(
> StandaloneConfig.java:42)
> at org.apache.kafka.connect.cli.ConnectStandalone.main(
> ConnectStandalone.java:70)
>
>
> The converter is available in the JSON module of course.
>
>
> Any idea ?
>
>
> Thanks,
>
> Paolo
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>

Re: ConnectStandalone : not found JsonConverter running in IntelliJ

[Randall Hauch]

Hi, Paolo.

How are the Kafka Connect libraries loaded into your IntelliJ project? If
they are loaded as external libraries, where in the order of the external
libraries does the "org.apache.kafka:connector:connect-json" Maven module
appear? Or, is that module loaded as source?

Best regards,

Randall

On Thu, Jun 8, 2017 at 8:17 AM, Paolo Patierno  wrote:

> Hi,
>
>
> I'm trying to run the ConnectStandalone application inside IntelliJ
> providing the worker and console source properties files from the config
> dir but I receive the following exception :
>
>
> Exception in thread "main" org.apache.kafka.common.config.ConfigException:
> Invalid value org.apache.kafka.connect.json.JsonConverter for
> configuration internal.key.converter: Class 
> org.apache.kafka.connect.json.JsonConverter
> could not be found.
> at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:711)
> at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:457)
> at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
> at org.apache.kafka.common.config.AbstractConfig.(
> AbstractConfig.java:62)
> at org.apache.kafka.common.config.AbstractConfig.(
> AbstractConfig.java:75)
> at org.apache.kafka.connect.runtime.WorkerConfig.(
> WorkerConfig.java:197)
> at org.apache.kafka.connect.runtime.standalone.StandaloneConfig.(
> StandaloneConfig.java:42)
> at org.apache.kafka.connect.cli.ConnectStandalone.main(
> ConnectStandalone.java:70)
>
>
> The converter is available in the JSON module of course.
>
>
> Any idea ?
>
>
> Thanks,
>
> Paolo
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience
>

Re: Mirror maker Configs 0.9.0

[Mohan]

Hello Experts,

 

I am trying to replicate data between On Prem Kafka Cluster(Source) and
another Kafka cluster (Target?) set up in a Cloud provider environment.

The On-Prem to cloud is connected via IPSec VPN and  Mirror maker tool is
used  on 0.10.2.x version.

 

The consumer configs are as follows:

bootstrap.servers=x.x.x.x:2181

zookeeper.connection.timeout.ms=6000

group.id=mirrormaker

auto.offset.reset=earliest

exclude.internal.topics=true

 

The producer configs are as follows:

bootstrap.servers=y.y.y.y:9092

compression.codec=2

batch.size=65536

client.id=mirrormaker

 

x.x.x.x and y.y.y.y are reacheable over the VPN tunnel

 

I can see the date in the On-Prem consumer side, however there is no data
received in the consumer on the Cloud end.

Just wondering if Kafka can send data across IPsec encrypted tunnel links?

 

Kafka Cluster- consumer issue

[Vinay Kumar]

Hi,

I setup Kafka Cluster with 3 kafka brokers in 3 separate VMs.

Upon testing the Fault tolerance among the 3 brokers, I see an issue with
the Consumer in displaying messages.

 >> The below command display the messages, along with a content
stating “*Using
the ConsoleConsumer with old consumer is deprecated and will be removed in
a future major release. Consider using the new consumer by passing
[bootstrap-server] instead of [zookeeper].*“

  bin/kafka-console-consumer.sh --zookeeper vm1:2181,vm2:2181,vm3:2181
-topic multikafka --from-beginning

Using the ConsoleConsumer with old consumer is deprecated and will be
removed in a future major release. Consider using the new consumer by
passing [bootstrap-server] instead of [zookeeper].

testing multikafka cluster

test2t

est

But, when tried by giving --bootstrap instead of --zookeeper as suggested
in the content, no messages are being displayed.

bin/kafka-console-consumer.sh --bootstrap-server vm1:9092,vm2:9092,vm3:9092
-topic multikafka --from-beginning

Is this a bug ? Please suggest.



Thanks,

Vinay

kafka-streams repeatedly rebalances on start up

[Tom Dearman]

We wanted to have less frequent polling in kafka streams (mostly because we 
have noticed quite a lot of object creation when polling a queue with no new 
messages on it), so we have set polling to 10 seconds.  On start up when 
rebalance first happens, the onPartitionsAssigned of ConsumerRebalanceListener 
keeps causing a rebalance as it takes too long and the heart beat thread hasn’t 
started up. I have applied the relevant configuration to the WordCountProcessor 
demo class from kafka streams and it does the same. The additional config I 
apply to the demo class is:

props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG,1);
props.put(StreamsConfig.POLL_MS_CONFIG, 1);
props.put(StreamsConfig.CONSUMER_PREFIX + 
ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 1);
props.put(StreamsConfig.CONSUMER_PREFIX + 
ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 3);
props.put(StreamsConfig.CONSUMER_PREFIX + 
ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 1);


This causes the following logging:

10630 [StreamThread-1] INFO 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining 
group streams-wordcount-processor
10632 [StreamThread-1] DEBUG 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] found [streams-file-input] topics possibly 
matching regex
10640 [StreamThread-1] INFO 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Constructed client metadata 
{0f988636-b4f7-42af-b84b-249c9feb5fcb=ClientMetadata{hostInfo=null, 
consumers=[streams-wordcount-processor-0f988636-b4f7-42af-b84b-249c9feb5fcb-StreamThread-1-consumer-774aa883-3900-4d50-b4f8-d9846486f1fe],
 state=[activeTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([]) 
prevAssignedTasks: ([]) capacity: 1.0 cost: 0.0]}} from the member 
subscriptions.
10643 [StreamThread-1] DEBUG 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Starting to validate internal topics in 
partition assignor.
10643 [StreamThread-1] INFO 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Completed validating internal topics in 
partition assignor
10643 [StreamThread-1] DEBUG 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Created repartition topics [] from the parsed 
topology.
10644 [StreamThread-1] DEBUG 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Starting to validate internal topics in 
partition assignor.
40664 [StreamThread-1] INFO 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Completed validating internal topics in 
partition assignor
40664 [StreamThread-1] DEBUG 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Created state changelog topics 
{streams-wordcount-processor-Counts-changelog=org.apache.kafka.streams.processor.internals.StreamPartitionAssignor$InternalTopicMetadata@a3de218}
 from the parsed topology.
40664 [StreamThread-1] DEBUG 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Assigning tasks [0_0] to clients 
{0f988636-b4f7-42af-b84b-249c9feb5fcb=[activeTasks: ([]) assignedTasks: ([]) 
prevActiveTasks: ([]) prevAssignedTasks: ([]) capacity: 1.0 cost: 0.0]} with 
number of replicas 0
40666 [StreamThread-1] INFO 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Assigned tasks to clients as 
{0f988636-b4f7-42af-b84b-249c9feb5fcb=[activeTasks: ([0_0]) assignedTasks: 
([0_0]) prevActiveTasks: ([]) prevAssignedTasks: ([]) capacity: 1.0 cost: 0.5]}.
40670 [StreamThread-1] INFO 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining 
group streams-wordcount-processor
40670 [StreamThread-1] DEBUG 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] found [streams-file-input] topics possibly 
matching regex
40672 [StreamThread-1] INFO 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Constructed client metadata 
{0f988636-b4f7-42af-b84b-249c9feb5fcb=ClientMetadata{hostInfo=null, 
consumers=[streams-wordcount-processor-0f988636-b4f7-42af-b84b-249c9feb5fcb-StreamThread-1-consumer-618818cd-e6dc-4ece-af8c-1a80b41de1c8],
 state=[activeTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([]) 
prevAssignedTasks: ([]) capacity: 1.0 cost: 0.0]}} from the member 
subscriptions.
40672 [StreamThread-1] DEBUG 
org.apache.kafka.streams.processor.internals.StreamPartitionAssignor - 
stream-thread [StreamThread-1] Starting to validate internal topics in 
partition assignor.
40672 [StreamThread-1] INFO 
org.apache.kafka.streams.processor.internals.StreamPartitionAss

[VOTE] 0.11.0.0 RC0

[Ismael Juma]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 0.11.0.0. It's
worth noting that there are a small number of unresolved issues (including
documentation and system tests) related to the new AdminClient and
Exactly-once functionality[1] that we hope to resolve in the next few days.
To encourage early testing, we are releasing the first release candidate
now, but there will be at least one more release candidate.

Any and all testing is welcome, but the following areas are worth
highlighting:

1. Client developers should verify that their clients can produce/consume
to/from 0.11.0 brokers (ideally with compressed and uncompressed data).
Even though we have compatibility tests for older Java clients and we have
verified that librdkafka works fine, the only way to be sure is to test
every client.
2. Performance and stress testing. Heroku and LinkedIn have helped with
this in the past (and issues have been found and fixed).
3. End users can verify that their apps work correctly with the new release.

This is a major version release of Apache Kafka. It includes 32 new KIPs. See
the release notes and release plan (https://cwiki.apache.org/
confluence/display/KAFKA/Release+Plan+0.11.0.0) for more details. A few
feature highlights:

* Exactly-once delivery and transactional messaging
* Streams exactly-once semantics
* Admin client with support for topic, ACLs and config management
* Record headers
* Request rate quotas
* Improved resiliency: replication protocol improvement and single-threaded
controller
* Richer and more efficient message format

Release notes for the 0.11.0.0 release:
http://home.apache.org/~ijuma/kafka-0.11.0.0-rc0/RELEASE_NOTES.html

Kafka's KEYS file containing PGP keys we use to sign the release:
http://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
http://home.apache.org/~ijuma/kafka-0.11.0.0-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
http://home.apache.org/~ijuma/kafka-0.11.0.0-rc0/javadoc/

* Tag to be voted upon (off 0.11.0 branch) is the 0.11.0.0 tag:
https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
d5ee02b187fafe08b63deb52e6b07c8d1d12f18d

* Documentation:
http://kafka.apache.org/0110/documentation.html

* Protocol:
http://kafka.apache.org/0110/protocol.html

* Successful Jenkins builds for the 0.11.0 branch:
Unit/integration tests: https://builds.apache.org/job/kafka-0.11.0-jdk7/121/

Thanks,
Ismael

[1] https://issues.apache.org/jira/issues/?jql=project%20%
3D%20KAFKA%20AND%20fixVersion%20%3D%200.11.0.0%20AND%20resolution%20%3D%
20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%
20DESC%2C%20created%20ASC

ticketing system Design

[Abhimanyu Nagrath]

Hi ,

Is Apache Kafka along with storm can be used to design a ticketing system.
By ticketing system, I mean that there are millions of tasks stored in
Kafka queues and there are processes/humans to take some actions on the
task. there are come constraints that same task should not be assigned to
two processes/humans and if a task flows to a process/human and no action
is performed it should be reassigned.
 I am not sure whether this can be solved using Kafka.Any help is
appreciated



Regards,
Abhimanyu

ConnectStandalone : not found JsonConverter running in IntelliJ

[Paolo Patierno]

Hi,


I'm trying to run the ConnectStandalone application inside IntelliJ providing 
the worker and console source properties files from the config dir but I 
receive the following exception :


Exception in thread "main" org.apache.kafka.common.config.ConfigException: 
Invalid value org.apache.kafka.connect.json.JsonConverter for configuration 
internal.key.converter: Class org.apache.kafka.connect.json.JsonConverter could 
not be found.
at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:711)
at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:457)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:450)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:62)
at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:75)
at org.apache.kafka.connect.runtime.WorkerConfig.(WorkerConfig.java:197)
at 
org.apache.kafka.connect.runtime.standalone.StandaloneConfig.(StandaloneConfig.java:42)
at 
org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:70)


The converter is available in the JSON module of course.


Any idea ?


Thanks,

Paolo


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience

network errors?

[Jon Yeargers]

What do these messages mean:

WARN  kafka.network.Processor  - Attempting to send response via channel
for which there is no open connection, connection id 2

Re: Debugging Kafka Streams Windowing

[Mahendra Kariya]

Yes. To some extent. But the rebalancing is now taking a lot of time. There
are situations where we have to manually restart the Streams app because
rebalancing is kind of "stuck" for several minutes.

On 7 June 2017 at 06:28, Garrett Barton  wrote:

> Mahendra,
>
>  Did increasing those two properties do the trick?  I am running into this
> exact issue testing streams out on a single Kafka instance.  Yet I can
> manually start a consumer and read the topics fine while its busy doing
> this dead stuffs.
>
> On Tue, May 23, 2017 at 12:30 AM, Mahendra Kariya <
> mahendra.kar...@go-jek.com> wrote:
>
> > On 22 May 2017 at 16:09, Guozhang Wang  wrote:
> >
> > > For
> > > that issue I'd suspect that there is a network issue, or maybe the
> > network
> > > is just saturated already and the heartbeat request / response were not
> > > exchanged in time between the consumer and the broker, or the sockets
> > being
> > > dropped because of socket limit. Under this cases not all consumers may
> > be
> > > affected, but since the associated issue is from "AbstractCoordinator"
> > > class which is part of the consumer client, I'd still be surprised if
> it
> > is
> > > actually due to Streams itself with the same consumer config settings,
> > but
> > > not to consumers.
> > >
> >
> > Yes. This is the conclusion that even we are coming to after further
> > investigation. But didn't want to post it here until we were sure.
> >
> > We are experimenting with increasing the default timeouts, particularly
> > hearbeat.interval.ms and session.timeout.ms. So far, the things have
> been
> > running fine. But we will let it run for a few more days before closing
> > this issue.
> >
>