Re: kafka streaming rocks db lock bug?

[Ara Ebrahimi]

This was in 10.1.0. What happened was that a kafka broker went down and then 
this happened on the kafka streaming instance which had connection to this 
broker. I can send you all logs I got.

Ara.

On Oct 24, 2016, at 10:41 PM, Guozhang Wang 
mailto:wangg...@gmail.com>> wrote:

Hello Ara,

Your encountered issue seems to be KAFKA-3812
, and KAFKA-3938
. Could you try to
upgrade to the newly released 0.10.1.0 version and see if this issue goes
away? If not I would love to investigate this issue further with you.


Guozhang



Guozhang


On Sun, Oct 23, 2016 at 1:45 PM, Ara Ebrahimi 
mailto:ara.ebrah...@argyledata.com>>
wrote:

And then this on a different node:

2016-10-23 13:43:57 INFO  StreamThread:286 - stream-thread
[StreamThread-3] Stream thread shutdown complete
2016-10-23 13:43:57 ERROR StreamPipeline:169 - An exception has occurred
org.apache.kafka.streams.errors.StreamsException: stream-thread
[StreamThread-3] Failed to rebalance
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(
StreamThread.java:401)
at org.apache.kafka.streams.processor.internals.
StreamThread.run(StreamThread.java:235)
Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error
while creating the state manager
at org.apache.kafka.streams.processor.internals.AbstractTask.(
AbstractTask.java:72)
at org.apache.kafka.streams.processor.internals.
StreamTask.(StreamTask.java:90)
at org.apache.kafka.streams.processor.internals.
StreamThread.createStreamTask(StreamThread.java:622)
at org.apache.kafka.streams.processor.internals.
StreamThread.addStreamTasks(StreamThread.java:649)
at org.apache.kafka.streams.processor.internals.StreamThread.access$000(
StreamThread.java:69)
at org.apache.kafka.streams.processor.internals.StreamThread$1.
onPartitionsAssigned(StreamThread.java:120)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.
onJoinComplete(ConsumerCoordinator.java:228)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
joinGroupIfNeeded(AbstractCoordinator.java:313)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
ensureActiveGroup(AbstractCoordinator.java:277)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(
ConsumerCoordinator.java:259)
at org.apache.kafka.clients.consumer.KafkaConsumer.
pollOnce(KafkaConsumer.java:1013)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(
KafkaConsumer.java:979)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(
StreamThread.java:398)
... 1 more
Caused by: java.io.IOException: task [7_1] Failed to lock the state
directory: /tmp/kafka-streams/argyle-streams/7_1
at org.apache.kafka.streams.processor.internals.
ProcessorStateManager.(ProcessorStateManager.java:98)
at org.apache.kafka.streams.processor.internals.AbstractTask.(
AbstractTask.java:69)
... 13 more

Ara.

On Oct 23, 2016, at 1:24 PM, Ara Ebrahimi 
mailto:ara.ebrah...@argyledata.com><
mailto:ara.ebrah...@argyledata.com>> wrote:

Hi,

This happens when I hammer our 5 kafka streaming nodes (each with 4
streaming threads) hard enough for an hour or so:

2016-10-23 13:04:17 ERROR StreamThread:324 - stream-thread
[StreamThread-2] Failed to flush state for StreamTask 3_8:
org.apache.kafka.streams.errors.ProcessorStateException: task [3_8]
Failed to flush state store streams-data-record-stats-avro-br-store
at org.apache.kafka.streams.processor.internals.
ProcessorStateManager.flush(ProcessorStateManager.java:322)
at org.apache.kafka.streams.processor.internals.AbstractTask.flushState(
AbstractTask.java:181)
at org.apache.kafka.streams.processor.internals.StreamThread$4.apply(
StreamThread.java:360)
at org.apache.kafka.streams.processor.internals.StreamThread.
performOnAllTasks(StreamThread.java:322)
at org.apache.kafka.streams.processor.internals.
StreamThread.flushAllState(StreamThread.java:357)
at org.apache.kafka.streams.processor.internals.StreamThread.
shutdownTasksAndState(StreamThread.java:295)
at org.apache.kafka.streams.processor.internals.StreamThread.shutdown(
StreamThread.java:262)
at org.apache.kafka.streams.processor.internals.
StreamThread.run(StreamThread.java:245)
Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error
opening store streams-data-record-stats-avro-br-store-20150516 at
location /tmp/kafka-streams/argyle-streams/3_8/streams-data-
record-stats-avro-br-store/streams-data-record-stats-
avro-br-store-20150516
at org.apache.kafka.streams.state.internals.RocksDBStore.
openDB(RocksDBStore.java:196)
at org.apache.kafka.streams.state.internals.RocksDBStore.
openDB(RocksDBStore.java:158)
at org.apache.kafka.streams.state.internals.RocksDBWindowStore$Segment.
openDB(RocksDBWindowStore.java:72)
at org.apache.kafka.streams.state.internals.RocksDBWindowStore.
getOrCreateSegment(RocksDBWindowStore.java:402)
at org.apache.kafka.streams.state.internals.RocksDBWindowStore.
putA

Re: kafka streaming rocks db lock bug?

[Guozhang Wang]

Hello Ara,

Your encountered issue seems to be KAFKA-3812
, and KAFKA-3938
. Could you try to
upgrade to the newly released 0.10.1.0 version and see if this issue goes
away? If not I would love to investigate this issue further with you.


Guozhang



Guozhang


On Sun, Oct 23, 2016 at 1:45 PM, Ara Ebrahimi 
wrote:

> And then this on a different node:
>
> 2016-10-23 13:43:57 INFO  StreamThread:286 - stream-thread
> [StreamThread-3] Stream thread shutdown complete
> 2016-10-23 13:43:57 ERROR StreamPipeline:169 - An exception has occurred
> org.apache.kafka.streams.errors.StreamsException: stream-thread
> [StreamThread-3] Failed to rebalance
> at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(
> StreamThread.java:401)
> at org.apache.kafka.streams.processor.internals.
> StreamThread.run(StreamThread.java:235)
> Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error
> while creating the state manager
> at org.apache.kafka.streams.processor.internals.AbstractTask.(
> AbstractTask.java:72)
> at org.apache.kafka.streams.processor.internals.
> StreamTask.(StreamTask.java:90)
> at org.apache.kafka.streams.processor.internals.
> StreamThread.createStreamTask(StreamThread.java:622)
> at org.apache.kafka.streams.processor.internals.
> StreamThread.addStreamTasks(StreamThread.java:649)
> at org.apache.kafka.streams.processor.internals.StreamThread.access$000(
> StreamThread.java:69)
> at org.apache.kafka.streams.processor.internals.StreamThread$1.
> onPartitionsAssigned(StreamThread.java:120)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.
> onJoinComplete(ConsumerCoordinator.java:228)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> joinGroupIfNeeded(AbstractCoordinator.java:313)
> at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.
> ensureActiveGroup(AbstractCoordinator.java:277)
> at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(
> ConsumerCoordinator.java:259)
> at org.apache.kafka.clients.consumer.KafkaConsumer.
> pollOnce(KafkaConsumer.java:1013)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(
> KafkaConsumer.java:979)
> at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(
> StreamThread.java:398)
> ... 1 more
> Caused by: java.io.IOException: task [7_1] Failed to lock the state
> directory: /tmp/kafka-streams/argyle-streams/7_1
> at org.apache.kafka.streams.processor.internals.
> ProcessorStateManager.(ProcessorStateManager.java:98)
> at org.apache.kafka.streams.processor.internals.AbstractTask.(
> AbstractTask.java:69)
> ... 13 more
>
> Ara.
>
> On Oct 23, 2016, at 1:24 PM, Ara Ebrahimi  mailto:ara.ebrah...@argyledata.com>> wrote:
>
> Hi,
>
> This happens when I hammer our 5 kafka streaming nodes (each with 4
> streaming threads) hard enough for an hour or so:
>
> 2016-10-23 13:04:17 ERROR StreamThread:324 - stream-thread
> [StreamThread-2] Failed to flush state for StreamTask 3_8:
> org.apache.kafka.streams.errors.ProcessorStateException: task [3_8]
> Failed to flush state store streams-data-record-stats-avro-br-store
> at org.apache.kafka.streams.processor.internals.
> ProcessorStateManager.flush(ProcessorStateManager.java:322)
> at org.apache.kafka.streams.processor.internals.AbstractTask.flushState(
> AbstractTask.java:181)
> at org.apache.kafka.streams.processor.internals.StreamThread$4.apply(
> StreamThread.java:360)
> at org.apache.kafka.streams.processor.internals.StreamThread.
> performOnAllTasks(StreamThread.java:322)
> at org.apache.kafka.streams.processor.internals.
> StreamThread.flushAllState(StreamThread.java:357)
> at org.apache.kafka.streams.processor.internals.StreamThread.
> shutdownTasksAndState(StreamThread.java:295)
> at org.apache.kafka.streams.processor.internals.StreamThread.shutdown(
> StreamThread.java:262)
> at org.apache.kafka.streams.processor.internals.
> StreamThread.run(StreamThread.java:245)
> Caused by: org.apache.kafka.streams.errors.ProcessorStateException: Error
> opening store streams-data-record-stats-avro-br-store-20150516 at
> location /tmp/kafka-streams/argyle-streams/3_8/streams-data-
> record-stats-avro-br-store/streams-data-record-stats-
> avro-br-store-20150516
> at org.apache.kafka.streams.state.internals.RocksDBStore.
> openDB(RocksDBStore.java:196)
> at org.apache.kafka.streams.state.internals.RocksDBStore.
> openDB(RocksDBStore.java:158)
> at org.apache.kafka.streams.state.internals.RocksDBWindowStore$Segment.
> openDB(RocksDBWindowStore.java:72)
> at org.apache.kafka.streams.state.internals.RocksDBWindowStore.
> getOrCreateSegment(RocksDBWindowStore.java:402)
> at org.apache.kafka.streams.state.internals.RocksDBWindowStore.
> putAndReturnInternalKey(RocksDBWindowStore.java:310)
> at org.apache.kafka.streams.state.internals.RocksDBWindowStore.put(
> RocksDBWindowStore.java:292)
> at org.apache.kafka.strea

Re: customised event time

[Guozhang Wang]

Hi Hamza,

You can create a windowed store in the processor API via the Stores factory
class: org.apache.kafka.streams.state.Stores

More specifically, you you do sth. like:

Stores.create().withKeys().withValues().persistent().windowed(/* you can
specify window size, retention period etc here */)


Which returns the RocksDBWindowStoreSupplier.

Guozhang


On Mon, Oct 24, 2016 at 2:23 AM, Hamza HACHANI 
wrote:

> And the start time and end time of the window.
>
> In other words i need the notion of windows in the proecessor API.
>
> Is this possible ?
>
> 
> De : Hamza HACHANI 
> Envoyé : dimanche 23 octobre 2016 20:43:05
> À : users@kafka.apache.org
> Objet : RE: customised event time
>
> To be more specific.
>
> What id do really need is the property of the retention time dor the
> window in the processor API.
>
> Because for the window  i think that i can manage to do this.
>
>
> Hamza
>
> 
> De : Hamza HACHANI 
> Envoyé : dimanche 23 octobre 2016 20:30:13
> À : users@kafka.apache.org
> Objet : RE: customised event time
>
> Hi,
>
> I think that maybe i'm asking much.
>
> But Ineed the aspect of windowing in the processor API not in the Stram
> DSL. Is this possible?
>
> The second question is how can i get rid of the intermediate results
> because i'm only interested in the final result given by the window.
>
> Hamza
>
> 
> De : Matthias J. Sax 
> Envoyé : samedi 22 octobre 2016 16:12:45
> À : users@kafka.apache.org
> Objet : Re: customised event time
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Hi,
>
> you can set window retention time via method Windows#until() (and this
> retention time is based on the timestamps returned from you custom
> timestamp extractor). This keeps all windows until the retention time
> passes and thus, all later arrival records will be processed correctly.
>
> However, Kafka Streams does not close windows as other framework, but
> rather gives you an (intermediate) result each time a window is
> updated with a new record (regardless if the record is in-order or
> late -- you will get a result record in both cases).
>
> As of Kafka 0.10.1 those (intermediate) results get deduplicated so
> you might not receive all (intermediate) results downstream. Of
> course, it is ensured, that you will eventually get the latest/final
> result sent downstream.
>
>
> - -Matthias
>
> On 10/21/16 7:42 AM, Hamza HACHANI wrote:
> > Hi,
> >
> >
> > I would like to process data based on a customised event time.(a
> > timestamp that I implement as part of the message).
> >
> > The data is processed in periodic windows of x time that are
> > parametered via the method punctuate.
> >
> > What I need is a retention time for the window to be able to treat
> > the late arriving messages.
> >
> > Can I do this : define/configure a retention time for windows . For
> > example the window which treat data between 15pm and 16pm forward
> > the result not in 16pm but in 16:15 pm.
> >
> > Thanks in advance for your help.
> >
> >
> > Hamza
> >
> >
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - https://gpgtools.org
>
> iQIcBAEBCgAGBQJYDDi9AAoJECnhiMLycopPhR8P+wVnRiRwuM7omj3l4EfBVP9Q
> 7Vflwzf9TFmCjmtpSkezQVNSOhU7maUEB7R/4rjqacLuALJBPrOZ+mrRfYIx/oJd
> E7oc3JXO1dlANMyKQ8SQ9dkXWxkeyNtl9ujkcFhMBejqtamteVaI/iNdUdbs6CCk
> AZG86yQKhhPL2Q+BZK0eu88bc0yB5Kwj86uCFFkzMaMXVkvCY1f/kkplRGADsSfd
> dfv5LONFzXmJ5dftC1giMis4Pg/kJU9gwvmh+WJB2tSM8KjXf1hZWe4nv1f+af8g
> pVLnyepv7i7I6PwX/wSdtY2uSLKjizLSbvR3JsfAqLWSVBMgnPpHjCvGecR8K3IC
> FwsnWUk9QJKn4EoahsfRq+afUYva8HRxNH7Aiq+gXvEx9LPgwCqbOGYy4ZnOaoi5
> PycddoNaK1QEn3WYSxN88kLWFx5+vem4LWiBSDBT+Px0p14ZgKwEjqNgyj4fxBtw
> SRsrxVvXXcj2qWvjtRCmTYpn7hanAbYT0QoNfIPs20/Jrh3qsaVJIt5xzRuYyxC8
> aobYLlRFLL82eYaPcdQNz/1T0Q7kT9NbEabiuADaZIz7X1yQluMWuTaGIDk/6j6v
> A2OqaJd8+KkMMOy9PUX1rBnJhJPZ4CTvkzmjnY10HU7sneNpBvU0tfnPdBBCU15B
> 8bsUiTf0lk6t9amGYT6q
> =PcW7
> -END PGP SIGNATURE-
>



-- 
-- Guozhang

RE: A question about kafka

[ZHU Hua B]

Hi,


Anybody could help to answer below question? If compression type could be 
modified through command " bin/kafka-console-producer.sh --producer.config 
"? Thanks!






Best Regards

Johnny


-Original Message-
From: ZHU Hua B 
Sent: 2016年10月17日 14:52
To: users@kafka.apache.org; Radoslaw Gruchalski
Subject: RE: A question about kafka

Hi,


Thanks for your reply!

OK, I got it. And, there is a parameter named compression.type in 
config/producer.properties, which is same usage as "--compression-codec " I 
think. I modify compression.type in config/producer.properties firstly, then 
run console producer with option "--producer.config " and send 
message, but the compression codec could not change as modification. Do you 
know the reason about it? Thanks!


# bin/kafka-console-producer.sh
Read data from standard input and publish it to Kafka.
Option   Description
--   ---
--producer.config   Producer config properties file. Note
   that [producer-property] takes
   precedence over this config.
# bin/kafka-console-producer.sh --producer.config config/producer.properties 
--broker-list localhost:9092 --topic test



Best Regards

Johnny


-Original Message-
From: Hans Jespersen [mailto:h...@confluent.io]
Sent: 2016年10月17日 14:29
To: users@kafka.apache.org; Radoslaw Gruchalski
Subject: RE: A question about kafka

Because the producer-property option is used to set other properties that are 
not compression type.
//h...@confluent.io
 Original message From: ZHU Hua B 
 Date: 10/16/16  11:20 PM  (GMT-08:00) To: 
Radoslaw Gruchalski , users@kafka.apache.org Subject: RE: 
A question about kafka Hi,


Thanks for your reply!

If console producer only allows for compression codec argument, why we could 
found option —producer-property defined in ConsoleProducer.scala? And we could 
find the usage also if we running console producer? The version we used is 
Kafka 0.10.0.0. Thanks!


# ./kafka-console-producer.sh
Read data from standard input and publish it to Kafka.
Option   Description
--   --- --compression-codec 
[compression-codec]  The compression codec: either 'none',
   'gzip', 'snappy', or 'lz4'.If
   specified without value, then it
   defaults to 'gzip'
--producer-property   A mechanism to pass user-defined
   properties in the form key=value to
   the producer.
--producer.config   Producer config properties file. Note
   that [producer-property] takes
   precedence over this config.
--property     A mechanism to pass user-defined
   properties in the form key=value to
   the message reader. This allows
   custom configuration for a user-
   defined message reader.



Best Regards

Johnny

From: Radoslaw Gruchalski [mailto:ra...@gruchalski.com]
Sent: 2016年10月17日 14:02
To: ZHU Hua B; users@kafka.apache.org
Subject: RE: A question about kafka

Hi,

I believe the answer is in the code. This is where the --compression-codec is 
processed:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L143
and this is —producer-property:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L234

The usage is here:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L114

The answer is: The console producer allows for compression codec only with 
—compression-codec argument.

–
Best regards,

Radek Gruchalski

ra...@gruchalski.com


On October 17, 2016 at 7:46:41 AM, ZHU Hua B 
(hua.b@alcatel-lucent.com) wrote:
Hi,


Anybody could help to answer this question? Thanks!






Best Regards

Johnny

-Original Message-
From: ZHU Hua B
Sent: 2016年10月14日 16:41
To: users@kafka.apache.org
Subject: [COMMERCIAL] A question about kafka

Hi,


I have a question about kafka, could you please help to have a look?

I want to send a message from producer with snappy compression codec. So I run 
the command "bin/kafka-console-producer.sh --compression-codec snappy 
--broker-list localhost:9092 --topic test", after that I checked the data log, 
compresscodec is SnappyCompressionCodec as expectation.

Then I tried another command "bin/kafka-console-producer.sh --producer-pro

RE: Mirror multi-embedded consumer's configuration

[ZHU Hua B]

Hi,


It is OK now, thanks for your help!






Best Regards

Johnny


-Original Message-
From: tao xiao [mailto:xiaotao...@gmail.com] 
Sent: 2016年10月24日 17:43
To: users@kafka.apache.org
Subject: Re: Mirror multi-embedded consumer's configuration

auto.offset.reset=smallest takes effect only when there is no offset committed 
prior or offset being requested is out of range.  I believe your mirror maker 
has committed offset therefore auto.offset.reset=smallest wont take effect. You 
can create a new consumer group and try again

On Mon, 24 Oct 2016 at 17:25 ZHU Hua B  wrote:

Hi,


Thanks for your info!

I set auto.offset.reset=smallest in consumer.properties on target cluster and 
retest, it still failed. Seems the topic with original messages could not 
mirror to target cluster while running mirror maker first time, unless produce 
a new message to this topic to trigger mirror. Thanks!






Best Regards

Johnny


-Original Message-
From: tao xiao [mailto:xiaotao...@gmail.com]
Sent: 2016年10月24日 17:10
To: users@kafka.apache.org
Subject: Re: Mirror multi-embedded consumer's configuration

You need to set auto.offset.reset=smallest to mirror data from beginning

On Mon, 24 Oct 2016 at 17:07 ZHU Hua B  wrote:

> Hi,
>
>
> Thanks for your info!
>
>
> Before I launch mirror maker first time, there is a topic include some 
> messages, which have been produced and consumed on source Kafka cluster.
> Then I launch mirror maker first time on target cluster, but this 
> topic did not mirror from source cluster.
>
> So I keep mirror maker running on target cluster, and produced a new 
> message to this topic on source cluster, this time the topic mirror 
> from source cluster to target, which include the new message only, if 
> this behavior is correct? Thanks!
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: 2016年10月24日 16:56
> To: users@kafka.apache.org
> Subject: Re: Mirror multi-embedded consumer's configuration
>
> Yes, it should mirror again. To mirror the topic from beginning, you 
> may have to start with different consumer group,
>
> On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B 
> 
> wrote:
>
> > Hi,
> >
> >
> > Many thanks for your confirm!
> >
> > I have another question, if I deleted a mirrored topic on the target 
> > Kafka cluster, if Kafka mirror maker could mirror the same topic 
> > again from source cluster when I launch mirror maker next time? Thanks!
> >
> >
> >
> >
> >
> >
> > Best Regards
> >
> > Johnny
> >
> >
> > -Original Message-
> > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > Sent: 2016年10月24日 15:03
> > To: users@kafka.apache.org
> > Subject: Re: Mirror multi-embedded consumer's configuration
> >
> > I think we need to update the docs. This behavior got changed 
> > longtime
> ago.
> > we need to pass only one consumer configuration. This config will be 
> > applicable for all the consumer threads/instances.
> >
> > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B 
> > 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > Because from some Kafka wiki I saw "At minimum, the mirror maker 
> > > takes one or more consumer configurations, a producer 
> > > configuration and either a whitelist or a blacklist", but the test 
> > > failed, so I want to know if Kafka mirror really support more 
> > > consumer configurations at
> > minimum? Thanks!
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > >
> > > -Original Message-
> > > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > > Sent: 2016年10月24日 13:48
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror multi-embedded consumer's configuration
> > >
> > > why are you passing "consumer.config" twice?
> > >
> > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B 
> > > 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > The version of Kafka I used is 0.10.0.0. Thanks!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Johnny
> > > >
> > > > -Original Message-
> > > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > > Sent: 2016年10月24日 12:22
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Mirror multi-embedded consumer's configuration
> > > >
> > > > Which version are you using for the MM?
> > > >
> > > > Guozhang
> > > >
> > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B 
> > > > 
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > Anybody could help to answer below question? Thanks!
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Best Regards
> > > > >
> > > > > Johnny
> > > > >
> > > > > From: ZHU Hua B
> > > > > Sent: 2016年10月19日 16:22
> > > > > To: 'users@kafka.apache.org'
> > > > > Subject: Mirror multi-embedded consumer's configuration
> > > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > I launch Kafka mirror maker with multi-embedded consumer's 
> > > > > configuration but failed as below, what’s the mean of “you 
> > > 

programmatic way to check for topic existence?

[Ben Osheroff]

Hiya!

I've been trying to merge https://github.com/zendesk/maxwell/pull/457,
which adds a much-requested feature of Maxwell, that of being able to
have a topic-per-mysql-table.  When we receive a row we programmatically
generate the topic name, and the first thing we do is call
`KafkaProducer#partitionsFor(topic)`, so that we know how to partition
the data.

The problem I'm running into is in trying to detect the case where a
topic doesn't exist.  If auto-creation is on, `partitionsFor()` seems to
correctly auto-create the topic, but if auto-creation is off the
behavior is kinda wonky; kafka goes into a metadata-fetch loop, logging

"Error while fetching metadata with correlation id 573 
:{topic=UNKNOWN_TOPIC_OR_PARTITION}"

but then ultimately throwing me back a `TimeoutException` after 60 tries
or so.

I can rescue/rethrow the TimeoutException, but it seems like there might
be a better way that I'm missing.  Any ideas?  I'd ideally just like a
way to fail fast and clean when the topic doesn't exist (and
auto-creation is off).

Thanks,
Ben Osheroff
zendesk.com





signature.asc
Description: PGP signature

Re: Consumer error : This consumer has already been closed

[Koen Vantomme]

Hello,
Could someone reply me if I made a mistake here or not. Did I make a rookie
mistake here ? Is this not the correct forum here to put my question ?
It's a bit strange to have this error  : Exception in thread "main"
java.lang.IllegalStateException: This consumer has already been closed
On the net there is also not much feedback  on this error.
Kind regards,
Koen


On Sun, Oct 23, 2016 at 6:10 PM, Koen Vantomme 
wrote:

> Hello,
>
> I'm creating a simple consumer in JAVA, the first time I run the consumer
> it works fine.
> I stop the application. When I want to rerun the consumer I get error
> message "This consumer has already been closed"
>
> Any suggestions ?
> Regards,
> Koen
>
> 2016-10-23 17:17:34,261 [main] INFO   AppInfoParser - Kafka commitId :
> 23c69d62a0cabf06
> Exception in thread "main" java.lang.IllegalStateException: This consumer
> has already been closed.
> at org.apache.kafka.clients.consumer.KafkaConsumer.
> ensureNotClosed(KafkaConsumer.java:1310)
> at org.apache.kafka.clients.consumer.KafkaConsumer.
> acquire(KafkaConsumer.java:1321)
> at org.apache.kafka.clients.consumer.KafkaConsumer.poll(
> KafkaConsumer.java:844)
>
>
> The code :
>
> String topic ="testmetrics";
> String group ="cg1";
>
> Properties props = new Properties();
> props.put("bootstrap.servers", "localhost:9092");
> props.put("group.id", group);
> props.put("enable.auto.commit", "true");
> props.put("auto.commit.interval.ms", "1000");
> props.put("session.timeout.ms", "3");
> props.put("key.deserializer",
> "org.apache.kafka.common.serialization.StringDeserializer");
> props.put("value.deserializer",
> "org.apache.kafka.common.serialization.StringDeserializer");
>
> KafkaConsumer consumer = new KafkaConsumer String>(props);
>
>
> consumer.subscribe(Arrays.asList(topic));
> System.out.println("Subscribed to topic " + topic);
> int i = 0;
>
> while (true) {
> ConsumerRecords records = consumer.poll(100);
> for (ConsumerRecord record : records)
> {
> System.out.printf("offset = %d, key = %s, value = %s\n",
> record.offset(), record.key(), record.value());
> }
> c
>
>

Re: Kafka Streaming

[Matthias J. Sax]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

It's a client issues... But CP 3.1 should be our in about 2 weeks...
Of course, you can use Kafka 0.10.1.0 for now. It was released last
week and does contain the fix.

- -Matthias

On 10/24/16 9:19 AM, Mohit Anchlia wrote:
> Would this be an issue if I connect to a remote Kafka instance
> running on the Linux box? Or is this a client issue. What's rockdb
> used for to keep state?
> 
> On Mon, Oct 24, 2016 at 12:08 AM, Matthias J. Sax
>  wrote:
> 
> Kafka 0.10.1.0 which was release last week does contain the fix 
> already. The fix will be in CP 3.1 coming up soon!
> 
> (sorry that I did mix up versions in a previous email)
> 
> -Matthias
> 
> On 10/23/16 12:10 PM, Mohit Anchlia wrote:
 So if I get it right I will not have this fix until 4
 months? Should I just create my own example with the next
 version of Kafka?
 
 On Sat, Oct 22, 2016 at 9:04 PM, Matthias J. Sax 
  wrote:
 
 Current version is 3.0.1 CP 3.1 should be release the next
 weeks
 
 So CP 3.2 should be there is about 4 month (Kafka follows a
 time base release cycle of 4 month and CP usually aligns with
 Kafka releases)
 
 -Matthias
 
 
 On 10/20/16 5:10 PM, Mohit Anchlia wrote:
>>> Any idea of when 3.2 is coming?
>>> 
>>> On Thu, Oct 20, 2016 at 4:53 PM, Matthias J. Sax 
>>>  wrote:
>>> 
>>> No problem. Asking questions is the purpose of mailing
>>> lists. :)
>>> 
>>> The issue will be fixed in next version of examples
>>> branch.
>>> 
>>> Examples branch is build with CP dependency and not
>>> with Kafka dependency. CP-3.2 is not available yet;
>>> only Kafka 0.10.1.0. Nevertheless, they should work
>>> with Kafka dependency, too. I never tried it, but you
>>> should give it a shot...
>>> 
>>> But you should use example master branch because of
>>> API changes from 0.10.0.x to 0.10.1 (and thus, changing
>>> CP-3.1 to 0.10.1.0 will not be compatible and not
>>> compile, while changing CP-3.2-SNAPSHOT to 0.10.1.0
>>> should work -- hopefully ;) )
>>> 
>>> 
>>> -Matthias
>>> 
>>> On 10/20/16 4:02 PM, Mohit Anchlia wrote:
>> So this issue I am seeing is fixed in the next
>> version of example branch? Can I change my pom to
>> point it the higher version of Kafka if that is
>> the issue? Or do I need to wait until new branch
>> is made available? Sorry lot of questions :)
>> 
>> On Thu, Oct 20, 2016 at 3:56 PM, Matthias J. Sax 
>>  wrote:
>> 
>> The branch is 0.10.0.1 and not 0.10.1.0 (sorry
>> for so many zeros and ones -- super easy to mix
>> up)
>> 
>> However, examples master branch uses
>> CP-3.1-SNAPSHOT (ie, Kafka 0.10.1.0) -- there
>> will be a 0.10.1 examples branch, after CP-3.1
>> was released
>> 
>> 
>> -Matthias
>> 
>> On 10/20/16 3:48 PM, Mohit Anchlia wrote:
> I just now cloned this repo. It seems to be
> using 10.1
> 
> https://github.com/confluentinc/examples
> and running examples in 
> https://github.com/confluentinc/examples/tree/kafka-0.10.0
.1-
>
> 
cp-
 
> 
> 3.0
>>> 
> 
 .1/
>> 
> 
>>> kafka-streams
> 
> On Thu, Oct 20, 2016 at 3:10 PM, Michael
> Noll  wrote:
> 
>> I suspect you are running Kafka 0.10.0.x
>> on Windows? If so, this is a known issue
>> that is fixed in Kafka 0.10.1 that was
>> just released today.
>> 
>> Also: which examples are you referring
>> to? And, to confirm: which git branch /
>> Kafka version / OS in case my guess above
>> was wrong.
>> 
>> 
>> On Thursday, October 20, 2016, Mohit
>> Anchlia  wrote:
>> 
>>> I am trying to run the examples from
>>> git. While running the wordcount
>>> example I see this error:
>>> 
>>> Caused by:
>>> *java.lang.RuntimeException*: 
>>> librocksdbjni-win64.dll was not found
>>> inside JAR.
>>> 
>>> 
>>> Am I expected to include this jar
>>> locally?
>>> 
>> 
>> 
>> -- *Michael G. Noll* Product Manager | 
>> Confluent +1 650 453 5860 | @miguno 
>>  Follow us:
>> Twitter 
>>  |
>> Blog 
>> 
> 
>>> 
>> 
 
>>> 
> 
 
>> 
> 

Re: Kafka Streaming

[Mohit Anchlia]

Would this be an issue if I connect to a remote Kafka instance running on
the Linux box? Or is this a client issue. What's rockdb used for to keep
state?

On Mon, Oct 24, 2016 at 12:08 AM, Matthias J. Sax 
wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> Kafka 0.10.1.0 which was release last week does contain the fix
> already. The fix will be in CP 3.1 coming up soon!
>
> (sorry that I did mix up versions in a previous email)
>
> - -Matthias
>
> On 10/23/16 12:10 PM, Mohit Anchlia wrote:
> > So if I get it right I will not have this fix until 4 months?
> > Should I just create my own example with the next version of
> > Kafka?
> >
> > On Sat, Oct 22, 2016 at 9:04 PM, Matthias J. Sax
> >  wrote:
> >
> > Current version is 3.0.1 CP 3.1 should be release the next weeks
> >
> > So CP 3.2 should be there is about 4 month (Kafka follows a time
> > base release cycle of 4 month and CP usually aligns with Kafka
> > releases)
> >
> > -Matthias
> >
> >
> > On 10/20/16 5:10 PM, Mohit Anchlia wrote:
>  Any idea of when 3.2 is coming?
> 
>  On Thu, Oct 20, 2016 at 4:53 PM, Matthias J. Sax
>   wrote:
> 
>  No problem. Asking questions is the purpose of mailing lists.
>  :)
> 
>  The issue will be fixed in next version of examples branch.
> 
>  Examples branch is build with CP dependency and not with
>  Kafka dependency. CP-3.2 is not available yet; only Kafka
>  0.10.1.0. Nevertheless, they should work with Kafka
>  dependency, too. I never tried it, but you should give it a
>  shot...
> 
>  But you should use example master branch because of API
>  changes from 0.10.0.x to 0.10.1 (and thus, changing CP-3.1 to
>  0.10.1.0 will not be compatible and not compile, while
>  changing CP-3.2-SNAPSHOT to 0.10.1.0 should work -- hopefully
>  ;) )
> 
> 
>  -Matthias
> 
>  On 10/20/16 4:02 PM, Mohit Anchlia wrote:
> >>> So this issue I am seeing is fixed in the next version
> >>> of example branch? Can I change my pom to point it the
> >>> higher version of Kafka if that is the issue? Or do I
> >>> need to wait until new branch is made available? Sorry
> >>> lot of questions :)
> >>>
> >>> On Thu, Oct 20, 2016 at 3:56 PM, Matthias J. Sax
> >>>  wrote:
> >>>
> >>> The branch is 0.10.0.1 and not 0.10.1.0 (sorry for so
> >>> many zeros and ones -- super easy to mix up)
> >>>
> >>> However, examples master branch uses CP-3.1-SNAPSHOT
> >>> (ie, Kafka 0.10.1.0) -- there will be a 0.10.1 examples
> >>> branch, after CP-3.1 was released
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 10/20/16 3:48 PM, Mohit Anchlia wrote:
> >> I just now cloned this repo. It seems to be using
> >> 10.1
> >>
> >> https://github.com/confluentinc/examples and
> >> running examples in
> >> https://github.com/confluentinc/examples/tree/kafka-0.10.0.1-
> cp-
> >
> >>
> 3.0
> 
> >>
> > .1/
> >>>
> >>
>  kafka-streams
> >>
> >> On Thu, Oct 20, 2016 at 3:10 PM, Michael Noll
> >>  wrote:
> >>
> >>> I suspect you are running Kafka 0.10.0.x on
> >>> Windows? If so, this is a known issue that is
> >>> fixed in Kafka 0.10.1 that was just released
> >>> today.
> >>>
> >>> Also: which examples are you referring to?
> >>> And, to confirm: which git branch / Kafka
> >>> version / OS in case my guess above was wrong.
> >>>
> >>>
> >>> On Thursday, October 20, 2016, Mohit Anchlia
> >>>  wrote:
> >>>
>  I am trying to run the examples from git.
>  While running the wordcount example I see
>  this error:
> 
>  Caused by: *java.lang.RuntimeException*:
>  librocksdbjni-win64.dll was not found inside
>  JAR.
> 
> 
>  Am I expected to include this jar locally?
> 
> >>>
> >>>
> >>> -- *Michael G. Noll* Product Manager |
> >>> Confluent +1 650 453 5860 | @miguno
> >>>  Follow us: Twitter
> >>>  | Blog
> >>> 
> >>>
> >>
> 
> >>>
> >
> 
> >>
> >
> -BEGIN PGP SIGNATURE-
> Comment: GPGTools - https://gpgtools.org
>
> iQIcBAEBCgAGBQJYDbOJAAoJECnhiMLycopPSF0QAKFi0lG3dMccIaBuSzsgj2Of
> Iu9BnoTCr4MJoY18loPE8DUy9kbEjaf6RjMzkvXvTV+Jjys7PgTa5qLlo7hEwgb6
> UxWZYrdOXUGQcJckmsdzMIoGY79GETQuvRkN8uzJ/hfJJ4YNgtLCXaNMDTfcRmtp
> hson3M7SYRISSz7Jz9jM7PmPnR6MIWXRV+8r9jdL8IH+gRtrnT1svr/pkiAL6e7B
> gZdQHhTVI+dqWRgHfuenr2MQH2Od3UgZsK3pqVlfv0Zy7c3IL6FKN4WQgDiYOYBH
> URPY9BBQGnOTINEmX63Bwx8ns+RNgtyIO9O3ytewRmyWRkQIzO/K0zszSjblPJeG
> rNnrC9wAQH9arx6ZHAw/moEfpsRCAJT5SGjCnYa8NaDq0wlbH

Re: Kafka unable to find topic details after sudden EC2 server restart

[Hans Jespersen]

You probably didn't setup permissions properly on the /data directory you
created.
You might want to do some free online training to learn some more Linux if
you are planning to become a Kafka administrator.
https://en.wikipedia.org/wiki/Chmod
https://www.edx.org/course/introduction-linux-linuxfoundationx-lfs101x-0

-hans

/**
 * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
 * h...@confluent.io (650)924-2670
 */

On Mon, Oct 24, 2016 at 1:50 AM, Gourab Chowdhury 
wrote:

> Thanks for the reply, I tried changing the data directory as follows:-
> dataDir=/data/zookeeper
>
> I also create a /data but still I am getting the following error.
>
> ERROR Unexpected exception, exiting abnormally
> (org.apache.zookeeper.server.
> ZooKeeperServerMain)
> java.io.IOException: Unable to create data directory
> /data/zookeeper/version-2
>
> Gourab
>
> On Sat, Oct 22, 2016 at 3:44 AM, Hans Jespersen  wrote:
>
> > You are going to lose everything you store in /tmp. In a production
> system
> > you never configure Kafka or zookeeper to store critical data in /tmp.
> > This has nothing to do with AWS or EBS it is just standard Linux than
> > everything under /tmp is deleted when Linux reboots.
> >
> > -hans
> >
> > /**
> >  * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
> >  * h...@confluent.io (650)924-2670
> >  */
> >
> > On Fri, Oct 21, 2016 at 10:38 AM, Gourab Chowdhury  >
> > wrote:
> >
> > > I was testing a scenario of a server crash while the Kafka is still
> > running
> > > and passing messages. (just did a sudo reboot). By default, I am
> storing
> > > the logs at /tmp/kafka-log.
> > >
> > > So in case an EC2 server restart, am I going to loose the data if I
> don't
> > > store kafka logs in an EBS drive?
> > >
> > > Also, I have another doubt, Does zookeeper stores Kafka info (viz.
> > topics,
> > > brokers list, consumer groups) at /tmp/zookeeper/version-2/log.x ?
> > >
> > > I am using python zookeeper client: *kazoo* and here we can find
> various
> > > Kafka info like '/brokers/topics', '/brokers/topics/topic/partitions'
> ,
> > > '/consumers' , '/consumers/consumer/offsets'. Where can I get this info
> > > directly?
> > >
> > > ~Gourab
> > >
> > > On Fri, Oct 21, 2016 at 8:13 PM, Ben Davison  >
> > > wrote:
> > >
> > > > Are you restarting or terminating the instance? If your terminating,
> > and
> > > > don't have an EBS drive attached you will lose all data on the drive.
> > > >
> > > > On Fri, Oct 21, 2016 at 2:09 PM, Gourab Chowdhury <
> > gourab@gmail.com>
> > > > wrote:
> > > >
> > > > > I am running kafka in a EC2 server.  While running if there is a
> > sudden
> > > > > restart of server, and then we restart the kafka broker, we can't
> > find
> > > > the
> > > > > details of all the topics.
> > > > >
> > > > > bin/kafka-topics.sh --describe --zookeeper 
> > > > > gives no result
> > > > >
> > > > > bin/kafka-console-producer.sh --broker-list  --topic
> > > test
> > > > > yields this particular error as follows:
> > > > >
> > > > > WARN Error while fetching metadata with correlation id n :
> > > > > {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.
> > > > NetworkClient)
> > > > >
> > > > > Is the details of topics from the zookeeper getting deleted while
> > > > restart?
> > > > > or this is something else?
> > > > >
> > > > > Also how can I cope with this in in real life production?
> > > > >
> > > > > Regards,
> > > > > Gourab Chowdhury,
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > > > This email, including attachments, is private and confidential. If
> you
> > > have
> > > > received this email in error please notify the sender and delete it
> > from
> > > > your system. Emails are not secure and may contain viruses. No
> > liability
> > > > can be accepted for viruses that might be transferred by this email
> or
> > > any
> > > > attachment. Any unauthorised copying of this message or unauthorised
> > > > distribution and publication of the information contained herein are
> > > > prohibited.
> > > >
> > > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A
> 2BB.
> > > > Registered in England and Wales. Registered No. 04843573.
> > > >
> > >
> >
>

Re: Removing kafka topic contents from Java

[Hans Jespersen]

The Kafka Java Admin client is actively being worked on now as part of
KIP-4

see https://issues.apache.org/jira/browse/KAFKA-1694

specifically sub-task #9 Create Java Admin Client

The Server side and wire protocol for admin commands like
create/delete/modify topic are already in Kafka 0.10.1

-hans



/**
 * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
 * h...@confluent.io (650)924-2670
 */

On Mon, Oct 24, 2016 at 7:01 AM, Demian Calcaprina 
wrote:

> Hi Guys,
>
> Is there a way to remove a kafka topic from the java api?
>
> I have the following scenario:
> We have a subscription service, which returns responses to subscribed
> clients through kafka topic. Per subscription, we create a new kafka topic
> and send the data by publishing in this new topic. And, when the
> subscription ends, this topic is not needed anymore, so I would like to
> purge it so that it does not store data it will not be used anymore.
> Does it make sense?
>
> Thanks!
>

Re: Removing kafka topic contents from Java

[sunil kalva]

Are you looking something like this

String command = String.format("--delete --zookeeper %s --topic %s",
"localhost:2181", "test_topic");
String[] split = command.split(" ");
TopicCommand.main(split);

t
SunilKalva

On Mon, Oct 24, 2016 at 9:09 PM, Ali Akhtar  wrote:

> There isn't a java API for this, you'd have to mess around with bash
> scripts which I haven't found to be worth it.
>
> Just let the data expire and get deleted. Set a short expiry time for the
> topic if necessary.
>
>
> On Mon, Oct 24, 2016 at 6:30 PM, Demian Calcaprina 
> wrote:
>
> > Hi Guys,
> >
> > Is there a way to remove a kafka topic from the java api?
> >
> > I have the following scenario:
> > We have a subscription service, which returns responses to subscribed
> > clients through kafka topic. Per subscription, we create a new kafka
> topic
> > and send the data by publishing in this new topic. And, when the
> > subscription ends, this topic is not needed anymore, so I would like to
> > purge it so that it does not store data it will not be used anymore.
> > Does it make sense?
> >
> > Thanks!
> >
> > Demian
> >
>



-- 
SunilKalva

Re: Removing kafka topic contents from Java

[Ali Akhtar]

There isn't a java API for this, you'd have to mess around with bash
scripts which I haven't found to be worth it.

Just let the data expire and get deleted. Set a short expiry time for the
topic if necessary.


On Mon, Oct 24, 2016 at 6:30 PM, Demian Calcaprina 
wrote:

> Hi Guys,
>
> Is there a way to remove a kafka topic from the java api?
>
> I have the following scenario:
> We have a subscription service, which returns responses to subscribed
> clients through kafka topic. Per subscription, we create a new kafka topic
> and send the data by publishing in this new topic. And, when the
> subscription ends, this topic is not needed anymore, so I would like to
> purge it so that it does not store data it will not be used anymore.
> Does it make sense?
>
> Thanks!
>
> Demian
>

RE: Removing kafka topic contents from Java

[Tauzell, Dave]

There is a script kafka-topics.sh which calls kafka.admin.TopicCommand.  I 
think you could probably call that same class from your application if you 
figure out which jars are needed.   When I do testing I'll clear out queues by 
setting the retention time to a second and then wait for kafka to remove all 
the messages.  If you turn on the ability to delete topics, you can also then 
delete them.

-Dave

-Original Message-
From: Demian Calcaprina [mailto:calcacue...@gmail.com]
Sent: Monday, October 24, 2016 8:31 AM
To: users@kafka.apache.org
Subject: Removing kafka topic contents from Java

Hi Guys,

Is there a way to remove a kafka topic from the java api?

I have the following scenario:
We have a subscription service, which returns responses to subscribed clients 
through kafka topic. Per subscription, we create a new kafka topic and send the 
data by publishing in this new topic. And, when the subscription ends, this 
topic is not needed anymore, so I would like to purge it so that it does not 
store data it will not be used anymore.
Does it make sense?

Thanks!

Demian
This e-mail and any files transmitted with it are confidential, may contain 
sensitive information, and are intended solely for the use of the individual or 
entity to whom they are addressed. If you have received this e-mail in error, 
please notify the sender by reply e-mail immediately and destroy all copies of 
the e-mail and any attachments.

Removing kafka topic contents from Java

[Demian Calcaprina]

Hi Guys,

Is there a way to remove a kafka topic from the java api?

I have the following scenario:
We have a subscription service, which returns responses to subscribed
clients through kafka topic. Per subscription, we create a new kafka topic
and send the data by publishing in this new topic. And, when the
subscription ends, this topic is not needed anymore, so I would like to
purge it so that it does not store data it will not be used anymore.
Does it make sense?

Thanks!

Demian

RE: Mirror multi-embedded consumer's configuration

[Hans Jespersen]

Yes.
//h...@confluent.io
 Original message From: ZHU Hua B 
 Date: 10/24/16  12:09 AM  (GMT-08:00) To: 
users@kafka.apache.org Subject: RE: Mirror multi-embedded consumer's 
configuration 
Hi,


Many thanks for your confirm!

I have another question, if I deleted a mirrored topic on the target Kafka 
cluster, if Kafka mirror maker could mirror the same topic again from source 
cluster when I launch mirror maker next time? Thanks!






Best Regards

Johnny


-Original Message-
From: Manikumar [mailto:manikumar.re...@gmail.com] 
Sent: 2016年10月24日 15:03
To: users@kafka.apache.org
Subject: Re: Mirror multi-embedded consumer's configuration

I think we need to update the docs. This behavior got changed longtime ago.
we need to pass only one consumer configuration. This config will be applicable 
for all the consumer threads/instances.

On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B 
wrote:

> Hi,
>
>
> Because from some Kafka wiki I saw "At minimum, the mirror maker takes 
> one or more consumer configurations, a producer configuration and 
> either a whitelist or a blacklist", but the test failed, so I want to 
> know if Kafka mirror really support more consumer configurations at minimum? 
> Thanks!
>
>
>
>
> Best Regards
>
> Johnny
>
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: 2016年10月24日 13:48
> To: users@kafka.apache.org
> Subject: Re: Mirror multi-embedded consumer's configuration
>
> why are you passing "consumer.config" twice?
>
> On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B 
> 
> wrote:
>
> > Hi,
> >
> >
> > The version of Kafka I used is 0.10.0.0. Thanks!
> >
> >
> >
> >
> >
> >
> > Best Regards
> >
> > Johnny
> >
> > -Original Message-
> > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > Sent: 2016年10月24日 12:22
> > To: users@kafka.apache.org
> > Subject: Re: Mirror multi-embedded consumer's configuration
> >
> > Which version are you using for the MM?
> >
> > Guozhang
> >
> > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B 
> > 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > Anybody could help to answer below question? Thanks!
> > >
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > > From: ZHU Hua B
> > > Sent: 2016年10月19日 16:22
> > > To: 'users@kafka.apache.org'
> > > Subject: Mirror multi-embedded consumer's configuration
> > >
> > > Hi,
> > >
> > >
> > > I launch Kafka mirror maker with multi-embedded consumer's 
> > > configuration but failed as below, what’s the mean of “you asked 
> > > for only one”, is there an option control it? Thanks!
> > >
> > > # bin/kafka-mirror-maker.sh --consumer.config 
> > > config/consumer-1.properties --consumer.config 
> > > config/consumer-2.properties --num.streams 2 --producer.config
> > config/producer.properties --whitelist '.*'
> > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker.
> > > (kafka.tools.MirrorMaker$)
> > > joptsimple.MultipleArgumentsForOptionException: Found multiple 
> > > arguments for option consumer.config, but you asked for only one
> > > at joptsimple.OptionSet.valueOf(OptionSet.java:179)
> > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235)
> > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > Exception in thread "main" java.lang.NullPointerException
> > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > >
> > >
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Removing kafka topic contents from Java

[Demian Calcaprina]

Hi Guys,

Is there a way to remove a kafka topic from the java api?

I have the following scenario:
We have a subscription service, which returns responses to subscribed
clients through kafka topic. Per subscription, we create a new kafka topic
and send the data by publishing in this new topic. And, when the
subscription ends, this topic is not needed anymore, so I would like to
purge it so that it does not store data it will not be used anymore.
Does it make sense?

Thanks!

Re: Mirror multi-embedded consumer's configuration

[tao xiao]

auto.offset.reset=smallest takes effect only when there is no offset
committed prior or offset being requested is out of range.  I believe your
mirror maker has committed offset therefore auto.offset.reset=smallest wont
take effect. You can create a new consumer group and try again

On Mon, 24 Oct 2016 at 17:25 ZHU Hua B  wrote:

Hi,


Thanks for your info!

I set auto.offset.reset=smallest in consumer.properties on target cluster
and retest, it still failed. Seems the topic with original messages could
not mirror to target cluster while running mirror maker first time, unless
produce a new message to this topic to trigger mirror. Thanks!






Best Regards

Johnny


-Original Message-
From: tao xiao [mailto:xiaotao...@gmail.com]
Sent: 2016年10月24日 17:10
To: users@kafka.apache.org
Subject: Re: Mirror multi-embedded consumer's configuration

You need to set auto.offset.reset=smallest to mirror data from beginning

On Mon, 24 Oct 2016 at 17:07 ZHU Hua B  wrote:

> Hi,
>
>
> Thanks for your info!
>
>
> Before I launch mirror maker first time, there is a topic include some
> messages, which have been produced and consumed on source Kafka cluster.
> Then I launch mirror maker first time on target cluster, but this
> topic did not mirror from source cluster.
>
> So I keep mirror maker running on target cluster, and produced a new
> message to this topic on source cluster, this time the topic mirror
> from source cluster to target, which include the new message only, if
> this behavior is correct? Thanks!
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: 2016年10月24日 16:56
> To: users@kafka.apache.org
> Subject: Re: Mirror multi-embedded consumer's configuration
>
> Yes, it should mirror again. To mirror the topic from beginning, you
> may have to start with different consumer group,
>
> On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B
> 
> wrote:
>
> > Hi,
> >
> >
> > Many thanks for your confirm!
> >
> > I have another question, if I deleted a mirrored topic on the target
> > Kafka cluster, if Kafka mirror maker could mirror the same topic
> > again from source cluster when I launch mirror maker next time? Thanks!
> >
> >
> >
> >
> >
> >
> > Best Regards
> >
> > Johnny
> >
> >
> > -Original Message-
> > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > Sent: 2016年10月24日 15:03
> > To: users@kafka.apache.org
> > Subject: Re: Mirror multi-embedded consumer's configuration
> >
> > I think we need to update the docs. This behavior got changed
> > longtime
> ago.
> > we need to pass only one consumer configuration. This config will be
> > applicable for all the consumer threads/instances.
> >
> > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B
> > 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > Because from some Kafka wiki I saw "At minimum, the mirror maker
> > > takes one or more consumer configurations, a producer
> > > configuration and either a whitelist or a blacklist", but the test
> > > failed, so I want to know if Kafka mirror really support more
> > > consumer configurations at
> > minimum? Thanks!
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > >
> > > -Original Message-
> > > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > > Sent: 2016年10月24日 13:48
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror multi-embedded consumer's configuration
> > >
> > > why are you passing "consumer.config" twice?
> > >
> > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B
> > > 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > The version of Kafka I used is 0.10.0.0. Thanks!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Johnny
> > > >
> > > > -Original Message-
> > > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > > Sent: 2016年10月24日 12:22
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Mirror multi-embedded consumer's configuration
> > > >
> > > > Which version are you using for the MM?
> > > >
> > > > Guozhang
> > > >
> > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B
> > > > 
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > Anybody could help to answer below question? Thanks!
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Best Regards
> > > > >
> > > > > Johnny
> > > > >
> > > > > From: ZHU Hua B
> > > > > Sent: 2016年10月19日 16:22
> > > > > To: 'users@kafka.apache.org'
> > > > > Subject: Mirror multi-embedded consumer's configuration
> > > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > I launch Kafka mirror maker with multi-embedded consumer's
> > > > > configuration but failed as below, what’s the mean of “you
> > > > > asked for only one”, is there an option control it? Thanks!
> > > > >
> > > > > # bin/kafka-mirror-maker.sh --consumer.config
> > > > > config/consumer-1.properties --consumer.config
> > > > > config/consumer-2.properties --num.streams 2 --producer.config
> > > > config/produce

Re: Kafka unable to find topic details after sudden EC2 server restart

[Gourab Chowdhury]

No issues, I have solved the problem by changing the permission of /data

Thanks,
Gourab

On Mon, Oct 24, 2016 at 2:20 PM, Gourab Chowdhury 
wrote:

> Thanks for the reply, I tried changing the data directory as follows:-
> dataDir=/data/zookeeper
>
> I also create a /data but still I am getting the following error.
>
> ERROR Unexpected exception, exiting abnormally
> (org.apache.zookeeper.server.ZooKeeperServerMain)
> java.io.IOException: Unable to create data directory
> /data/zookeeper/version-2
>
> Gourab
>
> On Sat, Oct 22, 2016 at 3:44 AM, Hans Jespersen  wrote:
>
>> You are going to lose everything you store in /tmp. In a production system
>> you never configure Kafka or zookeeper to store critical data in /tmp.
>> This has nothing to do with AWS or EBS it is just standard Linux than
>> everything under /tmp is deleted when Linux reboots.
>>
>> -hans
>>
>> /**
>>  * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
>>  * h...@confluent.io (650)924-2670
>>  */
>>
>> On Fri, Oct 21, 2016 at 10:38 AM, Gourab Chowdhury 
>> wrote:
>>
>> > I was testing a scenario of a server crash while the Kafka is still
>> running
>> > and passing messages. (just did a sudo reboot). By default, I am storing
>> > the logs at /tmp/kafka-log.
>> >
>> > So in case an EC2 server restart, am I going to loose the data if I
>> don't
>> > store kafka logs in an EBS drive?
>> >
>> > Also, I have another doubt, Does zookeeper stores Kafka info (viz.
>> topics,
>> > brokers list, consumer groups) at /tmp/zookeeper/version-2/log.x ?
>> >
>> > I am using python zookeeper client: *kazoo* and here we can find various
>> > Kafka info like '/brokers/topics', '/brokers/topics/topic/partitions' ,
>> > '/consumers' , '/consumers/consumer/offsets'. Where can I get this info
>> > directly?
>> >
>> > ~Gourab
>> >
>> > On Fri, Oct 21, 2016 at 8:13 PM, Ben Davison 
>> > wrote:
>> >
>> > > Are you restarting or terminating the instance? If your terminating,
>> and
>> > > don't have an EBS drive attached you will lose all data on the drive.
>> > >
>> > > On Fri, Oct 21, 2016 at 2:09 PM, Gourab Chowdhury <
>> gourab@gmail.com>
>> > > wrote:
>> > >
>> > > > I am running kafka in a EC2 server.  While running if there is a
>> sudden
>> > > > restart of server, and then we restart the kafka broker, we can't
>> find
>> > > the
>> > > > details of all the topics.
>> > > >
>> > > > bin/kafka-topics.sh --describe --zookeeper 
>> > > > gives no result
>> > > >
>> > > > bin/kafka-console-producer.sh --broker-list  --topic
>> > test
>> > > > yields this particular error as follows:
>> > > >
>> > > > WARN Error while fetching metadata with correlation id n :
>> > > > {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.
>> > > NetworkClient)
>> > > >
>> > > > Is the details of topics from the zookeeper getting deleted while
>> > > restart?
>> > > > or this is something else?
>> > > >
>> > > > Also how can I cope with this in in real life production?
>> > > >
>> > > > Regards,
>> > > > Gourab Chowdhury,
>> > > >
>> > >
>> > > --
>> > >
>> > >
>> > > This email, including attachments, is private and confidential. If you
>> > have
>> > > received this email in error please notify the sender and delete it
>> from
>> > > your system. Emails are not secure and may contain viruses. No
>> liability
>> > > can be accepted for viruses that might be transferred by this email or
>> > any
>> > > attachment. Any unauthorised copying of this message or unauthorised
>> > > distribution and publication of the information contained herein are
>> > > prohibited.
>> > >
>> > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A
>> 2BB.
>> > > Registered in England and Wales. Registered No. 04843573.
>> > >
>> >
>>
>
>

RE: Mirror multi-embedded consumer's configuration

[ZHU Hua B]

Hi,


Thanks for your info!

I set auto.offset.reset=smallest in consumer.properties on target cluster and 
retest, it still failed. Seems the topic with original messages could not 
mirror to target cluster while running mirror maker first time, unless produce 
a new message to this topic to trigger mirror. Thanks!






Best Regards

Johnny


-Original Message-
From: tao xiao [mailto:xiaotao...@gmail.com] 
Sent: 2016年10月24日 17:10
To: users@kafka.apache.org
Subject: Re: Mirror multi-embedded consumer's configuration

You need to set auto.offset.reset=smallest to mirror data from beginning

On Mon, 24 Oct 2016 at 17:07 ZHU Hua B  wrote:

> Hi,
>
>
> Thanks for your info!
>
>
> Before I launch mirror maker first time, there is a topic include some 
> messages, which have been produced and consumed on source Kafka cluster.
> Then I launch mirror maker first time on target cluster, but this 
> topic did not mirror from source cluster.
>
> So I keep mirror maker running on target cluster, and produced a new 
> message to this topic on source cluster, this time the topic mirror 
> from source cluster to target, which include the new message only, if 
> this behavior is correct? Thanks!
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: 2016年10月24日 16:56
> To: users@kafka.apache.org
> Subject: Re: Mirror multi-embedded consumer's configuration
>
> Yes, it should mirror again. To mirror the topic from beginning, you 
> may have to start with different consumer group,
>
> On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B 
> 
> wrote:
>
> > Hi,
> >
> >
> > Many thanks for your confirm!
> >
> > I have another question, if I deleted a mirrored topic on the target 
> > Kafka cluster, if Kafka mirror maker could mirror the same topic 
> > again from source cluster when I launch mirror maker next time? Thanks!
> >
> >
> >
> >
> >
> >
> > Best Regards
> >
> > Johnny
> >
> >
> > -Original Message-
> > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > Sent: 2016年10月24日 15:03
> > To: users@kafka.apache.org
> > Subject: Re: Mirror multi-embedded consumer's configuration
> >
> > I think we need to update the docs. This behavior got changed 
> > longtime
> ago.
> > we need to pass only one consumer configuration. This config will be 
> > applicable for all the consumer threads/instances.
> >
> > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B 
> > 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > Because from some Kafka wiki I saw "At minimum, the mirror maker 
> > > takes one or more consumer configurations, a producer 
> > > configuration and either a whitelist or a blacklist", but the test 
> > > failed, so I want to know if Kafka mirror really support more 
> > > consumer configurations at
> > minimum? Thanks!
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > >
> > > -Original Message-
> > > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > > Sent: 2016年10月24日 13:48
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror multi-embedded consumer's configuration
> > >
> > > why are you passing "consumer.config" twice?
> > >
> > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B 
> > > 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > The version of Kafka I used is 0.10.0.0. Thanks!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Johnny
> > > >
> > > > -Original Message-
> > > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > > Sent: 2016年10月24日 12:22
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Mirror multi-embedded consumer's configuration
> > > >
> > > > Which version are you using for the MM?
> > > >
> > > > Guozhang
> > > >
> > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B 
> > > > 
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > Anybody could help to answer below question? Thanks!
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Best Regards
> > > > >
> > > > > Johnny
> > > > >
> > > > > From: ZHU Hua B
> > > > > Sent: 2016年10月19日 16:22
> > > > > To: 'users@kafka.apache.org'
> > > > > Subject: Mirror multi-embedded consumer's configuration
> > > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > I launch Kafka mirror maker with multi-embedded consumer's 
> > > > > configuration but failed as below, what’s the mean of “you 
> > > > > asked for only one”, is there an option control it? Thanks!
> > > > >
> > > > > # bin/kafka-mirror-maker.sh --consumer.config 
> > > > > config/consumer-1.properties --consumer.config 
> > > > > config/consumer-2.properties --num.streams 2 --producer.config
> > > > config/producer.properties --whitelist '.*'
> > > > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror
> maker.
> > > > > (kafka.tools.MirrorMaker$)
> > > > > joptsimple.MultipleArgumentsForOptionException: Found multiple 
> > > > > arguments for option consumer.config, but you asked for only one
> > > > 

RE: customised event time

[Hamza HACHANI]

And the start time and end time of the window.

In other words i need the notion of windows in the proecessor API.

Is this possible ?


De : Hamza HACHANI 
Envoyé : dimanche 23 octobre 2016 20:43:05
À : users@kafka.apache.org
Objet : RE: customised event time

To be more specific.

What id do really need is the property of the retention time dor the window in 
the processor API.

Because for the window  i think that i can manage to do this.


Hamza


De : Hamza HACHANI 
Envoyé : dimanche 23 octobre 2016 20:30:13
À : users@kafka.apache.org
Objet : RE: customised event time

Hi,

I think that maybe i'm asking much.

But Ineed the aspect of windowing in the processor API not in the Stram DSL. Is 
this possible?

The second question is how can i get rid of the intermediate results because 
i'm only interested in the final result given by the window.

Hamza


De : Matthias J. Sax 
Envoyé : samedi 22 octobre 2016 16:12:45
À : users@kafka.apache.org
Objet : Re: customised event time

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi,

you can set window retention time via method Windows#until() (and this
retention time is based on the timestamps returned from you custom
timestamp extractor). This keeps all windows until the retention time
passes and thus, all later arrival records will be processed correctly.

However, Kafka Streams does not close windows as other framework, but
rather gives you an (intermediate) result each time a window is
updated with a new record (regardless if the record is in-order or
late -- you will get a result record in both cases).

As of Kafka 0.10.1 those (intermediate) results get deduplicated so
you might not receive all (intermediate) results downstream. Of
course, it is ensured, that you will eventually get the latest/final
result sent downstream.


- -Matthias

On 10/21/16 7:42 AM, Hamza HACHANI wrote:
> Hi,
>
>
> I would like to process data based on a customised event time.(a
> timestamp that I implement as part of the message).
>
> The data is processed in periodic windows of x time that are
> parametered via the method punctuate.
>
> What I need is a retention time for the window to be able to treat
> the late arriving messages.
>
> Can I do this : define/configure a retention time for windows . For
> example the window which treat data between 15pm and 16pm forward
> the result not in 16pm but in 16:15 pm.
>
> Thanks in advance for your help.
>
>
> Hamza
>
>
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJYDDi9AAoJECnhiMLycopPhR8P+wVnRiRwuM7omj3l4EfBVP9Q
7Vflwzf9TFmCjmtpSkezQVNSOhU7maUEB7R/4rjqacLuALJBPrOZ+mrRfYIx/oJd
E7oc3JXO1dlANMyKQ8SQ9dkXWxkeyNtl9ujkcFhMBejqtamteVaI/iNdUdbs6CCk
AZG86yQKhhPL2Q+BZK0eu88bc0yB5Kwj86uCFFkzMaMXVkvCY1f/kkplRGADsSfd
dfv5LONFzXmJ5dftC1giMis4Pg/kJU9gwvmh+WJB2tSM8KjXf1hZWe4nv1f+af8g
pVLnyepv7i7I6PwX/wSdtY2uSLKjizLSbvR3JsfAqLWSVBMgnPpHjCvGecR8K3IC
FwsnWUk9QJKn4EoahsfRq+afUYva8HRxNH7Aiq+gXvEx9LPgwCqbOGYy4ZnOaoi5
PycddoNaK1QEn3WYSxN88kLWFx5+vem4LWiBSDBT+Px0p14ZgKwEjqNgyj4fxBtw
SRsrxVvXXcj2qWvjtRCmTYpn7hanAbYT0QoNfIPs20/Jrh3qsaVJIt5xzRuYyxC8
aobYLlRFLL82eYaPcdQNz/1T0Q7kT9NbEabiuADaZIz7X1yQluMWuTaGIDk/6j6v
A2OqaJd8+KkMMOy9PUX1rBnJhJPZ4CTvkzmjnY10HU7sneNpBvU0tfnPdBBCU15B
8bsUiTf0lk6t9amGYT6q
=PcW7
-END PGP SIGNATURE-

Re: Mirror multi-embedded consumer's configuration

[tao xiao]

You need to set auto.offset.reset=smallest to mirror data from beginning

On Mon, 24 Oct 2016 at 17:07 ZHU Hua B  wrote:

> Hi,
>
>
> Thanks for your info!
>
>
> Before I launch mirror maker first time, there is a topic include some
> messages, which have been produced and consumed on source Kafka cluster.
> Then I launch mirror maker first time on target cluster, but this topic did
> not mirror from source cluster.
>
> So I keep mirror maker running on target cluster, and produced a new
> message to this topic on source cluster, this time the topic mirror from
> source cluster to target, which include the new message only, if this
> behavior is correct? Thanks!
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: 2016年10月24日 16:56
> To: users@kafka.apache.org
> Subject: Re: Mirror multi-embedded consumer's configuration
>
> Yes, it should mirror again. To mirror the topic from beginning, you may
> have to start with different consumer group,
>
> On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B 
> wrote:
>
> > Hi,
> >
> >
> > Many thanks for your confirm!
> >
> > I have another question, if I deleted a mirrored topic on the target
> > Kafka cluster, if Kafka mirror maker could mirror the same topic again
> > from source cluster when I launch mirror maker next time? Thanks!
> >
> >
> >
> >
> >
> >
> > Best Regards
> >
> > Johnny
> >
> >
> > -Original Message-
> > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > Sent: 2016年10月24日 15:03
> > To: users@kafka.apache.org
> > Subject: Re: Mirror multi-embedded consumer's configuration
> >
> > I think we need to update the docs. This behavior got changed longtime
> ago.
> > we need to pass only one consumer configuration. This config will be
> > applicable for all the consumer threads/instances.
> >
> > On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B
> > 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > Because from some Kafka wiki I saw "At minimum, the mirror maker
> > > takes one or more consumer configurations, a producer configuration
> > > and either a whitelist or a blacklist", but the test failed, so I
> > > want to know if Kafka mirror really support more consumer
> > > configurations at
> > minimum? Thanks!
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > >
> > > -Original Message-
> > > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > > Sent: 2016年10月24日 13:48
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror multi-embedded consumer's configuration
> > >
> > > why are you passing "consumer.config" twice?
> > >
> > > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B
> > > 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > The version of Kafka I used is 0.10.0.0. Thanks!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Johnny
> > > >
> > > > -Original Message-
> > > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > > Sent: 2016年10月24日 12:22
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Mirror multi-embedded consumer's configuration
> > > >
> > > > Which version are you using for the MM?
> > > >
> > > > Guozhang
> > > >
> > > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B
> > > > 
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > Anybody could help to answer below question? Thanks!
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Best Regards
> > > > >
> > > > > Johnny
> > > > >
> > > > > From: ZHU Hua B
> > > > > Sent: 2016年10月19日 16:22
> > > > > To: 'users@kafka.apache.org'
> > > > > Subject: Mirror multi-embedded consumer's configuration
> > > > >
> > > > > Hi,
> > > > >
> > > > >
> > > > > I launch Kafka mirror maker with multi-embedded consumer's
> > > > > configuration but failed as below, what’s the mean of “you asked
> > > > > for only one”, is there an option control it? Thanks!
> > > > >
> > > > > # bin/kafka-mirror-maker.sh --consumer.config
> > > > > config/consumer-1.properties --consumer.config
> > > > > config/consumer-2.properties --num.streams 2 --producer.config
> > > > config/producer.properties --whitelist '.*'
> > > > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror
> maker.
> > > > > (kafka.tools.MirrorMaker$)
> > > > > joptsimple.MultipleArgumentsForOptionException: Found multiple
> > > > > arguments for option consumer.config, but you asked for only one
> > > > > at joptsimple.OptionSet.valueOf(OptionSet.java:179)
> > > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235)
> > > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > > > Exception in thread "main" java.lang.NullPointerException
> > > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> > > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Best Regards
> > > > >
> > > > > Johnny
> > > > >
> > > > >
> > > >
> > > >

RE: Mirror multi-embedded consumer's configuration

[ZHU Hua B]

Hi,


Thanks for your info!


Before I launch mirror maker first time, there is a topic include some 
messages, which have been produced and consumed on source Kafka cluster. Then I 
launch mirror maker first time on target cluster, but this topic did not mirror 
from source cluster.

So I keep mirror maker running on target cluster, and produced a new message to 
this topic on source cluster, this time the topic mirror from source cluster to 
target, which include the new message only, if this behavior is correct? Thanks!






Best Regards

Johnny


-Original Message-
From: Manikumar [mailto:manikumar.re...@gmail.com] 
Sent: 2016年10月24日 16:56
To: users@kafka.apache.org
Subject: Re: Mirror multi-embedded consumer's configuration

Yes, it should mirror again. To mirror the topic from beginning, you may have 
to start with different consumer group,

On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B 
wrote:

> Hi,
>
>
> Many thanks for your confirm!
>
> I have another question, if I deleted a mirrored topic on the target 
> Kafka cluster, if Kafka mirror maker could mirror the same topic again 
> from source cluster when I launch mirror maker next time? Thanks!
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: 2016年10月24日 15:03
> To: users@kafka.apache.org
> Subject: Re: Mirror multi-embedded consumer's configuration
>
> I think we need to update the docs. This behavior got changed longtime ago.
> we need to pass only one consumer configuration. This config will be 
> applicable for all the consumer threads/instances.
>
> On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B 
> 
> wrote:
>
> > Hi,
> >
> >
> > Because from some Kafka wiki I saw "At minimum, the mirror maker 
> > takes one or more consumer configurations, a producer configuration 
> > and either a whitelist or a blacklist", but the test failed, so I 
> > want to know if Kafka mirror really support more consumer 
> > configurations at
> minimum? Thanks!
> >
> >
> >
> >
> > Best Regards
> >
> > Johnny
> >
> >
> > -Original Message-
> > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > Sent: 2016年10月24日 13:48
> > To: users@kafka.apache.org
> > Subject: Re: Mirror multi-embedded consumer's configuration
> >
> > why are you passing "consumer.config" twice?
> >
> > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B 
> > 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > The version of Kafka I used is 0.10.0.0. Thanks!
> > >
> > >
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > > -Original Message-
> > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > Sent: 2016年10月24日 12:22
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror multi-embedded consumer's configuration
> > >
> > > Which version are you using for the MM?
> > >
> > > Guozhang
> > >
> > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B 
> > > 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > Anybody could help to answer below question? Thanks!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Johnny
> > > >
> > > > From: ZHU Hua B
> > > > Sent: 2016年10月19日 16:22
> > > > To: 'users@kafka.apache.org'
> > > > Subject: Mirror multi-embedded consumer's configuration
> > > >
> > > > Hi,
> > > >
> > > >
> > > > I launch Kafka mirror maker with multi-embedded consumer's 
> > > > configuration but failed as below, what’s the mean of “you asked 
> > > > for only one”, is there an option control it? Thanks!
> > > >
> > > > # bin/kafka-mirror-maker.sh --consumer.config 
> > > > config/consumer-1.properties --consumer.config 
> > > > config/consumer-2.properties --num.streams 2 --producer.config
> > > config/producer.properties --whitelist '.*'
> > > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker.
> > > > (kafka.tools.MirrorMaker$)
> > > > joptsimple.MultipleArgumentsForOptionException: Found multiple 
> > > > arguments for option consumer.config, but you asked for only one
> > > > at joptsimple.OptionSet.valueOf(OptionSet.java:179)
> > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235)
> > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > > Exception in thread "main" java.lang.NullPointerException
> > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Johnny
> > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: Mirror multi-embedded consumer's configuration

[Manikumar]

Yes, it should mirror again. To mirror the topic from beginning, you may
have to start
with different consumer group,

On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B 
wrote:

> Hi,
>
>
> Many thanks for your confirm!
>
> I have another question, if I deleted a mirrored topic on the target Kafka
> cluster, if Kafka mirror maker could mirror the same topic again from
> source cluster when I launch mirror maker next time? Thanks!
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: 2016年10月24日 15:03
> To: users@kafka.apache.org
> Subject: Re: Mirror multi-embedded consumer's configuration
>
> I think we need to update the docs. This behavior got changed longtime ago.
> we need to pass only one consumer configuration. This config will be
> applicable for all the consumer threads/instances.
>
> On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B 
> wrote:
>
> > Hi,
> >
> >
> > Because from some Kafka wiki I saw "At minimum, the mirror maker takes
> > one or more consumer configurations, a producer configuration and
> > either a whitelist or a blacklist", but the test failed, so I want to
> > know if Kafka mirror really support more consumer configurations at
> minimum? Thanks!
> >
> >
> >
> >
> > Best Regards
> >
> > Johnny
> >
> >
> > -Original Message-
> > From: Manikumar [mailto:manikumar.re...@gmail.com]
> > Sent: 2016年10月24日 13:48
> > To: users@kafka.apache.org
> > Subject: Re: Mirror multi-embedded consumer's configuration
> >
> > why are you passing "consumer.config" twice?
> >
> > On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B
> > 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > The version of Kafka I used is 0.10.0.0. Thanks!
> > >
> > >
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > > -Original Message-
> > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > Sent: 2016年10月24日 12:22
> > > To: users@kafka.apache.org
> > > Subject: Re: Mirror multi-embedded consumer's configuration
> > >
> > > Which version are you using for the MM?
> > >
> > > Guozhang
> > >
> > > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B
> > > 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > >
> > > > Anybody could help to answer below question? Thanks!
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Johnny
> > > >
> > > > From: ZHU Hua B
> > > > Sent: 2016年10月19日 16:22
> > > > To: 'users@kafka.apache.org'
> > > > Subject: Mirror multi-embedded consumer's configuration
> > > >
> > > > Hi,
> > > >
> > > >
> > > > I launch Kafka mirror maker with multi-embedded consumer's
> > > > configuration but failed as below, what’s the mean of “you asked
> > > > for only one”, is there an option control it? Thanks!
> > > >
> > > > # bin/kafka-mirror-maker.sh --consumer.config
> > > > config/consumer-1.properties --consumer.config
> > > > config/consumer-2.properties --num.streams 2 --producer.config
> > > config/producer.properties --whitelist '.*'
> > > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker.
> > > > (kafka.tools.MirrorMaker$)
> > > > joptsimple.MultipleArgumentsForOptionException: Found multiple
> > > > arguments for option consumer.config, but you asked for only one
> > > > at joptsimple.OptionSet.valueOf(OptionSet.java:179)
> > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235)
> > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > > Exception in thread "main" java.lang.NullPointerException
> > > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> > > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards
> > > >
> > > > Johnny
> > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: Kafka unable to find topic details after sudden EC2 server restart

[Gourab Chowdhury]

Thanks for the reply, I tried changing the data directory as follows:-
dataDir=/data/zookeeper

I also create a /data but still I am getting the following error.

ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.
ZooKeeperServerMain)
java.io.IOException: Unable to create data directory
/data/zookeeper/version-2

Gourab

On Sat, Oct 22, 2016 at 3:44 AM, Hans Jespersen  wrote:

> You are going to lose everything you store in /tmp. In a production system
> you never configure Kafka or zookeeper to store critical data in /tmp.
> This has nothing to do with AWS or EBS it is just standard Linux than
> everything under /tmp is deleted when Linux reboots.
>
> -hans
>
> /**
>  * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
>  * h...@confluent.io (650)924-2670
>  */
>
> On Fri, Oct 21, 2016 at 10:38 AM, Gourab Chowdhury 
> wrote:
>
> > I was testing a scenario of a server crash while the Kafka is still
> running
> > and passing messages. (just did a sudo reboot). By default, I am storing
> > the logs at /tmp/kafka-log.
> >
> > So in case an EC2 server restart, am I going to loose the data if I don't
> > store kafka logs in an EBS drive?
> >
> > Also, I have another doubt, Does zookeeper stores Kafka info (viz.
> topics,
> > brokers list, consumer groups) at /tmp/zookeeper/version-2/log.x ?
> >
> > I am using python zookeeper client: *kazoo* and here we can find various
> > Kafka info like '/brokers/topics', '/brokers/topics/topic/partitions' ,
> > '/consumers' , '/consumers/consumer/offsets'. Where can I get this info
> > directly?
> >
> > ~Gourab
> >
> > On Fri, Oct 21, 2016 at 8:13 PM, Ben Davison 
> > wrote:
> >
> > > Are you restarting or terminating the instance? If your terminating,
> and
> > > don't have an EBS drive attached you will lose all data on the drive.
> > >
> > > On Fri, Oct 21, 2016 at 2:09 PM, Gourab Chowdhury <
> gourab@gmail.com>
> > > wrote:
> > >
> > > > I am running kafka in a EC2 server.  While running if there is a
> sudden
> > > > restart of server, and then we restart the kafka broker, we can't
> find
> > > the
> > > > details of all the topics.
> > > >
> > > > bin/kafka-topics.sh --describe --zookeeper 
> > > > gives no result
> > > >
> > > > bin/kafka-console-producer.sh --broker-list  --topic
> > test
> > > > yields this particular error as follows:
> > > >
> > > > WARN Error while fetching metadata with correlation id n :
> > > > {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.
> > > NetworkClient)
> > > >
> > > > Is the details of topics from the zookeeper getting deleted while
> > > restart?
> > > > or this is something else?
> > > >
> > > > Also how can I cope with this in in real life production?
> > > >
> > > > Regards,
> > > > Gourab Chowdhury,
> > > >
> > >
> > > --
> > >
> > >
> > > This email, including attachments, is private and confidential. If you
> > have
> > > received this email in error please notify the sender and delete it
> from
> > > your system. Emails are not secure and may contain viruses. No
> liability
> > > can be accepted for viruses that might be transferred by this email or
> > any
> > > attachment. Any unauthorised copying of this message or unauthorised
> > > distribution and publication of the information contained herein are
> > > prohibited.
> > >
> > > 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
> > > Registered in England and Wales. Registered No. 04843573.
> > >
> >
>

RE: customised event time

[Hamza HACHANI]

To be more specific.

What id do really need is the property of the retention time dor the window in 
the processor API.

Because for the window  i think that i can manage to do this.


Hamza


De : Hamza HACHANI 
Envoyé : dimanche 23 octobre 2016 20:30:13
À : users@kafka.apache.org
Objet : RE: customised event time

Hi,

I think that maybe i'm asking much.

But Ineed the aspect of windowing in the processor API not in the Stram DSL. Is 
this possible?

The second question is how can i get rid of the intermediate results because 
i'm only interested in the final result given by the window.

Hamza


De : Matthias J. Sax 
Envoyé : samedi 22 octobre 2016 16:12:45
À : users@kafka.apache.org
Objet : Re: customised event time

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi,

you can set window retention time via method Windows#until() (and this
retention time is based on the timestamps returned from you custom
timestamp extractor). This keeps all windows until the retention time
passes and thus, all later arrival records will be processed correctly.

However, Kafka Streams does not close windows as other framework, but
rather gives you an (intermediate) result each time a window is
updated with a new record (regardless if the record is in-order or
late -- you will get a result record in both cases).

As of Kafka 0.10.1 those (intermediate) results get deduplicated so
you might not receive all (intermediate) results downstream. Of
course, it is ensured, that you will eventually get the latest/final
result sent downstream.


- -Matthias

On 10/21/16 7:42 AM, Hamza HACHANI wrote:
> Hi,
>
>
> I would like to process data based on a customised event time.(a
> timestamp that I implement as part of the message).
>
> The data is processed in periodic windows of x time that are
> parametered via the method punctuate.
>
> What I need is a retention time for the window to be able to treat
> the late arriving messages.
>
> Can I do this : define/configure a retention time for windows . For
> example the window which treat data between 15pm and 16pm forward
> the result not in 16pm but in 16:15 pm.
>
> Thanks in advance for your help.
>
>
> Hamza
>
>
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJYDDi9AAoJECnhiMLycopPhR8P+wVnRiRwuM7omj3l4EfBVP9Q
7Vflwzf9TFmCjmtpSkezQVNSOhU7maUEB7R/4rjqacLuALJBPrOZ+mrRfYIx/oJd
E7oc3JXO1dlANMyKQ8SQ9dkXWxkeyNtl9ujkcFhMBejqtamteVaI/iNdUdbs6CCk
AZG86yQKhhPL2Q+BZK0eu88bc0yB5Kwj86uCFFkzMaMXVkvCY1f/kkplRGADsSfd
dfv5LONFzXmJ5dftC1giMis4Pg/kJU9gwvmh+WJB2tSM8KjXf1hZWe4nv1f+af8g
pVLnyepv7i7I6PwX/wSdtY2uSLKjizLSbvR3JsfAqLWSVBMgnPpHjCvGecR8K3IC
FwsnWUk9QJKn4EoahsfRq+afUYva8HRxNH7Aiq+gXvEx9LPgwCqbOGYy4ZnOaoi5
PycddoNaK1QEn3WYSxN88kLWFx5+vem4LWiBSDBT+Px0p14ZgKwEjqNgyj4fxBtw
SRsrxVvXXcj2qWvjtRCmTYpn7hanAbYT0QoNfIPs20/Jrh3qsaVJIt5xzRuYyxC8
aobYLlRFLL82eYaPcdQNz/1T0Q7kT9NbEabiuADaZIz7X1yQluMWuTaGIDk/6j6v
A2OqaJd8+KkMMOy9PUX1rBnJhJPZ4CTvkzmjnY10HU7sneNpBvU0tfnPdBBCU15B
8bsUiTf0lk6t9amGYT6q
=PcW7
-END PGP SIGNATURE-

RE: customised event time

[Hamza HACHANI]

Hi,

I think that maybe i'm asking much.

But Ineed the aspect of windowing in the processor API not in the Stram DSL. Is 
this possible?

The second question is how can i get rid of the intermediate results because 
i'm only interested in the final result given by the window.

Hamza


De : Matthias J. Sax 
Envoyé : samedi 22 octobre 2016 16:12:45
À : users@kafka.apache.org
Objet : Re: customised event time

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi,

you can set window retention time via method Windows#until() (and this
retention time is based on the timestamps returned from you custom
timestamp extractor). This keeps all windows until the retention time
passes and thus, all later arrival records will be processed correctly.

However, Kafka Streams does not close windows as other framework, but
rather gives you an (intermediate) result each time a window is
updated with a new record (regardless if the record is in-order or
late -- you will get a result record in both cases).

As of Kafka 0.10.1 those (intermediate) results get deduplicated so
you might not receive all (intermediate) results downstream. Of
course, it is ensured, that you will eventually get the latest/final
result sent downstream.


- -Matthias

On 10/21/16 7:42 AM, Hamza HACHANI wrote:
> Hi,
>
>
> I would like to process data based on a customised event time.(a
> timestamp that I implement as part of the message).
>
> The data is processed in periodic windows of x time that are
> parametered via the method punctuate.
>
> What I need is a retention time for the window to be able to treat
> the late arriving messages.
>
> Can I do this : define/configure a retention time for windows . For
> example the window which treat data between 15pm and 16pm forward
> the result not in 16pm but in 16:15 pm.
>
> Thanks in advance for your help.
>
>
> Hamza
>
>
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJYDDi9AAoJECnhiMLycopPhR8P+wVnRiRwuM7omj3l4EfBVP9Q
7Vflwzf9TFmCjmtpSkezQVNSOhU7maUEB7R/4rjqacLuALJBPrOZ+mrRfYIx/oJd
E7oc3JXO1dlANMyKQ8SQ9dkXWxkeyNtl9ujkcFhMBejqtamteVaI/iNdUdbs6CCk
AZG86yQKhhPL2Q+BZK0eu88bc0yB5Kwj86uCFFkzMaMXVkvCY1f/kkplRGADsSfd
dfv5LONFzXmJ5dftC1giMis4Pg/kJU9gwvmh+WJB2tSM8KjXf1hZWe4nv1f+af8g
pVLnyepv7i7I6PwX/wSdtY2uSLKjizLSbvR3JsfAqLWSVBMgnPpHjCvGecR8K3IC
FwsnWUk9QJKn4EoahsfRq+afUYva8HRxNH7Aiq+gXvEx9LPgwCqbOGYy4ZnOaoi5
PycddoNaK1QEn3WYSxN88kLWFx5+vem4LWiBSDBT+Px0p14ZgKwEjqNgyj4fxBtw
SRsrxVvXXcj2qWvjtRCmTYpn7hanAbYT0QoNfIPs20/Jrh3qsaVJIt5xzRuYyxC8
aobYLlRFLL82eYaPcdQNz/1T0Q7kT9NbEabiuADaZIz7X1yQluMWuTaGIDk/6j6v
A2OqaJd8+KkMMOy9PUX1rBnJhJPZ4CTvkzmjnY10HU7sneNpBvU0tfnPdBBCU15B
8bsUiTf0lk6t9amGYT6q
=PcW7
-END PGP SIGNATURE-

RE: Mirror multi-embedded consumer's configuration

[ZHU Hua B]

Hi,


Many thanks for your confirm!

I have another question, if I deleted a mirrored topic on the target Kafka 
cluster, if Kafka mirror maker could mirror the same topic again from source 
cluster when I launch mirror maker next time? Thanks!






Best Regards

Johnny


-Original Message-
From: Manikumar [mailto:manikumar.re...@gmail.com] 
Sent: 2016年10月24日 15:03
To: users@kafka.apache.org
Subject: Re: Mirror multi-embedded consumer's configuration

I think we need to update the docs. This behavior got changed longtime ago.
we need to pass only one consumer configuration. This config will be applicable 
for all the consumer threads/instances.

On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B 
wrote:

> Hi,
>
>
> Because from some Kafka wiki I saw "At minimum, the mirror maker takes 
> one or more consumer configurations, a producer configuration and 
> either a whitelist or a blacklist", but the test failed, so I want to 
> know if Kafka mirror really support more consumer configurations at minimum? 
> Thanks!
>
>
>
>
> Best Regards
>
> Johnny
>
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: 2016年10月24日 13:48
> To: users@kafka.apache.org
> Subject: Re: Mirror multi-embedded consumer's configuration
>
> why are you passing "consumer.config" twice?
>
> On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B 
> 
> wrote:
>
> > Hi,
> >
> >
> > The version of Kafka I used is 0.10.0.0. Thanks!
> >
> >
> >
> >
> >
> >
> > Best Regards
> >
> > Johnny
> >
> > -Original Message-
> > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > Sent: 2016年10月24日 12:22
> > To: users@kafka.apache.org
> > Subject: Re: Mirror multi-embedded consumer's configuration
> >
> > Which version are you using for the MM?
> >
> > Guozhang
> >
> > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B 
> > 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > Anybody could help to answer below question? Thanks!
> > >
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > > From: ZHU Hua B
> > > Sent: 2016年10月19日 16:22
> > > To: 'users@kafka.apache.org'
> > > Subject: Mirror multi-embedded consumer's configuration
> > >
> > > Hi,
> > >
> > >
> > > I launch Kafka mirror maker with multi-embedded consumer's 
> > > configuration but failed as below, what’s the mean of “you asked 
> > > for only one”, is there an option control it? Thanks!
> > >
> > > # bin/kafka-mirror-maker.sh --consumer.config 
> > > config/consumer-1.properties --consumer.config 
> > > config/consumer-2.properties --num.streams 2 --producer.config
> > config/producer.properties --whitelist '.*'
> > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker.
> > > (kafka.tools.MirrorMaker$)
> > > joptsimple.MultipleArgumentsForOptionException: Found multiple 
> > > arguments for option consumer.config, but you asked for only one
> > > at joptsimple.OptionSet.valueOf(OptionSet.java:179)
> > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235)
> > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > Exception in thread "main" java.lang.NullPointerException
> > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > >
> > >
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: Kafka Streaming

[Matthias J. Sax]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Kafka 0.10.1.0 which was release last week does contain the fix
already. The fix will be in CP 3.1 coming up soon!

(sorry that I did mix up versions in a previous email)

- -Matthias

On 10/23/16 12:10 PM, Mohit Anchlia wrote:
> So if I get it right I will not have this fix until 4 months?
> Should I just create my own example with the next version of
> Kafka?
> 
> On Sat, Oct 22, 2016 at 9:04 PM, Matthias J. Sax
>  wrote:
> 
> Current version is 3.0.1 CP 3.1 should be release the next weeks
> 
> So CP 3.2 should be there is about 4 month (Kafka follows a time
> base release cycle of 4 month and CP usually aligns with Kafka
> releases)
> 
> -Matthias
> 
> 
> On 10/20/16 5:10 PM, Mohit Anchlia wrote:
 Any idea of when 3.2 is coming?
 
 On Thu, Oct 20, 2016 at 4:53 PM, Matthias J. Sax 
  wrote:
 
 No problem. Asking questions is the purpose of mailing lists.
 :)
 
 The issue will be fixed in next version of examples branch.
 
 Examples branch is build with CP dependency and not with
 Kafka dependency. CP-3.2 is not available yet; only Kafka
 0.10.1.0. Nevertheless, they should work with Kafka
 dependency, too. I never tried it, but you should give it a
 shot...
 
 But you should use example master branch because of API
 changes from 0.10.0.x to 0.10.1 (and thus, changing CP-3.1 to
 0.10.1.0 will not be compatible and not compile, while
 changing CP-3.2-SNAPSHOT to 0.10.1.0 should work -- hopefully
 ;) )
 
 
 -Matthias
 
 On 10/20/16 4:02 PM, Mohit Anchlia wrote:
>>> So this issue I am seeing is fixed in the next version
>>> of example branch? Can I change my pom to point it the
>>> higher version of Kafka if that is the issue? Or do I
>>> need to wait until new branch is made available? Sorry
>>> lot of questions :)
>>> 
>>> On Thu, Oct 20, 2016 at 3:56 PM, Matthias J. Sax 
>>>  wrote:
>>> 
>>> The branch is 0.10.0.1 and not 0.10.1.0 (sorry for so
>>> many zeros and ones -- super easy to mix up)
>>> 
>>> However, examples master branch uses CP-3.1-SNAPSHOT
>>> (ie, Kafka 0.10.1.0) -- there will be a 0.10.1 examples
>>> branch, after CP-3.1 was released
>>> 
>>> 
>>> -Matthias
>>> 
>>> On 10/20/16 3:48 PM, Mohit Anchlia wrote:
>> I just now cloned this repo. It seems to be using
>> 10.1
>> 
>> https://github.com/confluentinc/examples and
>> running examples in 
>> https://github.com/confluentinc/examples/tree/kafka-0.10.0.1-
cp-
>
>> 
3.0
 
>> 
> .1/
>>> 
>> 
 kafka-streams
>> 
>> On Thu, Oct 20, 2016 at 3:10 PM, Michael Noll 
>>  wrote:
>> 
>>> I suspect you are running Kafka 0.10.0.x on
>>> Windows? If so, this is a known issue that is
>>> fixed in Kafka 0.10.1 that was just released
>>> today.
>>> 
>>> Also: which examples are you referring to?
>>> And, to confirm: which git branch / Kafka
>>> version / OS in case my guess above was wrong.
>>> 
>>> 
>>> On Thursday, October 20, 2016, Mohit Anchlia 
>>>  wrote:
>>> 
 I am trying to run the examples from git.
 While running the wordcount example I see
 this error:
 
 Caused by: *java.lang.RuntimeException*: 
 librocksdbjni-win64.dll was not found inside
 JAR.
 
 
 Am I expected to include this jar locally?
 
>>> 
>>> 
>>> -- *Michael G. Noll* Product Manager |
>>> Confluent +1 650 453 5860 | @miguno
>>>  Follow us: Twitter
>>>  | Blog
>>> 
>>> 
>> 
 
>>> 
> 
 
>> 
> 
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJYDbOJAAoJECnhiMLycopPSF0QAKFi0lG3dMccIaBuSzsgj2Of
Iu9BnoTCr4MJoY18loPE8DUy9kbEjaf6RjMzkvXvTV+Jjys7PgTa5qLlo7hEwgb6
UxWZYrdOXUGQcJckmsdzMIoGY79GETQuvRkN8uzJ/hfJJ4YNgtLCXaNMDTfcRmtp
hson3M7SYRISSz7Jz9jM7PmPnR6MIWXRV+8r9jdL8IH+gRtrnT1svr/pkiAL6e7B
gZdQHhTVI+dqWRgHfuenr2MQH2Od3UgZsK3pqVlfv0Zy7c3IL6FKN4WQgDiYOYBH
URPY9BBQGnOTINEmX63Bwx8ns+RNgtyIO9O3ytewRmyWRkQIzO/K0zszSjblPJeG
rNnrC9wAQH9arx6ZHAw/moEfpsRCAJT5SGjCnYa8NaDq0wlbHM+lb/AWge2q9C8N
/op4pBImyxpqbWVNjUhWwGoGAqTliXqs4QlyuBi0KhIKXYwsXELthillnDNGbhnA
ZRxcGOfA3CS2qb+GiwmsxKSLcTXq9U7RlNmCLmW0+Xbg/98CQGSKPMMzG7lkgK2v
d8SmN8dafa0yiXmf/CtdMGCRScNyLVtS5XLrQRyG3kXWSlNlxcjFMQGbuidFMIdv
JLRiF4FGv6zjWSWmP7L75vyZQ+yVApvk2XDX00wT/hHIFC2SE2GYWB0usLMAw8/K
jjWAQ1QCKsuW+ZzSkD7D
=DMlr
-END PGP SIGNATURE-

Re: Mirror multi-embedded consumer's configuration

[Manikumar]

I think we need to update the docs. This behavior got changed longtime ago.
we need to pass only one consumer configuration. This config will be
applicable
for all the consumer threads/instances.

On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B 
wrote:

> Hi,
>
>
> Because from some Kafka wiki I saw "At minimum, the mirror maker takes one
> or more consumer configurations, a producer configuration and either a
> whitelist or a blacklist", but the test failed, so I want to know if Kafka
> mirror really support more consumer configurations at minimum? Thanks!
>
>
>
>
> Best Regards
>
> Johnny
>
>
> -Original Message-
> From: Manikumar [mailto:manikumar.re...@gmail.com]
> Sent: 2016年10月24日 13:48
> To: users@kafka.apache.org
> Subject: Re: Mirror multi-embedded consumer's configuration
>
> why are you passing "consumer.config" twice?
>
> On Mon, Oct 24, 2016 at 11:07 AM, ZHU Hua B 
> wrote:
>
> > Hi,
> >
> >
> > The version of Kafka I used is 0.10.0.0. Thanks!
> >
> >
> >
> >
> >
> >
> > Best Regards
> >
> > Johnny
> >
> > -Original Message-
> > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > Sent: 2016年10月24日 12:22
> > To: users@kafka.apache.org
> > Subject: Re: Mirror multi-embedded consumer's configuration
> >
> > Which version are you using for the MM?
> >
> > Guozhang
> >
> > On Thu, Oct 20, 2016 at 10:13 PM, ZHU Hua B
> > 
> > wrote:
> >
> > > Hi,
> > >
> > >
> > > Anybody could help to answer below question? Thanks!
> > >
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > > From: ZHU Hua B
> > > Sent: 2016年10月19日 16:22
> > > To: 'users@kafka.apache.org'
> > > Subject: Mirror multi-embedded consumer's configuration
> > >
> > > Hi,
> > >
> > >
> > > I launch Kafka mirror maker with multi-embedded consumer's
> > > configuration but failed as below, what’s the mean of “you asked for
> > > only one”, is there an option control it? Thanks!
> > >
> > > # bin/kafka-mirror-maker.sh --consumer.config
> > > config/consumer-1.properties --consumer.config
> > > config/consumer-2.properties --num.streams 2 --producer.config
> > config/producer.properties --whitelist '.*'
> > > [2016-10-19 16:00:14,183] ERROR Exception when starting mirror maker.
> > > (kafka.tools.MirrorMaker$)
> > > joptsimple.MultipleArgumentsForOptionException: Found multiple
> > > arguments for option consumer.config, but you asked for only one
> > > at joptsimple.OptionSet.valueOf(OptionSet.java:179)
> > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:235)
> > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > > Exception in thread "main" java.lang.NullPointerException
> > > at kafka.tools.MirrorMaker$.main(MirrorMaker.scala:286)
> > > at kafka.tools.MirrorMaker.main(MirrorMaker.scala)
> > >
> > >
> > >
> > >
> > >
> > >
> > > Best Regards
> > >
> > > Johnny
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>