[jira] [Resolved] (KAFKA-10587) Rename kafka-mirror-maker CLI command line arguments for KIP-629

[Konstantine Karantasis (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantine Karantasis resolved KAFKA-10587.

Resolution: Fixed

> Rename kafka-mirror-maker CLI command line arguments for KIP-629
> 
>
> Key: KAFKA-10587
> URL: https://issues.apache.org/jira/browse/KAFKA-10587
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Xavier Léauté
>Assignee: Omnia Ibrahim
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10587) Rename kafka-mirror-maker CLI command line arguments for KIP-629

[Jira]

Xavier Léauté created KAFKA-10587:
-

 Summary: Rename kafka-mirror-maker CLI command line arguments for 
KIP-629
 Key: KAFKA-10587
 URL: https://issues.apache.org/jira/browse/KAFKA-10587
 Project: Kafka
  Issue Type: Sub-task
Reporter: Xavier Léauté






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9745) Attribute is missed in kafka mirror maker metrics

[Nikita Krasnov (Jira)]

Nikita Krasnov created KAFKA-9745:
-

 Summary: Attribute is missed in kafka mirror maker metrics
 Key: KAFKA-9745
 URL: https://issues.apache.org/jira/browse/KAFKA-9745
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 2.5.0
Reporter: Nikita Krasnov
 Attachments: kmm_metrics.png

As it described in Kafka mirror maker 
[doc|https://github.com/apache/kafka/tree/trunk/connect/mirror#monitoring-an-mm2-process]
 metrics are tagged with the following properties:

*target*: alias of target cluster
*source*: alias of source cluster
*topic*: remote topic on target cluster
*partition*: partition being replicated

But in real life *source* attribute is missed  !kmm_metrics.png! 





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Kafka mirror maker help

[Peter Bukowinski]

I run instances of Mirror Maker as supervisord tasks (http://supervisord.org 
). I’d recommend looking into it. In addition to 
letting you sidestep the service issue, supervisord watches the processes and 
can auto-restart them if they stop for any reason.

—
Peter Bukowinski

> On Apr 27, 2018, at 11:58 AM, Hans Jespersen  wrote:
> 
> Sorry I hit send a bit too soon. I was so focused on the systemd part of
> the email and not the Mirror Maker part.
> Confluent packages include Mirror Maker but the systemd scripts are setup
> to use Confluent Replicator rather than Mirror Maker.
> My apologies.
> 
> -hans
> 
> /**
> * Hans Jespersen, Director Systems Engineering, Confluent Inc.
> * h...@confluent.io (650)924-2670
> */
> 
> On Fri, Apr 27, 2018 at 11:56 AM, Hans Jespersen  wrote:
> 
>> The latest Confluent packages now ship with systemd scripts. That is since
>> Confluent Version 4.1 - which included Apache Kafka 1.1
>> 
>> -hans
>> 
>> /**
>> * Hans Jespersen, Director Systems Engineering, Confluent Inc.
>> * h...@confluent.io (650)924-2670
>> */
>> 
>> On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto  wrote:
>> 
>>> Hiya,
>>> 
>>> Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
>>> packaging.  I’ll reply here.
>>> 
>>> We now use Confluent’s Kafka debian packaging which does not (or did not?)
>>> ship with init scripts.  We don’t use Sys V init.d scripts anymore either,
>>> but use systemd instead.  Our systemd service unit (ERB template format)
>>> is
>>> here:
>>> 
>>> https://github.com/wikimedia/puppet/blob/production/modules/
>>> confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
>>> 
>>> 
>>> 
>>> On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid 
>>> wrote:
>>> 
 You should share related info, such source-destination Kafka versions,
 sample Config or error if any.
 
 FYI,  Go through
 https://kafka.apache.org/documentation/#basic_ops_mirror_maker
 
>>> 
>> 
>> 

Re: Kafka mirror maker help

[Hans Jespersen]

Sorry I hit send a bit too soon. I was so focused on the systemd part of
the email and not the Mirror Maker part.
Confluent packages include Mirror Maker but the systemd scripts are setup
to use Confluent Replicator rather than Mirror Maker.
My apologies.

-hans

/**
 * Hans Jespersen, Director Systems Engineering, Confluent Inc.
 * h...@confluent.io (650)924-2670
 */

On Fri, Apr 27, 2018 at 11:56 AM, Hans Jespersen  wrote:

> The latest Confluent packages now ship with systemd scripts. That is since
> Confluent Version 4.1 - which included Apache Kafka 1.1
>
> -hans
>
> /**
>  * Hans Jespersen, Director Systems Engineering, Confluent Inc.
>  * h...@confluent.io (650)924-2670
>  */
>
> On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto  wrote:
>
>> Hiya,
>>
>> Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
>> packaging.  I’ll reply here.
>>
>> We now use Confluent’s Kafka debian packaging which does not (or did not?)
>> ship with init scripts.  We don’t use Sys V init.d scripts anymore either,
>> but use systemd instead.  Our systemd service unit (ERB template format)
>> is
>> here:
>>
>> https://github.com/wikimedia/puppet/blob/production/modules/
>> confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
>>
>>
>>
>> On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid 
>> wrote:
>>
>> > You should share related info, such source-destination Kafka versions,
>> > sample Config or error if any.
>> >
>> > FYI,  Go through
>> > https://kafka.apache.org/documentation/#basic_ops_mirror_maker
>> >
>>
>
>

Re: Kafka mirror maker help

[Hans Jespersen]

The latest Confluent packages now ship with systemd scripts. That is since
Confluent Version 4.1 - which included Apache Kafka 1.1

-hans

/**
 * Hans Jespersen, Director Systems Engineering, Confluent Inc.
 * h...@confluent.io (650)924-2670
 */

On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto  wrote:

> Hiya,
>
> Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
> packaging.  I’ll reply here.
>
> We now use Confluent’s Kafka debian packaging which does not (or did not?)
> ship with init scripts.  We don’t use Sys V init.d scripts anymore either,
> but use systemd instead.  Our systemd service unit (ERB template format) is
> here:
>
> https://github.com/wikimedia/puppet/blob/production/
> modules/confluent/templates/initscripts/kafka-mirror-instance.systemd.erb
>
>
>
> On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid 
> wrote:
>
> > You should share related info, such source-destination Kafka versions,
> > sample Config or error if any.
> >
> > FYI,  Go through
> > https://kafka.apache.org/documentation/#basic_ops_mirror_maker
> >
>

Re: Kafka mirror maker help

[Andrew Otto]

Hiya,

Saravanan, I saw you emailed my colleague Alex about WMF’s old debian
packaging.  I’ll reply here.

We now use Confluent’s Kafka debian packaging which does not (or did not?)
ship with init scripts.  We don’t use Sys V init.d scripts anymore either,
but use systemd instead.  Our systemd service unit (ERB template format) is
here:

https://github.com/wikimedia/puppet/blob/production/modules/confluent/templates/initscripts/kafka-mirror-instance.systemd.erb



On Fri, Apr 27, 2018 at 1:35 AM, Amrit Jangid 
wrote:

> You should share related info, such source-destination Kafka versions,
> sample Config or error if any.
>
> FYI,  Go through
> https://kafka.apache.org/documentation/#basic_ops_mirror_maker
>

Re: Kafka mirror maker help

[Amrit Jangid]

You should share related info, such source-destination Kafka versions,
sample Config or error if any.

FYI,  Go through
https://kafka.apache.org/documentation/#basic_ops_mirror_maker

Kafka mirror maker help

[saravanan kannappan]

Hello,

I'm setting up mirror maker for kafka however it doesn't  help
can you guide me some steps or else share the model shell script to execute
the mirror maker ?

highly appreciated your help.

Thanks
Sara

[jira] [Resolved] (KAFKA-3327) Warning from kafka mirror maker about ssl properties not valid

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3327.
--
Resolution: Cannot Reproduce

mostly related to config issue.  Pl reopen if you think the issue still exists


> Warning from kafka mirror maker about ssl properties not valid
> --
>
> Key: KAFKA-3327
> URL: https://issues.apache.org/jira/browse/KAFKA-3327
> Project: Kafka
>  Issue Type: Test
>  Components: config
>Affects Versions: 0.9.0.1
> Environment: CentOS release 6.5
>Reporter: Munir Khan
>Priority: Minor
>  Labels: kafka, mirror-maker, ssl
>
> I am trying to run Mirror maker  over SSL. I have configured my broker 
> following the procedure described in this document 
> http://kafka.apache.org/documentation.html#security_overview 
> I get the following warning when I start the mirror maker:
> [root@munkhan-kafka1 kafka_2.10-0.9.0.1]# bin/kafka-run-class.sh 
> kafka.tools.MirrorMaker --consumer.config 
> config/datapush-consumer-ssl.properties --producer.config 
> config/datapush-producer-ssl.properties --num.streams 2 --whitelist test1&
> [1] 4701
> [root@munkhan-kafka1 kafka_2.10-0.9.0.1]# [2016-03-03 10:24:35,348] WARN 
> block.on.buffer.full config is deprecated and will be removed soon. Please 
> use max.block.ms (org.apache.kafka.clients.producer.KafkaProducer)
> [2016-03-03 10:24:35,523] WARN The configuration producer.type = sync was 
> supplied but isn't a known config. 
> (org.apache.kafka.clients.producer.ProducerConfig)
> [2016-03-03 10:24:35,523] WARN The configuration ssl.keypassword = test1234 
> was supplied but isn't a known config. 
> (org.apache.kafka.clients.producer.ProducerConfig)
> [2016-03-03 10:24:35,523] WARN The configuration compression.codec = none was 
> supplied but isn't a known config. 
> (org.apache.kafka.clients.producer.ProducerConfig)
> [2016-03-03 10:24:35,523] WARN The configuration serializer.class = 
> kafka.serializer.DefaultEncoder was supplied but isn't a known config. 
> (org.apache.kafka.clients.producer.ProducerConfig)
> [2016-03-03 10:24:35,617] WARN Property security.protocol is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,617] WARN Property ssl.keypassword is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,617] WARN Property ssl.keystore.location is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,618] WARN Property ssl.keystore.password is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,618] WARN Property ssl.truststore.location is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,618] WARN Property ssl.truststore.password is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,752] WARN Property security.protocol is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,752] WARN Property ssl.keypassword is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,752] WARN Property ssl.keystore.location is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,752] WARN Property ssl.keystore.password is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,752] WARN Property ssl.truststore.location is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:35,753] WARN Property ssl.truststore.password is not valid 
> (kafka.utils.VerifiableProperties)
> [2016-03-03 10:24:36,251] WARN No broker partitions consumed by consumer 
> thread test-consumer-group_munkhan-kafka1.cisco.com-1457018675755-b9bb4c75-0 
> for topic test1 (kafka.consumer.RangeAssignor)
> [2016-03-03 10:24:36,251] WARN No broker partitions consumed by consumer 
> thread test-consumer-group_munkhan-kafka1.cisco.com-1457018675755-b9bb4c75-0 
> for topic test1 (kafka.consumer.RangeAssignor)
> However the Mirror maker is able to mirror data . If I remove the 
> configurations related to the warning messages from my producer  mirror maker 
> does not work . So it seems despite the warning shown above the 
> ssl.configuration properties are used somehow. 
> My question is these are those warnings harmless in this context ?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-4102) Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-4102.
--
Resolution: Won't Fix

[yf] Currently the Java clients are only compatible with brokers that are the 
same version or newer. Since MM uses the clients, it has this same limitation.

This won't be fixed for older versions of Kafka, but 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
 will be improving compatibility for 0.10+ brokers & clients.

> Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?
> -
>
> Key: KAFKA-4102
> URL: https://issues.apache.org/jira/browse/KAFKA-4102
> Project: Kafka
>  Issue Type: Bug
>Reporter: yf
> Fix For: 0.10.2.0
>
>
> I want this feature 
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement)
>  in kafka new version.
> But it seems can't support kafka 0.8.X ?
> Cmd: ./kafka_2.11-0.10.0.1/bin/kafka-mirror-maker.sh --producer.config 
> svc_run/mirror_kafka_write_online2streaming/producer.properties 
> --consumer.config 
> svc_run/mirror_kafka_write_online2streaming/consumer.properties --whitelist 
> sandbox
> Producer config:
> queue.buffering.max.messages=16384
> bootstrap.servers=XXX
> send.buffer.bytes=131072
> message.send.max.retries=1048576
> consumer config:
> zookeeper.connect=
> group.id=test_mirror
> consumer.timeout.ms=-1
> zookeeper.connection.timeout.ms=6
> zookeeper.session.timeout.ms=6
> socket.receive.buffer.bytes=-1
> auto.commit.interval.ms=5000
> auto.commit.enable=false
> But I got failures like follows:
> FetchRequest@33437902 (kafka.consumer.ConsumerFetcherThread)
> java.lang.IllegalArgumentException
>   at java.nio.Buffer.limit(Buffer.java:275)
>   at 
> kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
>   at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
>   at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
>   at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
>   at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at 
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
>   at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
>   at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
>   at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
>   at 
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
>   at 
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
>   at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
>   at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4102) Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-4102:
-
Fix Version/s: (was: 0.10.2.0)

> Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?
> -
>
> Key: KAFKA-4102
> URL: https://issues.apache.org/jira/browse/KAFKA-4102
> Project: Kafka
>  Issue Type: Bug
>Reporter: yf
>
> I want this feature 
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement)
>  in kafka new version.
> But it seems can't support kafka 0.8.X ?
> Cmd: ./kafka_2.11-0.10.0.1/bin/kafka-mirror-maker.sh --producer.config 
> svc_run/mirror_kafka_write_online2streaming/producer.properties 
> --consumer.config 
> svc_run/mirror_kafka_write_online2streaming/consumer.properties --whitelist 
> sandbox
> Producer config:
> queue.buffering.max.messages=16384
> bootstrap.servers=XXX
> send.buffer.bytes=131072
> message.send.max.retries=1048576
> consumer config:
> zookeeper.connect=
> group.id=test_mirror
> consumer.timeout.ms=-1
> zookeeper.connection.timeout.ms=6
> zookeeper.session.timeout.ms=6
> socket.receive.buffer.bytes=-1
> auto.commit.interval.ms=5000
> auto.commit.enable=false
> But I got failures like follows:
> FetchRequest@33437902 (kafka.consumer.ConsumerFetcherThread)
> java.lang.IllegalArgumentException
>   at java.nio.Buffer.limit(Buffer.java:275)
>   at 
> kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
>   at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
>   at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
>   at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
>   at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at 
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
>   at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
>   at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
>   at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
>   at 
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
>   at 
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
>   at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
>   at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4102) Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-4102:
---
Fix Version/s: (was: 0.10.1.0)
   0.10.2.0

> Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?
> -
>
> Key: KAFKA-4102
> URL: https://issues.apache.org/jira/browse/KAFKA-4102
> Project: Kafka
>  Issue Type: Bug
>Reporter: yf
> Fix For: 0.10.2.0
>
>
> I want this feature 
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement)
>  in kafka new version.
> But it seems can't support kafka 0.8.X ?
> Cmd: ./kafka_2.11-0.10.0.1/bin/kafka-mirror-maker.sh --producer.config 
> svc_run/mirror_kafka_write_online2streaming/producer.properties 
> --consumer.config 
> svc_run/mirror_kafka_write_online2streaming/consumer.properties --whitelist 
> sandbox
> Producer config:
> queue.buffering.max.messages=16384
> bootstrap.servers=XXX
> send.buffer.bytes=131072
> message.send.max.retries=1048576
> consumer config:
> zookeeper.connect=
> group.id=test_mirror
> consumer.timeout.ms=-1
> zookeeper.connection.timeout.ms=6
> zookeeper.session.timeout.ms=6
> socket.receive.buffer.bytes=-1
> auto.commit.interval.ms=5000
> auto.commit.enable=false
> But I got failures like follows:
> FetchRequest@33437902 (kafka.consumer.ConsumerFetcherThread)
> java.lang.IllegalArgumentException
>   at java.nio.Buffer.limit(Buffer.java:275)
>   at 
> kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
>   at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
>   at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
>   at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
>   at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at 
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
>   at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
>   at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
>   at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
>   at 
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
>   at 
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
>   at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
>   at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4102) Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?

[yf (JIRA)]

)
at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)


> Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?
> -
>
> Key: KAFKA-4102
> URL: https://issues.apache.org/jira/browse/KAFKA-4102
> Project: Kafka
>  Issue Type: Bug
>Reporter: yf
> Fix For: 0.10.1.0
>
>
> I want this feature 
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement)
>  in kafka new version.
> But it seems can't support kafka 0.8.X ?
> Cmd: ./kafka_2.11-0.10.0.1/bin/kafka-mirror-maker.sh --producer.config 
> svc_run/mirror_kafka_write_online2streaming/producer.properties 
> --consumer.config 
> svc_run/mirror_kafka_write_online2streaming/consumer.properties --whitelist 
> sandbox
> Producer config:
> queue.buffering.max.messages=16384
> bootstrap.servers=XXX
> send.buffer.bytes=131072
> message.send.max.retries=1048576
> consumer config:
> zookeeper.connect=
> group.id=test_mirror
> consumer.timeout.ms=-1
> zookeeper.connection.timeout.ms=6
> zookeeper.session.timeout.ms=6
> socket.receive.buffer.bytes=-1
> auto.commit.interval.ms=5000
> auto.commit.enable=false
> But I got failures like follows:
> FetchRequest@33437902 (kafka.consumer.ConsumerFetcherThread)
> java.lang.IllegalArgumentException
>   at java.nio.Buffer.limit(Buffer.java:275)
>   at 
> kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
>   at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
>   at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
>   at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
>   at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at 
> scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
>   at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
>   at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
>   at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
>   at 
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
>   at 
> kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
>   at 
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
>   at 
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
>   at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4102) Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?

[yf (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yf updated KAFKA-4102:
--
Description: 
I want this feature 
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement)
 in kafka new version.
But it seems can't support kafka 0.8.X ?

FetchRequest@33437902 (kafka.consumer.ConsumerFetcherThread)
java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:275)
at 
kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at 
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
at 
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
at 
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)

  was:
FetchRequest@33437902 (kafka.consumer.ConsumerFetcherThread)
java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:275)
at 
kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at 
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
at 
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
at 
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)


> Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?
> -
>
> Key: KAFKA-4102
> URL: https://issues.apache.org/jira/browse/KAFKA-4102
> Project: Kafka
>  Issue Type: Bug
>Reporter: yf
> Fix For: 0.10.1.0
>
>
> I want this feature 
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-3+-+Mirror+Maker+Enhancement)
>  in kafka new version.
> But it seems can't support kafka 0.8.X ?
> FetchReques

[jira] [Created] (KAFKA-4102) Is kafka mirror maker in 0.10.X don't support mirror kafka 0.8.X?

[yf (JIRA)]

yf created KAFKA-4102:
-

 Summary: Is kafka mirror maker in 0.10.X don't support mirror 
kafka 0.8.X?
 Key: KAFKA-4102
 URL: https://issues.apache.org/jira/browse/KAFKA-4102
 Project: Kafka
  Issue Type: Bug
Reporter: yf
 Fix For: 0.10.1.0


FetchRequest@33437902 (kafka.consumer.ConsumerFetcherThread)
java.lang.IllegalArgumentException
at java.nio.Buffer.limit(Buffer.java:275)
at 
kafka.api.FetchResponsePartitionData$.readFrom(FetchResponse.scala:38)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:100)
at kafka.api.TopicData$$anonfun$1.apply(FetchResponse.scala:98)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at kafka.api.TopicData$.readFrom(FetchResponse.scala:98)
at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:170)
at kafka.api.FetchResponse$$anonfun$4.apply(FetchResponse.scala:169)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at 
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.Range.foreach(Range.scala:160)
at 
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
at kafka.api.FetchResponse$.readFrom(FetchResponse.scala:169)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:135)
at 
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:108)
at 
kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:29)
at 
kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
at 
kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Kafka Mirror maker duplicate issue

[Ghosh, Achintya (Contractor)]

Hi there,

I created a broker as stand by using Kafka Mirror maker but same messages gets 
consumed by both Source broker and mirror broker.

Ex:

I send 1000 messages let's say offset value 1 to 1000 and consumed  500 
messages from the source broker. Now my broker goes down and want to read rest 
of the 500 messages  from Mirror broker.
But Mirror broker again gives all 1000 messages what I expected rest 500(offset 
value 501 to 1000). So there is no sync between the source and mirror broker.

Any idea what settings we have to do to solve the issue.

Thanks

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401136#comment-15401136
 ] 

ASF GitHub Bot commented on KAFKA-3950:
---

Github user omkreddy closed the pull request at:

https://github.com/apache/kafka/pull/1687


> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
> Fix For: 0.10.0.1
>
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1687: KAFKA-3950: kafka mirror maker tool is not respect...

[omkreddy]

Github user omkreddy closed the pull request at:

https://github.com/apache/kafka/pull/1687


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401069#comment-15401069
 ] 

Ismael Juma commented on KAFKA-3950:


Note that the fix was only included in the 0.10.0 branch. In trunk, we'll 
pursue KAFKA-3396 instead.

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
> Fix For: 0.10.0.1
>
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3950.

Resolution: Fixed

Issue resolved by pull request 1687
[https://github.com/apache/kafka/pull/1687]

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
> Fix For: 0.10.0.1
>
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-3950:
---
Affects Version/s: (was: 0.10.0.0)
   0.9.0.0

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
> Fix For: 0.10.0.1
>
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-3950:
---
 Reviewer: Ismael Juma
Fix Version/s: 0.10.0.1

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
> Fix For: 0.10.0.1
>
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-3950:
---
Affects Version/s: 0.10.0.0

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
> Fix For: 0.10.0.1
>
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400428#comment-15400428
 ] 

ASF GitHub Bot commented on KAFKA-3950:
---

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/1687

KAFKA-3950: kafka mirror maker tool is not respecting whitelist option



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka KAFKA-3950

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1687.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1687


commit f43d29418fbfb79d40acbda7ad2ec18dd57bcac7
Author: Manikumar Reddy O <manikumar.re...@gmail.com>
Date:   2016-07-29T17:39:59Z

KAFKA-3950: kafka mirror maker tool is not respecting whitelist option

----


> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1687: KAFKA-3950: kafka mirror maker tool is not respect...

[omkreddy]

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/1687

KAFKA-3950: kafka mirror maker tool is not respecting whitelist option



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka KAFKA-3950

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1687.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1687


commit f43d29418fbfb79d40acbda7ad2ec18dd57bcac7
Author: Manikumar Reddy O <manikumar.re...@gmail.com>
Date:   2016-07-29T17:39:59Z

KAFKA-3950: kafka mirror maker tool is not respecting whitelist option




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400419#comment-15400419
 ] 

ASF GitHub Bot commented on KAFKA-3950:
---

Github user omkreddy closed the pull request at:

https://github.com/apache/kafka/pull/1615


> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1615: KAFKA-3950: kafka mirror maker tool is not respect...

[omkreddy]

Github user omkreddy closed the pull request at:

https://github.com/apache/kafka/pull/1615


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399444#comment-15399444
 ] 

Manikumar Reddy commented on KAFKA-3950:


[~ijuma] Can we include this in 0.10.0.1 release?

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373307#comment-15373307
 ] 

ASF GitHub Bot commented on KAFKA-3950:
---

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/1615

KAFKA-3950: kafka mirror maker tool is not respecting whitelist option



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka KAFKA-3950-CONSUMER-PATTREN

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1615


commit f2a6e806c8c3d181dc3d16ab90548bdffde404af
Author: Manikumar Reddy O <manikumar.re...@gmail.com>
Date:   2016-07-12T17:29:46Z

KAFKA-3950: kafka mirror maker tool is not respecting whitelist option

----


> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1615: KAFKA-3950: kafka mirror maker tool is not respect...

[omkreddy]

GitHub user omkreddy opened a pull request:

https://github.com/apache/kafka/pull/1615

KAFKA-3950: kafka mirror maker tool is not respecting whitelist option



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/omkreddy/kafka KAFKA-3950-CONSUMER-PATTREN

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1615.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1615


commit f2a6e806c8c3d181dc3d16ab90548bdffde404af
Author: Manikumar Reddy O <manikumar.re...@gmail.com>
Date:   2016-07-12T17:29:46Z

KAFKA-3950: kafka mirror maker tool is not respecting whitelist option




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373115#comment-15373115
 ] 

Manikumar Reddy commented on KAFKA-3950:


I was also thinking similar lines. Instead of removing the check, we can move 
it to else block and have separate check for pattern subscriptions (check below 
pseudo code).  with this, we can still  still throw error for  topics that 
meant to include in your regex and ignore other unauthorized topics.

I have a patch. I will test and raise PR.

{code}
if (subscriptions.hasPatternSubscription()) {

for (String topic : cluster.unauthorizedTopics()) {
if 
(subscriptions.getSubscribedPattern().matcher(topic).matches())
throw new TopicAuthorizationException(topic);
}

   .
   .
} else if (!cluster.unauthorizedTopics().isEmpty()) {
throw new TopicAuthorizationException(new 
HashSet<>(cluster.unauthorizedTopics()));
}
{code}

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372991#comment-15372991
 ] 

Ismael Juma commented on KAFKA-3950:


My point is that if you are missing the authorization for a topic that you 
meant to include in your regex, you won't get an error. I think that's OK, but 
something that is important to call out.

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[TAO XIAO (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372982#comment-15372982
 ] 

TAO XIAO commented on KAFKA-3950:
-

As only the topics that the users have permission to read are returned I think 
we can consider this is a authorization success. And we can throw exception if 
none of topics matched.

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372740#comment-15372740
 ] 

Ismael Juma edited comment on KAFKA-3950 at 7/12/16 11:39 AM:
--

[~xiaotao183], that's a good point, we can perhaps change the semantics so that 
we apply the regex over the authorized topics only. The downside is that 
authorization errors will be harder to detect, but it seems to make sense.


was (Author: ijuma):
[~xiaotao183], that's a good point, we can perhaps change the semantics so that 
we apply the regex over the authorized topics only.

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372740#comment-15372740
 ] 

Ismael Juma commented on KAFKA-3950:


[~xiaotao183], that's a good point, we can perhaps change the semantics so that 
we apply the regex over the authorized topics only.

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[TAO XIAO (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372705#comment-15372705
 ] 

TAO XIAO commented on KAFKA-3950:
-

How about we still keep the filtering on client side and fix the broken piece?

Here is the trouble maker 
[ConsumerCoordinator.java|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L150]
 that fails the validation. Can we remove below check? 

{code}
if (!cluster.unauthorizedTopics().isEmpty())
throw new TopicAuthorizationException(new 
HashSet<>(cluster.unauthorizedTopics()));
{code}

As the authorization check has been done on server-side already when fetching 
metadata all topics stored in {code}cluster.topics(){code} should the ones the 
consumer has permission to read. We can simply return them that matches the 
pattern to end user

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[TAO XIAO (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372705#comment-15372705
 ] 

TAO XIAO edited comment on KAFKA-3950 at 7/12/16 11:18 AM:
---

How about we still keep the filtering on client side and fix the broken piece?

Here is the trouble maker 
[ConsumerCoordinator.java|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L150]
 that fails the validation. Can we remove below check? 

{code}
if (!cluster.unauthorizedTopics().isEmpty())
throw new TopicAuthorizationException(new 
HashSet<>(cluster.unauthorizedTopics()));
{code}

As the authorization check has been done on server-side already when fetching 
metadata all topics stored in {code}cluster.topics(){code} should be the ones 
the consumer has permission to read. We can simply return them that matches the 
pattern to end user


was (Author: xiaotao183):
How about we still keep the filtering on client side and fix the broken piece?

Here is the trouble maker 
[ConsumerCoordinator.java|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java#L150]
 that fails the validation. Can we remove below check? 

{code}
if (!cluster.unauthorizedTopics().isEmpty())
throw new TopicAuthorizationException(new 
HashSet<>(cluster.unauthorizedTopics()));
{code}

As the authorization check has been done on server-side already when fetching 
metadata all topics stored in {code}cluster.topics(){code} should the ones the 
consumer has permission to read. We can simply return them that matches the 
pattern to end user

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372516#comment-15372516
 ] 

Ismael Juma commented on KAFKA-3950:


[~omkreddy], yes, it's a current limitation that if you use pattern 
subscriptions, you need to have describe permission for all topics. There are 
many advantages in doing the filtering server-side, but that's a bit tricky 
because the protocol-level regex would have to work across languages. One 
option is to use a perl regex library in Java as that is probably the closest 
one would have to a standard. In any case, it's a significant change which 
would require a KIP and quite a bit of discussion.

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-3950:
---

consumer.subscribe(Pattern p , ..) method is failing when authorizer is enabled.
irrespective of pattern supplied, Consumer.subscribe(Pattren p, ..) method is 
always fetches metadata of all the topics(including internal topics) available 
in the cluster.
Since consumer is trying to fetch all the topics available in the cluster, it 
will fail for internal topics and topics which does not have describe 
permission.
Even if we give permission to all the topics which are matching the pattern, 
consumer.subscribe(Pattern p,..) will fail by complaining about internal topics
and other uninterested topics.

Current Consumer.subscribe(Pattern p,..) method implementation is fetching all 
the topic metadata and then it is applying the
pattern matching on client side. One possible solution is, send the pattern 
string to server and fetch only required topic metadata


[~ijuma] [~hachikuji]] I would like to know your views on this issue.

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Sriharsha Chintalapani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sriharsha Chintalapani updated KAFKA-3950:
--
Assignee: Manikumar Reddy

> kafka mirror maker tool is not respecting whitelist option
> --
>
> Key: KAFKA-3950
> URL: https://issues.apache.org/jira/browse/KAFKA-3950
> Project: Kafka
>  Issue Type: Bug
>Reporter: Raghav Kumar Gautam
>Assignee: Manikumar Reddy
>Priority: Critical
>
> A mirror maker launched like this:
> {code}
> /usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
> kfkt...@example.com
> JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
> /usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
>  --producer.config 
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
>  --new.consumer --whitelist="test.*" >>  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
>  2>&1 & echo pid:$! >  
> /usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
> {code}
> Lead to TopicAuthorizationException:
> {code}
> WARN Error while fetching metadata with correlation id 44 : 
> {__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
> (org.apache.kafka.clients.NetworkClient)
> [2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
> failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
> access topics: [__consumer_offsets]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3950) kafka mirror maker tool is not respecting whitelist option

[Raghav Kumar Gautam (JIRA)]

Raghav Kumar Gautam created KAFKA-3950:
--

 Summary: kafka mirror maker tool is not respecting whitelist option
 Key: KAFKA-3950
 URL: https://issues.apache.org/jira/browse/KAFKA-3950
 Project: Kafka
  Issue Type: Bug
Reporter: Raghav Kumar Gautam
Priority: Critical


A mirror maker launched like this:
{code}
/usr/bin/kinit -k -t /home/kfktest/hadoopqa/keytabs/kfktest.headless.keytab 
kfkt...@example.com
JAVA_HOME=/usr/jdk64/jdk1.8.0_77 JMX_PORT=9112 
/usr/kafka/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config 
/usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_consumer_12.properties
 --producer.config 
/usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/config/mirror_producer_12.properties
 --new.consumer --whitelist="test.*" >>  
/usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/mirror_maker_12.log
 2>&1 & echo pid:$! >  
/usr/kafka/system_test/mirror_maker_testsuite/testcase_15001/logs/mirror_maker-12/entity_12_pid
{code}
Lead to TopicAuthorizationException:
{code}
WARN Error while fetching metadata with correlation id 44 : 
{__consumer_offsets=TOPIC_AUTHORIZATION_FAILED} 
(org.apache.kafka.clients.NetworkClient)
[2016-06-20 13:24:49,983] FATAL [mirrormaker-thread-0] Mirror maker thread 
failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to 
access topics: [__consumer_offsets]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3609) Implement partial mirroring functionality in Kafka Mirror Maker

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253761#comment-15253761
 ] 

Manikumar Reddy commented on KAFKA-3609:


Current Mirror Maker supports partial mirroring of topics by using --whitelist 
config property. 

> Implement partial mirroring functionality in Kafka Mirror Maker 
> 
>
> Key: KAFKA-3609
> URL: https://issues.apache.org/jira/browse/KAFKA-3609
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Sukhdev Saini
>
> It will be useful to implement partial or subset of partitions mirroring 
> functionality in Kafka Mirror Maker. 
> This will be useful in number of scenarios:
> 1. Where business hosts two separate AWS clusters and mirroring everything 
> might be quite expensive.
> 2. Mirror cluster is maintaining some kind of local state associated with 
> subset of partitions (like a local on-disk key-value store) and hence it 
> should only mirror partitions it is maintaining on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3609) Implement partial mirroring functionality in Kafka Mirror Maker

[Sukhdev Saini (JIRA)]

Sukhdev Saini created KAFKA-3609:


 Summary: Implement partial mirroring functionality in Kafka Mirror 
Maker 
 Key: KAFKA-3609
 URL: https://issues.apache.org/jira/browse/KAFKA-3609
 Project: Kafka
  Issue Type: New Feature
  Components: core
Affects Versions: 0.9.0.0
Reporter: Sukhdev Saini


It will be useful to implement partial or subset of partitions mirroring 
functionality in Kafka Mirror Maker. 
This will be useful in number of scenarios:
1. Where business hosts two separate AWS clusters and mirroring everything 
might be quite expensive.
2. Mirror cluster is maintaining some kind of local state associated with 
subset of partitions (like a local on-disk key-value store) and hence it should 
only mirror partitions it is maintaining on disk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3327) Warning from kafka mirror maker about ssl properties not valid

[Munir Khan (JIRA)]

Munir Khan created KAFKA-3327:
-

 Summary: Warning from kafka mirror maker about ssl properties not 
valid
 Key: KAFKA-3327
 URL: https://issues.apache.org/jira/browse/KAFKA-3327
 Project: Kafka
  Issue Type: Test
  Components: config
Affects Versions: 0.9.0.1
 Environment: CentOS release 6.5
Reporter: Munir Khan
Priority: Minor


I am trying to run Mirror maker  over SSL. I have configured my broker 
following the procedure described in this document 
http://kafka.apache.org/documentation.html#security_overview 

I get the following warning when I start the mirror maker:

[root@munkhan-kafka1 kafka_2.10-0.9.0.1]# bin/kafka-run-class.sh 
kafka.tools.MirrorMaker --consumer.config 
config/datapush-consumer-ssl.properties --producer.config 
config/datapush-producer-ssl.properties --num.streams 2 --whitelist test1&
[1] 4701
[root@munkhan-kafka1 kafka_2.10-0.9.0.1]# [2016-03-03 10:24:35,348] WARN 
block.on.buffer.full config is deprecated and will be removed soon. Please use 
max.block.ms (org.apache.kafka.clients.producer.KafkaProducer)
[2016-03-03 10:24:35,523] WARN The configuration producer.type = sync was 
supplied but isn't a known config. 
(org.apache.kafka.clients.producer.ProducerConfig)
[2016-03-03 10:24:35,523] WARN The configuration ssl.keypassword = test1234 was 
supplied but isn't a known config. 
(org.apache.kafka.clients.producer.ProducerConfig)
[2016-03-03 10:24:35,523] WARN The configuration compression.codec = none was 
supplied but isn't a known config. 
(org.apache.kafka.clients.producer.ProducerConfig)
[2016-03-03 10:24:35,523] WARN The configuration serializer.class = 
kafka.serializer.DefaultEncoder was supplied but isn't a known config. 
(org.apache.kafka.clients.producer.ProducerConfig)
[2016-03-03 10:24:35,617] WARN Property security.protocol is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,617] WARN Property ssl.keypassword is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,617] WARN Property ssl.keystore.location is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,618] WARN Property ssl.keystore.password is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,618] WARN Property ssl.truststore.location is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,618] WARN Property ssl.truststore.password is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,752] WARN Property security.protocol is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,752] WARN Property ssl.keypassword is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,752] WARN Property ssl.keystore.location is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,752] WARN Property ssl.keystore.password is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,752] WARN Property ssl.truststore.location is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:35,753] WARN Property ssl.truststore.password is not valid 
(kafka.utils.VerifiableProperties)
[2016-03-03 10:24:36,251] WARN No broker partitions consumed by consumer thread 
test-consumer-group_munkhan-kafka1.cisco.com-1457018675755-b9bb4c75-0 for topic 
test1 (kafka.consumer.RangeAssignor)
[2016-03-03 10:24:36,251] WARN No broker partitions consumed by consumer thread 
test-consumer-group_munkhan-kafka1.cisco.com-1457018675755-b9bb4c75-0 for topic 
test1 (kafka.consumer.RangeAssignor)

However the Mirror maker is able to mirror data . If I remove the 
configurations related to the warning messages from my producer  mirror maker 
does not work . So it seems despite the warning shown above the 
ssl.configuration properties are used somehow. 

My question is these are those warnings harmless in this context ?





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: KAfka Mirror Maker

[Jiangjie Qin]

Mirror Maker does not have specific restrictions on cluster size.

The error you saw was because consumer was not able to talk to the broker.
Can you try to use kafka-console-consumer to consume some data from your
source cluster and see if it works? It should be under KAFKA_HOME/bin/

Jiangjie (Becket) Qin

On Tue, Jul 28, 2015 at 11:01 AM, Prabhjot Bharaj prabhbha...@gmail.com
wrote:

 Hi,

 I'm using Mirror Maker with a cluster of 3 nodes and cluster of 5 nodes.

 I would like to ask - is the number of nodes a restriction for Mirror
 Maker?
 Also, are there any other restrictions or properties that should be common
 across both the clusters so that they continue mirroring.


 I'm asking this because I've got this error while mirroring:-

 [2015-07-28 17:51:10,943] WARN Fetching topic metadata with correlation id
 0 for topics [Set(fromIndiaWithLove)] from broker
 [id:3,host:a10.2.3.4,port:9092] failed (kafka.client.ClientUtils$)

 java.nio.channels.ClosedChannelException

 at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)

 at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)

 at

 kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)

 at kafka.producer.SyncProducer.send(SyncProducer.scala:113)

 at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)

 at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)

 at

 kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)

 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

 [2015-07-28 17:51:18,955] WARN Fetching topic metadata with correlation id
 0 for topics [Set(fromIndiaWithLove)] from broker
 [id:2,host:10.2.3.5,port:9092] failed (kafka.client.ClientUtils$)

 java.nio.channels.ClosedChannelException

 at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)

 at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)

 at

 kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)

 at kafka.producer.SyncProducer.send(SyncProducer.scala:113)

 at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)

 at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)

 at

 kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)

 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

 [2015-07-28 17:51:27,043] WARN Fetching topic metadata with correlation id
 0 for topics [Set(fromIndiaWithLove)] from broker
 [id:5,host:a10.2.3.6port:9092] failed (kafka.client.ClientUtils$)

 java.nio.channels.ClosedChannelException

 at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)

 at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)

 at

 kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)

 at kafka.producer.SyncProducer.send(SyncProducer.scala:113)

 at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)

 at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)

 at

 kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)

 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)



 This is what my *consumer config* looks like:-

 *zookeeper.connect=10.2.3.4:2182 http://10.2.3.4:2182*

 *zookeeper.connection.timeout.ms
 http://zookeeper.connection.timeout.ms=100*

 *consumer.timeout.ms http://consumer.timeout.ms=-1*

 *group.id http://group.id=dp-mirrorMaker-test-datap1*

 *shallow.iterator.enable=true*

 *auto.create.topics.enable=true*



 I've used the default* producer.properties* in kafka/config/ which has
 these properteis:-

 *metadata.broker.list=localhost:9092*


 *producer.type=sync*

 *compression.codec=none*


 *serializer.class=kafka.serializer.DefaultEncoder*


 I'm running Mirror Maker via this command:-


  /kafka_2.10-0.8.2.0/bin/kafka-run-class.sh kafka.tools.MirrorMaker
 --consumer.config ~/sourceCluster1Consumer.config  --num.streams 1
 --producer.config producer.properties --whitelist=.*

 Regards,

 prabcs

KAfka Mirror Maker

[Prabhjot Bharaj]

Hi,

I'm using Mirror Maker with a cluster of 3 nodes and cluster of 5 nodes.

I would like to ask - is the number of nodes a restriction for Mirror Maker?
Also, are there any other restrictions or properties that should be common
across both the clusters so that they continue mirroring.


I'm asking this because I've got this error while mirroring:-

[2015-07-28 17:51:10,943] WARN Fetching topic metadata with correlation id
0 for topics [Set(fromIndiaWithLove)] from broker
[id:3,host:a10.2.3.4,port:9092] failed (kafka.client.ClientUtils$)

java.nio.channels.ClosedChannelException

at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)

at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)

at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)

at kafka.producer.SyncProducer.send(SyncProducer.scala:113)

at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)

at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)

at
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

[2015-07-28 17:51:18,955] WARN Fetching topic metadata with correlation id
0 for topics [Set(fromIndiaWithLove)] from broker
[id:2,host:10.2.3.5,port:9092] failed (kafka.client.ClientUtils$)

java.nio.channels.ClosedChannelException

at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)

at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)

at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)

at kafka.producer.SyncProducer.send(SyncProducer.scala:113)

at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)

at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)

at
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)

[2015-07-28 17:51:27,043] WARN Fetching topic metadata with correlation id
0 for topics [Set(fromIndiaWithLove)] from broker
[id:5,host:a10.2.3.6port:9092] failed (kafka.client.ClientUtils$)

java.nio.channels.ClosedChannelException

at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)

at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)

at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)

at kafka.producer.SyncProducer.send(SyncProducer.scala:113)

at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)

at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93)

at
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)



This is what my *consumer config* looks like:-

*zookeeper.connect=10.2.3.4:2182 http://10.2.3.4:2182*

*zookeeper.connection.timeout.ms
http://zookeeper.connection.timeout.ms=100*

*consumer.timeout.ms http://consumer.timeout.ms=-1*

*group.id http://group.id=dp-mirrorMaker-test-datap1*

*shallow.iterator.enable=true*

*auto.create.topics.enable=true*



I've used the default* producer.properties* in kafka/config/ which has
these properteis:-

*metadata.broker.list=localhost:9092*


*producer.type=sync*

*compression.codec=none*


*serializer.class=kafka.serializer.DefaultEncoder*


I'm running Mirror Maker via this command:-


 /kafka_2.10-0.8.2.0/bin/kafka-run-class.sh kafka.tools.MirrorMaker
--consumer.config ~/sourceCluster1Consumer.config  --num.streams 1
--producer.config producer.properties --whitelist=.*

Regards,

prabcs