Re: Number of partitions for offsets topic cannot be changed

[Peter Bukowinski]

Once the offsets topic has been created, changing any server.properties 
affecting it will have no effect. If you are pre-production, you can start over 
using the new server properties. 

By start over, I mean stopping brokers, purging the kafka logs/data directories 
of all brokers, purging cluster data from zookeepers, then starting brokers 
again with your desired server properties settings.

-- Peter Bukowinski

> On Mar 25, 2018, at 1:01 PM, Anu P  wrote:
> 
> Thanks Swapnil.
> 
> I changed *offsets.topic.num.partitions* in server.properties file and
> restarted the broker. But, still this config change does not take effect.
> 
> Any ideas?
> 
> Thanks in advance
> Tanvi
> 
> On Sun, Mar 25, 2018 at 12:30 AM, Swapnil Gupta 
> wrote:
> 
>> In brief, this is system level configuration by Kafka.
>> 
>> Consumer offsets partitions can't be changed through command line.
>> 
>> You have to change the configuration file and set this
>> *offsets.topic.num.partitions* property to change this.
>> 
>>> On Sun, Mar 25, 2018, 12:49 Anu P  wrote:
>>> 
>>> Hi,
>>> 
>>> I am trying to change the number of partitions for __consumer_offsets
>> topic
>>> by using the following command. However, I get an error stating "Number
>> of
>>> partitions for offsets topic cannot be changed"
>>> 
>>> 
>>> */opt/kafka/bin/kafka-topics.sh --zookeeper  --alter --topic
>>> __consumer_offsets --partitions 54   *
>>> 
>>> 
>>> Error while executing topic command : The number of partitions for the
>>> offsets topic cannot be changed.
>>> ERROR java.lang.IllegalArgumentException: The number of partitions for
>> the
>>> offsets topic cannot be changed.
>>> at
>>> 
>>> kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(
>> TopicCommand.scala:142)
>>> .
>>> 
>>> 
>>> 
>>> Environment details:
>>> 
>>>   1. 3 zookeepers and 3 kafka brokers (version 1.0.1)
>>>   2. After deploying kafka, I tried the kafka-topics command (above) to
>>>   increase it partitions. But it fails. Can someone please help me
>>> understand
>>>   why? Am I missing something?
>>>   3. I want to distribute partitions leaders equally among brokers. So
>>>   want to increase the number of partitions for offsets topic.
>>> 
>>> 
>>> Thanks in advance.
>>> 
>>> Tanvi
>>> 
>> 

Re: Number of partitions for offsets topic cannot be changed

[Anu P]

Thanks Swapnil.

I changed *offsets.topic.num.partitions* in server.properties file and
restarted the broker. But, still this config change does not take effect.

Any ideas?

Thanks in advance
Tanvi

On Sun, Mar 25, 2018 at 12:30 AM, Swapnil Gupta 
wrote:

> In brief, this is system level configuration by Kafka.
>
> Consumer offsets partitions can't be changed through command line.
>
> You have to change the configuration file and set this
> *offsets.topic.num.partitions* property to change this.
>
> On Sun, Mar 25, 2018, 12:49 Anu P  wrote:
>
> > Hi,
> >
> > I am trying to change the number of partitions for __consumer_offsets
> topic
> > by using the following command. However, I get an error stating "Number
> of
> > partitions for offsets topic cannot be changed"
> >
> >
> > */opt/kafka/bin/kafka-topics.sh --zookeeper  --alter --topic
> > __consumer_offsets --partitions 54   *
> >
> >
> > Error while executing topic command : The number of partitions for the
> > offsets topic cannot be changed.
> > ERROR java.lang.IllegalArgumentException: The number of partitions for
> the
> > offsets topic cannot be changed.
> > at
> >
> > kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(
> TopicCommand.scala:142)
> > .
> >
> >
> >
> > Environment details:
> >
> >1. 3 zookeepers and 3 kafka brokers (version 1.0.1)
> >2. After deploying kafka, I tried the kafka-topics command (above) to
> >increase it partitions. But it fails. Can someone please help me
> > understand
> >why? Am I missing something?
> >3. I want to distribute partitions leaders equally among brokers. So
> >want to increase the number of partitions for offsets topic.
> >
> >
> > Thanks in advance.
> >
> > Tanvi
> >
>

Re: Developing with kafka and other non jvm languages

[Matthias J. Sax]

Gary,

The Apache Kafka project itself, only maintains Java clients -- thus,
your are right that those are the primary and best supported clients.
(Kafka used to have Scala clients, but those are all deprecated now and
will be removed eventually.)

Clients in other languages are not part of Apache Kafka project itself
but developed by third parties. Thus, the quality and feature set of
those clients can vary. A list of known third party clients is
maintained in the Kafka Wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Clients

If a client does not support a feature, it's a client limitation only.
The Kafka protocol is a binary protocol and thus it's up to the client
to (fully) implement the protocol -- the broker does not even know in
which language a client is written. (cf
https://kafka.apache.org/protocol) Thus, you should open feature
requests for clients to support pattern subscription if it's missing and
you need it.

If you need help to get librdkafka to compile on arm, you might want to
reach out to on Github (https://github.com/edenhill/librdkafka) or
Confluent mailing list
(https://groups.google.com/forum/#!forum/confluent-platform) or
Confluent Slack (https://launchpass.com/confluentcommunity).

Hope this helps!


-Matthias




On 3/25/18 12:46 AM, Gary Taylor wrote:
> Hi,
>   This is a fairly generic question but has some specifics too
> 
> Ill ask the specific first - I am trying to use golang to talk to kafka and 
> it works, but a fairly important part of my application is to subscribe to 
> information in many topics where the topic is matched server side and will 
> include new topics added since the subscribe.  I have read that kafka can do 
> this, but I cannot see any way of asking it to do it via golang apart from 
> with the library https://github.com/confluentinc/confluent-kafka-go 
>  - which uses a library 
> called librdkafka which I would have to compile and I am struggling as I am 
> using an arm processor (long story behind that one - but I don’t think this 
> solution will ever work with an arm processor).  All other libraries seem to 
> not offer this.  Is this because of some underlying restriction in kafka or 
> do you think it is going to be a case of keep hunting until I find one that 
> does ?
> 
> Then the more generic question - I get the feeling that Scala and Java are 
> first class citizens when it comes to using kafka.  Whilst I love Scala - I 
> am learning golang and am more generally a ruby developer.  Am I likely go 
> get a ‘second class’ service with these other languages - with restrictions 
> around zookeeper for example (I think I read something about having to have 
> knowledge of which partition to subscribe to - where zookeeper would normally 
> track this but there is no API for it) ?
> 
> Many Thanks
> 
> Gary Taylor
> 



signature.asc
Description: OpenPGP digital signature

Kafka Mirrormaker issue

[Siva A]

Hi,

We have 3 nodes Kafka cluster(0.10.0.1) and its mirroring the data from
another 3 node cluster of same Kafka version.
Both the clusters are Kerberized and we are running the Mirrormaker on the
target cluster using the single principal/keytab with the one way trust on
the KDC.

At times, the mirrormaker stops functioning(Doesn't mirror the data) but
the process is still running. If we restart the service then it works fine
for a day or so.

I don't see any error on the Kafka logs as well.
Is there anyone seen this kind of issue?

Thanks
Siva

Developing with kafka and other non jvm languages

[Gary Taylor]

Hi,
  This is a fairly generic question but has some specifics too

Ill ask the specific first - I am trying to use golang to talk to kafka and it 
works, but a fairly important part of my application is to subscribe to 
information in many topics where the topic is matched server side and will 
include new topics added since the subscribe.  I have read that kafka can do 
this, but I cannot see any way of asking it to do it via golang apart from with 
the library https://github.com/confluentinc/confluent-kafka-go 
 - which uses a library 
called librdkafka which I would have to compile and I am struggling as I am 
using an arm processor (long story behind that one - but I don’t think this 
solution will ever work with an arm processor).  All other libraries seem to 
not offer this.  Is this because of some underlying restriction in kafka or do 
you think it is going to be a case of keep hunting until I find one that does ?

Then the more generic question - I get the feeling that Scala and Java are 
first class citizens when it comes to using kafka.  Whilst I love Scala - I am 
learning golang and am more generally a ruby developer.  Am I likely go get a 
‘second class’ service with these other languages - with restrictions around 
zookeeper for example (I think I read something about having to have knowledge 
of which partition to subscribe to - where zookeeper would normally track this 
but there is no API for it) ?

Many Thanks

Gary Taylor

Re: Number of partitions for offsets topic cannot be changed

[Swapnil Gupta]

In brief, this is system level configuration by Kafka.

Consumer offsets partitions can't be changed through command line.

You have to change the configuration file and set this
*offsets.topic.num.partitions* property to change this.

On Sun, Mar 25, 2018, 12:49 Anu P  wrote:

> Hi,
>
> I am trying to change the number of partitions for __consumer_offsets topic
> by using the following command. However, I get an error stating "Number of
> partitions for offsets topic cannot be changed"
>
>
> */opt/kafka/bin/kafka-topics.sh --zookeeper  --alter --topic
> __consumer_offsets --partitions 54   *
>
>
> Error while executing topic command : The number of partitions for the
> offsets topic cannot be changed.
> ERROR java.lang.IllegalArgumentException: The number of partitions for the
> offsets topic cannot be changed.
> at
>
> kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:142)
> .
>
>
>
> Environment details:
>
>1. 3 zookeepers and 3 kafka brokers (version 1.0.1)
>2. After deploying kafka, I tried the kafka-topics command (above) to
>increase it partitions. But it fails. Can someone please help me
> understand
>why? Am I missing something?
>3. I want to distribute partitions leaders equally among brokers. So
>want to increase the number of partitions for offsets topic.
>
>
> Thanks in advance.
>
> Tanvi
>

Number of partitions for offsets topic cannot be changed

[Anu P]

Hi,

I am trying to change the number of partitions for __consumer_offsets topic
by using the following command. However, I get an error stating "Number of
partitions for offsets topic cannot be changed"


*/opt/kafka/bin/kafka-topics.sh --zookeeper  --alter --topic
__consumer_offsets --partitions 54   *


Error while executing topic command : The number of partitions for the
offsets topic cannot be changed.
ERROR java.lang.IllegalArgumentException: The number of partitions for the
offsets topic cannot be changed.
at
kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:142)
.



Environment details:

   1. 3 zookeepers and 3 kafka brokers (version 1.0.1)
   2. After deploying kafka, I tried the kafka-topics command (above) to
   increase it partitions. But it fails. Can someone please help me understand
   why? Am I missing something?
   3. I want to distribute partitions leaders equally among brokers. So
   want to increase the number of partitions for offsets topic.


Thanks in advance.

Tanvi