Re: Partitions reassignment is failing in Kafka 1.1.0
See KAFKA-7124
Cheers
On Sun, Jul 1, 2018 at 3:26 AM, Debraj Manna
wrote:
> Is there any bug filed for this ?
>
> On Sun, Jul 1, 2018 at 6:37 AM, Ted Yu wrote:
>
> > Code snippet from ReassignPartitionsCommand.scala :
> >
> > "log_dirs" -> replicas.map(r =>
> > replicaLogDirAssignment.getOrElse(new TopicPartitionReplica(tp.topic,
> > tp.partition, r), AnyLogDir)).asJava
> >
> > We know that the appearance of "any" was due to the OrElse clause.
> > Arguably there is a bug in the above code that the number of AnyLogDir
> > should match the length of the replicas list, or "log_dirs" should be
> > omitted in such case.
> >
> > On Sat, Jun 30, 2018 at 12:06 AM, Manikumar
> > wrote:
> >
> > > It will be taken as "any" directory for each replica, which means
> replica
> > > will placed on any one of the
> > > configured directory on that broker.
> > >
> > > Since it is "log_dirs" optional, you can remove from the json.
> > >
> > > On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna <
> subharaj.ma...@gmail.com>
> > > wrote:
> > >
> > > > It is problem on my side. The code was changing the replicas count
> but
> > > not
> > > > the log_dirs. Since I am migrating from 0.10 this part of the code
> was
> > > not
> > > > changed.
> > > >
> > > > I have a follow up question what is the default value of log_dirs if
> I
> > > > don't specify it in reassignment.json ?
> > > >
> > > > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <
> > subharaj.ma...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I am generating the reassignent.json like below
> > > > >
> > > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > --zookeeper
> > > > 127.0.0.1:2181 --generate --topics-to-move-json-file
> > > > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s'
> |tail
> > > -1 >
> > > > /home/ubuntu/deploy/kafka/reassignment.json"
> > > > >
> > > > > Then I am doing the reassignment using the generated file
> > > > >
> > > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > --zookeeper
> > > > 127.0.0.1:2181 --execute --reassignment-json-file
> > > > /home/ubuntu/deploy/kafka/reassignment.json
> > > > >
> > > > > kafka-reassign-partitions.sh helps states
> > > > >
> > > > > The JSON file with the partition reassignment configurationThe
> format
> > > to
> > > > >> use is -
> > > > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas":
> [1,2,3],
> > > > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that
> > > "log_dirs"
> > > > is
> > > > >> optional. When it is specified, its length must equal the length
> of
> > > the
> > > > >> replicas list. The value in this list can be either "any" or the
> > > > absolution
> > > > >> path of the log directory on the broker. If absolute log directory
> > > path
> > > > is
> > > > >> specified, it is currently required that the replica has not
> already
> > > > been
> > > > >> created on that broker. The replica will then be created in the
> > > > specified
> > > > >> log directory on the broker later.
> > > > >
> > > > >
> > > > > So it appears reassignment json that is generated by
> > > > > kafka-reassign-partions.sh is creating an issue with logdirs. Is
> this
> > > > > some issue in kafka-reassign-partitions.sh or some misconfiguration
> > > from
> > > > my
> > > > > side. ?
> > > > >
> > > > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <
> > > subharaj.ma...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Please find the server.properties from one of the broker.
> > > > >>
> > > > >> broker.id=0
> > > > >> port=9092
> > > > >> num.network.threads=3
> > > > >> num.io.threads=8
> > > > >> socket.send.buffer.bytes=102400
> > > > >> socket.receive.buffer.bytes=102400
> > > > >> socket.request.max.bytes=104857600
> > > > >> log.dirs=/var/lib/kafka/kafka-logs
> > > > >> num.recovery.threads.per.data.dir=1
> > > > >> log.retention.hours=36
> > > > >> log.retention.bytes=1073741824
> > > > >> log.segment.bytes=536870912
> > > > >> log.retention.check.interval.ms=30
> > > > >> log.cleaner.enable=false
> > > > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> > > > >> message.max.bytes=1500
> > > > >> replica.fetch.max.bytes=1500
> > > > >> auto.create.topics.enable=true
> > > > >> zookeeper.connection.timeout.ms=6000
> > > > >> unclean.leader.election.enable=false
> > > > >> delete.topic.enable=false
> > > > >> offsets.topic.replication.factor=1
> > > > >> transaction.state.log.replication.factor=1
> > > > >> transaction.state.log.min.isr=1
> > > > >>
> > > > >> I have placed server.log from a broker at
> > https://gist.github.com/deb
> > > > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> > > > >>
> > > > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu
> > wrote:
> > > > >>
> > > > >>> Seems to be related to KIP-113.
> > > > >>>
> > > > >>> server.properties didn't go thru. Do you mind pastebin'ing its
> > > content
> > > > ?
> > > > >>>
> > > > >>> If you can pastebin logs from broker, th
Re: Partitions reassignment is failing in Kafka 1.1.0
Is there any bug filed for this ?
On Sun, Jul 1, 2018 at 6:37 AM, Ted Yu wrote:
> Code snippet from ReassignPartitionsCommand.scala :
>
> "log_dirs" -> replicas.map(r =>
> replicaLogDirAssignment.getOrElse(new TopicPartitionReplica(tp.topic,
> tp.partition, r), AnyLogDir)).asJava
>
> We know that the appearance of "any" was due to the OrElse clause.
> Arguably there is a bug in the above code that the number of AnyLogDir
> should match the length of the replicas list, or "log_dirs" should be
> omitted in such case.
>
> On Sat, Jun 30, 2018 at 12:06 AM, Manikumar
> wrote:
>
> > It will be taken as "any" directory for each replica, which means replica
> > will placed on any one of the
> > configured directory on that broker.
> >
> > Since it is "log_dirs" optional, you can remove from the json.
> >
> > On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna
> > wrote:
> >
> > > It is problem on my side. The code was changing the replicas count but
> > not
> > > the log_dirs. Since I am migrating from 0.10 this part of the code was
> > not
> > > changed.
> > >
> > > I have a follow up question what is the default value of log_dirs if I
> > > don't specify it in reassignment.json ?
> > >
> > > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna <
> subharaj.ma...@gmail.com
> > >
> > > wrote:
> > >
> > > > I am generating the reassignent.json like below
> > > >
> > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> --zookeeper
> > > 127.0.0.1:2181 --generate --topics-to-move-json-file
> > > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail
> > -1 >
> > > /home/ubuntu/deploy/kafka/reassignment.json"
> > > >
> > > > Then I am doing the reassignment using the generated file
> > > >
> > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> --zookeeper
> > > 127.0.0.1:2181 --execute --reassignment-json-file
> > > /home/ubuntu/deploy/kafka/reassignment.json
> > > >
> > > > kafka-reassign-partitions.sh helps states
> > > >
> > > > The JSON file with the partition reassignment configurationThe format
> > to
> > > >> use is -
> > > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> > > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that
> > "log_dirs"
> > > is
> > > >> optional. When it is specified, its length must equal the length of
> > the
> > > >> replicas list. The value in this list can be either "any" or the
> > > absolution
> > > >> path of the log directory on the broker. If absolute log directory
> > path
> > > is
> > > >> specified, it is currently required that the replica has not already
> > > been
> > > >> created on that broker. The replica will then be created in the
> > > specified
> > > >> log directory on the broker later.
> > > >
> > > >
> > > > So it appears reassignment json that is generated by
> > > > kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> > > > some issue in kafka-reassign-partitions.sh or some misconfiguration
> > from
> > > my
> > > > side. ?
> > > >
> > > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <
> > subharaj.ma...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Please find the server.properties from one of the broker.
> > > >>
> > > >> broker.id=0
> > > >> port=9092
> > > >> num.network.threads=3
> > > >> num.io.threads=8
> > > >> socket.send.buffer.bytes=102400
> > > >> socket.receive.buffer.bytes=102400
> > > >> socket.request.max.bytes=104857600
> > > >> log.dirs=/var/lib/kafka/kafka-logs
> > > >> num.recovery.threads.per.data.dir=1
> > > >> log.retention.hours=36
> > > >> log.retention.bytes=1073741824
> > > >> log.segment.bytes=536870912
> > > >> log.retention.check.interval.ms=30
> > > >> log.cleaner.enable=false
> > > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> > > >> message.max.bytes=1500
> > > >> replica.fetch.max.bytes=1500
> > > >> auto.create.topics.enable=true
> > > >> zookeeper.connection.timeout.ms=6000
> > > >> unclean.leader.election.enable=false
> > > >> delete.topic.enable=false
> > > >> offsets.topic.replication.factor=1
> > > >> transaction.state.log.replication.factor=1
> > > >> transaction.state.log.min.isr=1
> > > >>
> > > >> I have placed server.log from a broker at
> https://gist.github.com/deb
> > > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> > > >>
> > > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu
> wrote:
> > > >>
> > > >>> Seems to be related to KIP-113.
> > > >>>
> > > >>> server.properties didn't go thru. Do you mind pastebin'ing its
> > content
> > > ?
> > > >>>
> > > >>> If you can pastebin logs from broker, that should help.
> > > >>>
> > > >>> Thanks
> > > >>>
> > > >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> > > subharaj.ma...@gmail.com
> > > >>> >
> > > >>> wrote:
> > > >>>
> > > >>> > Hi
> > > >>> >
> > > >>> > I altered a topic like below in kafka 1.1.0
> > > >>> >
> > > >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> > > >>> 127.0.0.1:2181
> > > >>> > --alter
Re: Partitions reassignment is failing in Kafka 1.1.0
Code snippet from ReassignPartitionsCommand.scala :
"log_dirs" -> replicas.map(r =>
replicaLogDirAssignment.getOrElse(new TopicPartitionReplica(tp.topic,
tp.partition, r), AnyLogDir)).asJava
We know that the appearance of "any" was due to the OrElse clause.
Arguably there is a bug in the above code that the number of AnyLogDir
should match the length of the replicas list, or "log_dirs" should be
omitted in such case.
On Sat, Jun 30, 2018 at 12:06 AM, Manikumar
wrote:
> It will be taken as "any" directory for each replica, which means replica
> will placed on any one of the
> configured directory on that broker.
>
> Since it is "log_dirs" optional, you can remove from the json.
>
> On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna
> wrote:
>
> > It is problem on my side. The code was changing the replicas count but
> not
> > the log_dirs. Since I am migrating from 0.10 this part of the code was
> not
> > changed.
> >
> > I have a follow up question what is the default value of log_dirs if I
> > don't specify it in reassignment.json ?
> >
> > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna >
> > wrote:
> >
> > > I am generating the reassignent.json like below
> > >
> > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> > 127.0.0.1:2181 --generate --topics-to-move-json-file
> > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail
> -1 >
> > /home/ubuntu/deploy/kafka/reassignment.json"
> > >
> > > Then I am doing the reassignment using the generated file
> > >
> > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> > 127.0.0.1:2181 --execute --reassignment-json-file
> > /home/ubuntu/deploy/kafka/reassignment.json
> > >
> > > kafka-reassign-partitions.sh helps states
> > >
> > > The JSON file with the partition reassignment configurationThe format
> to
> > >> use is -
> > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that
> "log_dirs"
> > is
> > >> optional. When it is specified, its length must equal the length of
> the
> > >> replicas list. The value in this list can be either "any" or the
> > absolution
> > >> path of the log directory on the broker. If absolute log directory
> path
> > is
> > >> specified, it is currently required that the replica has not already
> > been
> > >> created on that broker. The replica will then be created in the
> > specified
> > >> log directory on the broker later.
> > >
> > >
> > > So it appears reassignment json that is generated by
> > > kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> > > some issue in kafka-reassign-partitions.sh or some misconfiguration
> from
> > my
> > > side. ?
> > >
> > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <
> subharaj.ma...@gmail.com
> > >
> > > wrote:
> > >
> > >> Please find the server.properties from one of the broker.
> > >>
> > >> broker.id=0
> > >> port=9092
> > >> num.network.threads=3
> > >> num.io.threads=8
> > >> socket.send.buffer.bytes=102400
> > >> socket.receive.buffer.bytes=102400
> > >> socket.request.max.bytes=104857600
> > >> log.dirs=/var/lib/kafka/kafka-logs
> > >> num.recovery.threads.per.data.dir=1
> > >> log.retention.hours=36
> > >> log.retention.bytes=1073741824
> > >> log.segment.bytes=536870912
> > >> log.retention.check.interval.ms=30
> > >> log.cleaner.enable=false
> > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> > >> message.max.bytes=1500
> > >> replica.fetch.max.bytes=1500
> > >> auto.create.topics.enable=true
> > >> zookeeper.connection.timeout.ms=6000
> > >> unclean.leader.election.enable=false
> > >> delete.topic.enable=false
> > >> offsets.topic.replication.factor=1
> > >> transaction.state.log.replication.factor=1
> > >> transaction.state.log.min.isr=1
> > >>
> > >> I have placed server.log from a broker at https://gist.github.com/deb
> > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> > >>
> > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu wrote:
> > >>
> > >>> Seems to be related to KIP-113.
> > >>>
> > >>> server.properties didn't go thru. Do you mind pastebin'ing its
> content
> > ?
> > >>>
> > >>> If you can pastebin logs from broker, that should help.
> > >>>
> > >>> Thanks
> > >>>
> > >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> > subharaj.ma...@gmail.com
> > >>> >
> > >>> wrote:
> > >>>
> > >>> > Hi
> > >>> >
> > >>> > I altered a topic like below in kafka 1.1.0
> > >>> >
> > >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> > >>> 127.0.0.1:2181
> > >>> > --alter --topic Topic3 --config min.insync.replicas=2
> > >>> >
> > >>> > But whenever I am trying to verify the reassignment it is showing
> the
> > >>> > below exception
> > >>> >
> > >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> > >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> > >>> >
> > >>> > Partitions
Re: Partitions reassignment is failing in Kafka 1.1.0
Thanks
On Sat, Jun 30, 2018 at 12:36 PM, Manikumar
wrote:
> It will be taken as "any" directory for each replica, which means replica
> will placed on any one of the
> configured directory on that broker.
>
> Since it is "log_dirs" optional, you can remove from the json.
>
> On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna
> wrote:
>
> > It is problem on my side. The code was changing the replicas count but
> not
> > the log_dirs. Since I am migrating from 0.10 this part of the code was
> not
> > changed.
> >
> > I have a follow up question what is the default value of log_dirs if I
> > don't specify it in reassignment.json ?
> >
> > On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna >
> > wrote:
> >
> > > I am generating the reassignent.json like below
> > >
> > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> > 127.0.0.1:2181 --generate --topics-to-move-json-file
> > /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail
> -1 >
> > /home/ubuntu/deploy/kafka/reassignment.json"
> > >
> > > Then I am doing the reassignment using the generated file
> > >
> > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> > 127.0.0.1:2181 --execute --reassignment-json-file
> > /home/ubuntu/deploy/kafka/reassignment.json
> > >
> > > kafka-reassign-partitions.sh helps states
> > >
> > > The JSON file with the partition reassignment configurationThe format
> to
> > >> use is -
> > >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> > >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that
> "log_dirs"
> > is
> > >> optional. When it is specified, its length must equal the length of
> the
> > >> replicas list. The value in this list can be either "any" or the
> > absolution
> > >> path of the log directory on the broker. If absolute log directory
> path
> > is
> > >> specified, it is currently required that the replica has not already
> > been
> > >> created on that broker. The replica will then be created in the
> > specified
> > >> log directory on the broker later.
> > >
> > >
> > > So it appears reassignment json that is generated by
> > > kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> > > some issue in kafka-reassign-partitions.sh or some misconfiguration
> from
> > my
> > > side. ?
> > >
> > > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna <
> subharaj.ma...@gmail.com
> > >
> > > wrote:
> > >
> > >> Please find the server.properties from one of the broker.
> > >>
> > >> broker.id=0
> > >> port=9092
> > >> num.network.threads=3
> > >> num.io.threads=8
> > >> socket.send.buffer.bytes=102400
> > >> socket.receive.buffer.bytes=102400
> > >> socket.request.max.bytes=104857600
> > >> log.dirs=/var/lib/kafka/kafka-logs
> > >> num.recovery.threads.per.data.dir=1
> > >> log.retention.hours=36
> > >> log.retention.bytes=1073741824
> > >> log.segment.bytes=536870912
> > >> log.retention.check.interval.ms=30
> > >> log.cleaner.enable=false
> > >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> > >> message.max.bytes=1500
> > >> replica.fetch.max.bytes=1500
> > >> auto.create.topics.enable=true
> > >> zookeeper.connection.timeout.ms=6000
> > >> unclean.leader.election.enable=false
> > >> delete.topic.enable=false
> > >> offsets.topic.replication.factor=1
> > >> transaction.state.log.replication.factor=1
> > >> transaction.state.log.min.isr=1
> > >>
> > >> I have placed server.log from a broker at https://gist.github.com/deb
> > >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> > >>
> > >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu wrote:
> > >>
> > >>> Seems to be related to KIP-113.
> > >>>
> > >>> server.properties didn't go thru. Do you mind pastebin'ing its
> content
> > ?
> > >>>
> > >>> If you can pastebin logs from broker, that should help.
> > >>>
> > >>> Thanks
> > >>>
> > >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> > subharaj.ma...@gmail.com
> > >>> >
> > >>> wrote:
> > >>>
> > >>> > Hi
> > >>> >
> > >>> > I altered a topic like below in kafka 1.1.0
> > >>> >
> > >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> > >>> 127.0.0.1:2181
> > >>> > --alter --topic Topic3 --config min.insync.replicas=2
> > >>> >
> > >>> > But whenever I am trying to verify the reassignment it is showing
> the
> > >>> > below exception
> > >>> >
> > >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> > >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> > >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> > >>> >
> > >>> > Partitions reassignment failed due to Size of replicas list
> Vector(3,
> > >>> 0, 2) is different from size of log dirs list Vector(any) for
> partition
> > >>> Topic3-7
> > >>> > kafka.common.AdminCommandFailedException: Size of replicas list
> > >>> Vector(3, 0, 2) is different from size of log dirs list Vector(any)
> for
> > >>> partition Topic3-7
> > >>> > at kafka.admin.ReassignPartitionsCommand$$
> anonfun$parsePartitio
Re: Partitions reassignment is failing in Kafka 1.1.0
It will be taken as "any" directory for each replica, which means replica
will placed on any one of the
configured directory on that broker.
Since it is "log_dirs" optional, you can remove from the json.
On Sat, Jun 30, 2018 at 12:02 PM Debraj Manna
wrote:
> It is problem on my side. The code was changing the replicas count but not
> the log_dirs. Since I am migrating from 0.10 this part of the code was not
> changed.
>
> I have a follow up question what is the default value of log_dirs if I
> don't specify it in reassignment.json ?
>
> On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna
> wrote:
>
> > I am generating the reassignent.json like below
> >
> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> 127.0.0.1:2181 --generate --topics-to-move-json-file
> /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail -1 >
> /home/ubuntu/deploy/kafka/reassignment.json"
> >
> > Then I am doing the reassignment using the generated file
> >
> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> 127.0.0.1:2181 --execute --reassignment-json-file
> /home/ubuntu/deploy/kafka/reassignment.json
> >
> > kafka-reassign-partitions.sh helps states
> >
> > The JSON file with the partition reassignment configurationThe format to
> >> use is -
> >> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> >> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that "log_dirs"
> is
> >> optional. When it is specified, its length must equal the length of the
> >> replicas list. The value in this list can be either "any" or the
> absolution
> >> path of the log directory on the broker. If absolute log directory path
> is
> >> specified, it is currently required that the replica has not already
> been
> >> created on that broker. The replica will then be created in the
> specified
> >> log directory on the broker later.
> >
> >
> > So it appears reassignment json that is generated by
> > kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> > some issue in kafka-reassign-partitions.sh or some misconfiguration from
> my
> > side. ?
> >
> > On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna >
> > wrote:
> >
> >> Please find the server.properties from one of the broker.
> >>
> >> broker.id=0
> >> port=9092
> >> num.network.threads=3
> >> num.io.threads=8
> >> socket.send.buffer.bytes=102400
> >> socket.receive.buffer.bytes=102400
> >> socket.request.max.bytes=104857600
> >> log.dirs=/var/lib/kafka/kafka-logs
> >> num.recovery.threads.per.data.dir=1
> >> log.retention.hours=36
> >> log.retention.bytes=1073741824
> >> log.segment.bytes=536870912
> >> log.retention.check.interval.ms=30
> >> log.cleaner.enable=false
> >> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> >> message.max.bytes=1500
> >> replica.fetch.max.bytes=1500
> >> auto.create.topics.enable=true
> >> zookeeper.connection.timeout.ms=6000
> >> unclean.leader.election.enable=false
> >> delete.topic.enable=false
> >> offsets.topic.replication.factor=1
> >> transaction.state.log.replication.factor=1
> >> transaction.state.log.min.isr=1
> >>
> >> I have placed server.log from a broker at https://gist.github.com/deb
> >> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
> >>
> >> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu wrote:
> >>
> >>> Seems to be related to KIP-113.
> >>>
> >>> server.properties didn't go thru. Do you mind pastebin'ing its content
> ?
> >>>
> >>> If you can pastebin logs from broker, that should help.
> >>>
> >>> Thanks
> >>>
> >>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna <
> subharaj.ma...@gmail.com
> >>> >
> >>> wrote:
> >>>
> >>> > Hi
> >>> >
> >>> > I altered a topic like below in kafka 1.1.0
> >>> >
> >>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
> >>> 127.0.0.1:2181
> >>> > --alter --topic Topic3 --config min.insync.replicas=2
> >>> >
> >>> > But whenever I am trying to verify the reassignment it is showing the
> >>> > below exception
> >>> >
> >>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
> >>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
> >>> /home/ubuntu/deploy/kafka/reassignment.json --verify
> >>> >
> >>> > Partitions reassignment failed due to Size of replicas list Vector(3,
> >>> 0, 2) is different from size of log dirs list Vector(any) for partition
> >>> Topic3-7
> >>> > kafka.common.AdminCommandFailedException: Size of replicas list
> >>> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for
> >>> partition Topic3-7
> >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> >>> ReassignPartitionsCommand.scala:262)
> >>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
> >>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
> >>> ReassignPartitionsCommand.scala:251)
> >>> > at scala.collection.Iterator$class.foreach(Iterator.scala:891)
>
Re: Partitions reassignment is failing in Kafka 1.1.0
It is problem on my side. The code was changing the replicas count but not
the log_dirs. Since I am migrating from 0.10 this part of the code was not
changed.
I have a follow up question what is the default value of log_dirs if I
don't specify it in reassignment.json ?
On Sat, Jun 30, 2018 at 11:15 AM, Debraj Manna
wrote:
> I am generating the reassignent.json like below
>
> /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> 127.0.0.1:2181 --generate --topics-to-move-json-file
> /home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail -1 >
> /home/ubuntu/deploy/kafka/reassignment.json"
>
> Then I am doing the reassignment using the generated file
>
> /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
> 127.0.0.1:2181 --execute --reassignment-json-file
> /home/ubuntu/deploy/kafka/reassignment.json
>
> kafka-reassign-partitions.sh helps states
>
> The JSON file with the partition reassignment configurationThe format to
>> use is -
>> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
>> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that "log_dirs" is
>> optional. When it is specified, its length must equal the length of the
>> replicas list. The value in this list can be either "any" or the absolution
>> path of the log directory on the broker. If absolute log directory path is
>> specified, it is currently required that the replica has not already been
>> created on that broker. The replica will then be created in the specified
>> log directory on the broker later.
>
>
> So it appears reassignment json that is generated by
> kafka-reassign-partions.sh is creating an issue with logdirs. Is this
> some issue in kafka-reassign-partitions.sh or some misconfiguration from my
> side. ?
>
> On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna
> wrote:
>
>> Please find the server.properties from one of the broker.
>>
>> broker.id=0
>> port=9092
>> num.network.threads=3
>> num.io.threads=8
>> socket.send.buffer.bytes=102400
>> socket.receive.buffer.bytes=102400
>> socket.request.max.bytes=104857600
>> log.dirs=/var/lib/kafka/kafka-logs
>> num.recovery.threads.per.data.dir=1
>> log.retention.hours=36
>> log.retention.bytes=1073741824
>> log.segment.bytes=536870912
>> log.retention.check.interval.ms=30
>> log.cleaner.enable=false
>> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
>> message.max.bytes=1500
>> replica.fetch.max.bytes=1500
>> auto.create.topics.enable=true
>> zookeeper.connection.timeout.ms=6000
>> unclean.leader.election.enable=false
>> delete.topic.enable=false
>> offsets.topic.replication.factor=1
>> transaction.state.log.replication.factor=1
>> transaction.state.log.min.isr=1
>>
>> I have placed server.log from a broker at https://gist.github.com/deb
>> raj-manna/4b4bdae8a1c15c36b313a04f37e8776d
>>
>> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu wrote:
>>
>>> Seems to be related to KIP-113.
>>>
>>> server.properties didn't go thru. Do you mind pastebin'ing its content ?
>>>
>>> If you can pastebin logs from broker, that should help.
>>>
>>> Thanks
>>>
>>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna >> >
>>> wrote:
>>>
>>> > Hi
>>> >
>>> > I altered a topic like below in kafka 1.1.0
>>> >
>>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
>>> 127.0.0.1:2181
>>> > --alter --topic Topic3 --config min.insync.replicas=2
>>> >
>>> > But whenever I am trying to verify the reassignment it is showing the
>>> > below exception
>>> >
>>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh
>>> --zookeeper 127.0.0.1:2181 --reassignment-json-file
>>> /home/ubuntu/deploy/kafka/reassignment.json --verify
>>> >
>>> > Partitions reassignment failed due to Size of replicas list Vector(3,
>>> 0, 2) is different from size of log dirs list Vector(any) for partition
>>> Topic3-7
>>> > kafka.common.AdminCommandFailedException: Size of replicas list
>>> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for
>>> partition Topic3-7
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
>>> ReassignPartitionsCommand.scala:262)
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(
>>> ReassignPartitionsCommand.scala:251)
>>> > at scala.collection.Iterator$class.foreach(Iterator.scala:891)
>>> > at scala.collection.AbstractIterator.foreach(Iterator.scala:133
>>> 4)
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
>>> sCommand.scala:251)
>>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartition
>>> sCommand.scala:250)
>>> > at scala.collection.immutable.List.foreach(List.scala:392)
>>> > at kafka.admin.ReassignParti
Re: Partitions reassignment is failing in Kafka 1.1.0
I am generating the reassignent.json like below
/home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
127.0.0.1:2181 --generate --topics-to-move-json-file
/home/ubuntu/deploy/kafka/topics_to_move.json --broker-list '%s' |tail
-1 > /home/ubuntu/deploy/kafka/reassignment.json"
Then I am doing the reassignment using the generated file
/home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
127.0.0.1:2181 --execute --reassignment-json-file
/home/ubuntu/deploy/kafka/reassignment.json
kafka-reassign-partitions.sh helps states
The JSON file with the partition reassignment configurationThe format to
> use is -
> {"partitions":[{"topic": "foo", "partition": 1, "replicas": [1,2,3],
> "log_dirs": ["dir1","dir2","dir3"]}], "version":1} Note that "log_dirs" is
> optional. When it is specified, its length must equal the length of the
> replicas list. The value in this list can be either "any" or the absolution
> path of the log directory on the broker. If absolute log directory path is
> specified, it is currently required that the replica has not already been
> created on that broker. The replica will then be created in the specified
> log directory on the broker later.
So it appears reassignment json that is generated by
kafka-reassign-partions.sh is creating an issue with logdirs. Is this some
issue in kafka-reassign-partitions.sh or some misconfiguration from my
side. ?
On Sat, Jun 30, 2018 at 10:26 AM, Debraj Manna
wrote:
> Please find the server.properties from one of the broker.
>
> broker.id=0
> port=9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/var/lib/kafka/kafka-logs
> num.recovery.threads.per.data.dir=1
> log.retention.hours=36
> log.retention.bytes=1073741824
> log.segment.bytes=536870912
> log.retention.check.interval.ms=30
> log.cleaner.enable=false
> zookeeper.connect=platform1:2181,platform2:2181,platform3:2181
> message.max.bytes=1500
> replica.fetch.max.bytes=1500
> auto.create.topics.enable=true
> zookeeper.connection.timeout.ms=6000
> unclean.leader.election.enable=false
> delete.topic.enable=false
> offsets.topic.replication.factor=1
> transaction.state.log.replication.factor=1
> transaction.state.log.min.isr=1
>
> I have placed server.log from a broker at https://gist.github.com/
> debraj-manna/4b4bdae8a1c15c36b313a04f37e8776d
>
> On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu wrote:
>
>> Seems to be related to KIP-113.
>>
>> server.properties didn't go thru. Do you mind pastebin'ing its content ?
>>
>> If you can pastebin logs from broker, that should help.
>>
>> Thanks
>>
>> On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna
>> wrote:
>>
>> > Hi
>> >
>> > I altered a topic like below in kafka 1.1.0
>> >
>> > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper
>> 127.0.0.1:2181
>> > --alter --topic Topic3 --config min.insync.replicas=2
>> >
>> > But whenever I am trying to verify the reassignment it is showing the
>> > below exception
>> >
>> > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper
>> 127.0.0.1:2181 --reassignment-json-file
>> /home/ubuntu/deploy/kafka/reassignment.json
>> --verify
>> >
>> > Partitions reassignment failed due to Size of replicas list Vector(3,
>> 0, 2) is different from size of log dirs list Vector(any) for partition
>> Topic3-7
>> > kafka.common.AdminCommandFailedException: Size of replicas list
>> Vector(3, 0, 2) is different from size of log dirs list Vector(any) for
>> partition Topic3-7
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.
>> apply(ReassignPartitionsCommand.scala:262)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.
>> apply(ReassignPartitionsCommand.scala:251)
>> > at scala.collection.Iterator$class.foreach(Iterator.scala:891)
>> > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitio
>> nsCommand.scala:251)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitio
>> nsCommand.scala:250)
>> > at scala.collection.immutable.List.foreach(List.scala:392)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:250)
>> > at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitio
>> nReassignmentData$1.apply(ReassignPartitionsCommand.scala:249)
>> > at scala.collection.immutable.List.foreach(List.scala:392)
>> > at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassig
>> nmentData(ReassignPartitionsCommand.scala:249)
>> > at kafka.admin.Reassi
Re: Partitions reassignment is failing in Kafka 1.1.0
Please find the server.properties from one of the broker. broker.id=0 port=9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/var/lib/kafka/kafka-logs num.recovery.threads.per.data.dir=1 log.retention.hours=36 log.retention.bytes=1073741824 log.segment.bytes=536870912 log.retention.check.interval.ms=30 log.cleaner.enable=false zookeeper.connect=platform1:2181,platform2:2181,platform3:2181 message.max.bytes=1500 replica.fetch.max.bytes=1500 auto.create.topics.enable=true zookeeper.connection.timeout.ms=6000 unclean.leader.election.enable=false delete.topic.enable=false offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 I have placed server.log from a broker at https://gist.github.com/debraj-manna/4b4bdae8a1c15c36b313a04f37e8776d On Sat, Jun 30, 2018 at 8:16 AM, Ted Yu wrote: > Seems to be related to KIP-113. > > server.properties didn't go thru. Do you mind pastebin'ing its content ? > > If you can pastebin logs from broker, that should help. > > Thanks > > On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna > wrote: > > > Hi > > > > I altered a topic like below in kafka 1.1.0 > > > > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 > > --alter --topic Topic3 --config min.insync.replicas=2 > > > > But whenever I am trying to verify the reassignment it is showing the > > below exception > > > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper > 127.0.0.1:2181 --reassignment-json-file > /home/ubuntu/deploy/kafka/reassignment.json > --verify > > > > Partitions reassignment failed due to Size of replicas list Vector(3, 0, > 2) is different from size of log dirs list Vector(any) for partition > Topic3-7 > > kafka.common.AdminCommandFailedException: Size of replicas list > Vector(3, 0, 2) is different from size of log dirs list Vector(any) for > partition Topic3-7 > > at kafka.admin.ReassignPartitionsCommand$$anonfun$ > parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply( > ReassignPartitionsCommand.scala:262) > > at kafka.admin.ReassignPartitionsCommand$$anonfun$ > parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply( > ReassignPartitionsCommand.scala:251) > > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > > at kafka.admin.ReassignPartitionsCommand$$anonfun$ > parsePartitionReassignmentData$1$$anonfun$apply$4.apply( > ReassignPartitionsCommand.scala:251) > > at kafka.admin.ReassignPartitionsCommand$$anonfun$ > parsePartitionReassignmentData$1$$anonfun$apply$4.apply( > ReassignPartitionsCommand.scala:250) > > at scala.collection.immutable.List.foreach(List.scala:392) > > at kafka.admin.ReassignPartitionsCommand$$anonfun$ > parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand. > scala:250) > > at kafka.admin.ReassignPartitionsCommand$$anonfun$ > parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand. > scala:249) > > at scala.collection.immutable.List.foreach(List.scala:392) > > at kafka.admin.ReassignPartitionsCommand$. > parsePartitionReassignmentData(ReassignPartitionsCommand.scala:249) > > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment( > ReassignPartitionsCommand.scala:90) > > at kafka.admin.ReassignPartitionsCommand$.verifyAssignment( > ReassignPartitionsCommand.scala:84) > > at kafka.admin.ReassignPartitionsCommand$.main( > ReassignPartitionsCommand.scala:58) > > at kafka.admin.ReassignPartitionsCommand.main( > ReassignPartitionsCommand.scala) > > > > > > My reassignment.json & server.properties is attached. Same thing used to > > work fine in kafka 0.10. Can someone let me what is going wrong? Is > > anything changed related to this in kafka 1.1.0 ? > > >
Re: Partitions reassignment is failing in Kafka 1.1.0
Seems to be related to KIP-113. server.properties didn't go thru. Do you mind pastebin'ing its content ? If you can pastebin logs from broker, that should help. Thanks On Fri, Jun 29, 2018 at 10:37 AM, Debraj Manna wrote: > Hi > > I altered a topic like below in kafka 1.1.0 > > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 > --alter --topic Topic3 --config min.insync.replicas=2 > > But whenever I am trying to verify the reassignment it is showing the > below exception > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper > 127.0.0.1:2181 --reassignment-json-file > /home/ubuntu/deploy/kafka/reassignment.json --verify > > Partitions reassignment failed due to Size of replicas list Vector(3, 0, 2) > is different from size of log dirs list Vector(any) for partition Topic3-7 > kafka.common.AdminCommandFailedException: Size of replicas list Vector(3, 0, > 2) is different from size of log dirs list Vector(any) for partition Topic3-7 > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:262) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:251) > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:251) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:250) > at scala.collection.immutable.List.foreach(List.scala:392) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:250) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:249) > at scala.collection.immutable.List.foreach(List.scala:392) > at > kafka.admin.ReassignPartitionsCommand$.parsePartitionReassignmentData(ReassignPartitionsCommand.scala:249) > at > kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90) > at > kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:84) > at > kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:58) > at > kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) > > > My reassignment.json & server.properties is attached. Same thing used to > work fine in kafka 0.10. Can someone let me what is going wrong? Is > anything changed related to this in kafka 1.1.0 ? >
Re: Partitions reassignment is failing in Kafka 1.1.0
Hi Anyone any thoughts? On Fri 29 Jun, 2018, 11:07 PM Debraj Manna, wrote: > Hi > > I altered a topic like below in kafka 1.1.0 > > /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 > --alter --topic Topic3 --config min.insync.replicas=2 > > But whenever I am trying to verify the reassignment it is showing the > below exception > > /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper > 127.0.0.1:2181 --reassignment-json-file > /home/ubuntu/deploy/kafka/reassignment.json --verify > > Partitions reassignment failed due to Size of replicas list Vector(3, 0, 2) > is different from size of log dirs list Vector(any) for partition Topic3-7 > kafka.common.AdminCommandFailedException: Size of replicas list Vector(3, 0, > 2) is different from size of log dirs list Vector(any) for partition Topic3-7 > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:262) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:251) > at scala.collection.Iterator$class.foreach(Iterator.scala:891) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:251) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:250) > at scala.collection.immutable.List.foreach(List.scala:392) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:250) > at > kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:249) > at scala.collection.immutable.List.foreach(List.scala:392) > at > kafka.admin.ReassignPartitionsCommand$.parsePartitionReassignmentData(ReassignPartitionsCommand.scala:249) > at > kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90) > at > kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:84) > at > kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:58) > at > kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) > > > My reassignment.json & server.properties is attached. Same thing used to > work fine in kafka 0.10. Can someone let me what is going wrong? Is > anything changed related to this in kafka 1.1.0 ? >
Partitions reassignment is failing in Kafka 1.1.0
Hi I altered a topic like below in kafka 1.1.0 /home/ubuntu/deploy/kafka/bin/kafka-topics.sh --zookeeper 127.0.0.1:2181 --alter --topic Topic3 --config min.insync.replicas=2 But whenever I am trying to verify the reassignment it is showing the below exception /home/ubuntu/deploy/kafka/bin/kafka-reassign-partitions.sh --zookeeper 127.0.0.1:2181 --reassignment-json-file /home/ubuntu/deploy/kafka/reassignment.json --verify Partitions reassignment failed due to Size of replicas list Vector(3, 0, 2) is different from size of log dirs list Vector(any) for partition Topic3-7 kafka.common.AdminCommandFailedException: Size of replicas list Vector(3, 0, 2) is different from size of log dirs list Vector(any) for partition Topic3-7 at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:262) at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4$$anonfun$apply$5.apply(ReassignPartitionsCommand.scala:251) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:251) at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1$$anonfun$apply$4.apply(ReassignPartitionsCommand.scala:250) at scala.collection.immutable.List.foreach(List.scala:392) at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:250) at kafka.admin.ReassignPartitionsCommand$$anonfun$parsePartitionReassignmentData$1.apply(ReassignPartitionsCommand.scala:249) at scala.collection.immutable.List.foreach(List.scala:392) at kafka.admin.ReassignPartitionsCommand$.parsePartitionReassignmentData(ReassignPartitionsCommand.scala:249) at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90) at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:84) at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:58) at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) My reassignment.json & server.properties is attached. Same thing used to work fine in kafka 0.10. Can someone let me what is going wrong? Is anything changed related to this in kafka 1.1.0 ? reassignment.json Description: application/json
