Re: Review Request 50529: Kafka brokers went down after Ambari upgrade due to IllegalArgumentException

2016-07-27 Thread Sriharsha Chintalapani 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50529/#review143796
---


Ship it!




Ship It!

- Sriharsha Chintalapani


On July 27, 2016, 7:40 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50529/
> ---
> 
> (Updated July 27, 2016, 7:40 p.m.)
> 
> 
> Review request for Ambari, Robert Levas, Sumit Mohanty, and Sriharsha 
> Chintalapani.
> 
> 
> Bugs: AMBARI-17929
> https://issues.apache.org/jira/browse/AMBARI-17929
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *Steps*
> # Deploy HDP-2.4.2 cluster with Ambari 2.2.2.0
> # Upgrade Ambari to 2.4.0.0
> # Observe the status of Kafka brokers
> 
> *Result*
> All brokers report down
> Logs indicate below:
> {code}
> [2016-07-27 05:48:26,535] INFO Initializing Kafka Timeline Metrics Sink 
> (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
> [2016-07-27 05:48:26,571] INFO Started Kafka Timeline metrics reporter with 
> polling period 10 seconds 
> (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
> [2016-07-27 05:48:26,716] INFO KafkaConfig values:
>   request.timeout.ms = 3
>   log.roll.hours = 168
>   inter.broker.protocol.version = 0.9.0.X
>   log.preallocate = false
>   security.inter.broker.protocol = PLAINTEXTSASL
>   controller.socket.timeout.ms = 3
>   broker.id.generation.enable = true
>   ssl.keymanager.algorithm = SunX509
>   ssl.key.password = [hidden]
>   log.cleaner.enable = true
>   ssl.provider = null
>   num.recovery.threads.per.data.dir = 1
>   background.threads = 10
>   unclean.leader.election.enable = true
>   sasl.kerberos.kinit.cmd = /usr/bin/kinit
>   replica.lag.time.max.ms = 1
>   ssl.endpoint.identification.algorithm = null
>   auto.create.topics.enable = true
>   zookeeper.sync.time.ms = 2000
>   ssl.client.auth = none
>   ssl.keystore.password = [hidden]
>   log.cleaner.io.buffer.load.factor = 0.9
>   offsets.topic.compression.codec = 0
>   log.retention.hours = 168
>   log.dirs = /kafka-logs
>   ssl.protocol = TLS
>   log.index.size.max.bytes = 10485760
>   sasl.kerberos.min.time.before.relogin = 6
>   log.retention.minutes = null
>   connections.max.idle.ms = 60
>   ssl.trustmanager.algorithm = PKIX
>   offsets.retention.minutes = 8640
>   max.connections.per.ip = 2147483647
>   replica.fetch.wait.max.ms = 500
>   metrics.num.samples = 2
>   port = 6667
>   offsets.retention.check.interval.ms = 60
>   log.cleaner.dedupe.buffer.size = 134217728
>   log.segment.bytes = 1073741824
>   group.min.session.timeout.ms = 6000
>   producer.purgatory.purge.interval.requests = 1
>   min.insync.replicas = 1
>   ssl.truststore.password = [hidden]
>   log.flush.scheduler.interval.ms = 9223372036854775807
>   socket.receive.buffer.bytes = 102400
>   leader.imbalance.per.broker.percentage = 10
>   num.io.threads = 8
>   zookeeper.connect = 
> nats11-36-alzs-dgm10toeriedwngdha-s11-3.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-4.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:2181
>   queued.max.requests = 500
>   offsets.topic.replication.factor = 3
>   replica.socket.timeout.ms = 3
>   offsets.topic.segment.bytes = 104857600
>   replica.high.watermark.checkpoint.interval.ms = 5000
>   broker.id = -1
>   ssl.keystore.location = /etc/security/serverKeys/keystore.jks
>   listeners = 
> PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
>   log.flush.interval.messages = 9223372036854775807
>   principal.builder.class = class 
> org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
>   log.retention.ms = null
>   offsets.commit.required.acks = -1
>   sasl.kerberos.principal.to.local.rules = [DEFAULT]
>   group.max.session.timeout.ms = 3
>   num.replica.fetchers = 1
>   advertised.listeners = 
> PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
>   replica.socket.receive.buffer.bytes = 65536
>   delete.topic.enable = false
>   log.index.interval.bytes = 4096
>   metric.reporters = []
>   compression.type = producer
>   log.cleanup.policy = delete
>   controlled.shutdown.max.retries = 3
>   log.cleaner.threads = 1
>   quota.window.size.seconds = 1
>   zookeeper.connection.timeout.ms = 25000
>

Re: Review Request 50529: Kafka brokers went down after Ambari upgrade due to IllegalArgumentException

2016-07-27 Thread Robert Levas 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50529/#review143794
---


Ship it!




Ship It!


ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog240.java
 (line 1906)


kafkaBrocker --> kafkaBroker  (small nit)


- Robert Levas


On July 27, 2016, 3:40 p.m., Vitalyi Brodetskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50529/
> ---
> 
> (Updated July 27, 2016, 3:40 p.m.)
> 
> 
> Review request for Ambari, Robert Levas, Sumit Mohanty, and Sriharsha 
> Chintalapani.
> 
> 
> Bugs: AMBARI-17929
> https://issues.apache.org/jira/browse/AMBARI-17929
> 
> 
> Repository: ambari
> 
> 
> Description
> ---
> 
> *Steps*
> # Deploy HDP-2.4.2 cluster with Ambari 2.2.2.0
> # Upgrade Ambari to 2.4.0.0
> # Observe the status of Kafka brokers
> 
> *Result*
> All brokers report down
> Logs indicate below:
> {code}
> [2016-07-27 05:48:26,535] INFO Initializing Kafka Timeline Metrics Sink 
> (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
> [2016-07-27 05:48:26,571] INFO Started Kafka Timeline metrics reporter with 
> polling period 10 seconds 
> (org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
> [2016-07-27 05:48:26,716] INFO KafkaConfig values:
>   request.timeout.ms = 3
>   log.roll.hours = 168
>   inter.broker.protocol.version = 0.9.0.X
>   log.preallocate = false
>   security.inter.broker.protocol = PLAINTEXTSASL
>   controller.socket.timeout.ms = 3
>   broker.id.generation.enable = true
>   ssl.keymanager.algorithm = SunX509
>   ssl.key.password = [hidden]
>   log.cleaner.enable = true
>   ssl.provider = null
>   num.recovery.threads.per.data.dir = 1
>   background.threads = 10
>   unclean.leader.election.enable = true
>   sasl.kerberos.kinit.cmd = /usr/bin/kinit
>   replica.lag.time.max.ms = 1
>   ssl.endpoint.identification.algorithm = null
>   auto.create.topics.enable = true
>   zookeeper.sync.time.ms = 2000
>   ssl.client.auth = none
>   ssl.keystore.password = [hidden]
>   log.cleaner.io.buffer.load.factor = 0.9
>   offsets.topic.compression.codec = 0
>   log.retention.hours = 168
>   log.dirs = /kafka-logs
>   ssl.protocol = TLS
>   log.index.size.max.bytes = 10485760
>   sasl.kerberos.min.time.before.relogin = 6
>   log.retention.minutes = null
>   connections.max.idle.ms = 60
>   ssl.trustmanager.algorithm = PKIX
>   offsets.retention.minutes = 8640
>   max.connections.per.ip = 2147483647
>   replica.fetch.wait.max.ms = 500
>   metrics.num.samples = 2
>   port = 6667
>   offsets.retention.check.interval.ms = 60
>   log.cleaner.dedupe.buffer.size = 134217728
>   log.segment.bytes = 1073741824
>   group.min.session.timeout.ms = 6000
>   producer.purgatory.purge.interval.requests = 1
>   min.insync.replicas = 1
>   ssl.truststore.password = [hidden]
>   log.flush.scheduler.interval.ms = 9223372036854775807
>   socket.receive.buffer.bytes = 102400
>   leader.imbalance.per.broker.percentage = 10
>   num.io.threads = 8
>   zookeeper.connect = 
> nats11-36-alzs-dgm10toeriedwngdha-s11-3.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-4.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:2181
>   queued.max.requests = 500
>   offsets.topic.replication.factor = 3
>   replica.socket.timeout.ms = 3
>   offsets.topic.segment.bytes = 104857600
>   replica.high.watermark.checkpoint.interval.ms = 5000
>   broker.id = -1
>   ssl.keystore.location = /etc/security/serverKeys/keystore.jks
>   listeners = 
> PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
>   log.flush.interval.messages = 9223372036854775807
>   principal.builder.class = class 
> org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
>   log.retention.ms = null
>   offsets.commit.required.acks = -1
>   sasl.kerberos.principal.to.local.rules = [DEFAULT]
>   group.max.session.timeout.ms = 3
>   num.replica.fetchers = 1
>   advertised.listeners = 
> PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
>   replica.socket.receive.buffer.bytes = 65536
>   delete.topic.enable = false
>   log.index.interval.bytes = 4096
>   metric.reporters = []
>   compression.type = producer
>

Review Request 50529: Kafka brokers went down after Ambari upgrade due to IllegalArgumentException

2016-07-27 Thread Vitalyi Brodetskyi 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50529/
---

Review request for Ambari, Robert Levas and Sumit Mohanty.


Bugs: AMBARI-17929
https://issues.apache.org/jira/browse/AMBARI-17929


Repository: ambari


Description
---

*Steps*
# Deploy HDP-2.4.2 cluster with Ambari 2.2.2.0
# Upgrade Ambari to 2.4.0.0
# Observe the status of Kafka brokers

*Result*
All brokers report down
Logs indicate below:
{code}
[2016-07-27 05:48:26,535] INFO Initializing Kafka Timeline Metrics Sink 
(org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
[2016-07-27 05:48:26,571] INFO Started Kafka Timeline metrics reporter with 
polling period 10 seconds 
(org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter)
[2016-07-27 05:48:26,716] INFO KafkaConfig values:
request.timeout.ms = 3
log.roll.hours = 168
inter.broker.protocol.version = 0.9.0.X
log.preallocate = false
security.inter.broker.protocol = PLAINTEXTSASL
controller.socket.timeout.ms = 3
broker.id.generation.enable = true
ssl.keymanager.algorithm = SunX509
ssl.key.password = [hidden]
log.cleaner.enable = true
ssl.provider = null
num.recovery.threads.per.data.dir = 1
background.threads = 10
unclean.leader.election.enable = true
sasl.kerberos.kinit.cmd = /usr/bin/kinit
replica.lag.time.max.ms = 1
ssl.endpoint.identification.algorithm = null
auto.create.topics.enable = true
zookeeper.sync.time.ms = 2000
ssl.client.auth = none
ssl.keystore.password = [hidden]
log.cleaner.io.buffer.load.factor = 0.9
offsets.topic.compression.codec = 0
log.retention.hours = 168
log.dirs = /kafka-logs
ssl.protocol = TLS
log.index.size.max.bytes = 10485760
sasl.kerberos.min.time.before.relogin = 6
log.retention.minutes = null
connections.max.idle.ms = 60
ssl.trustmanager.algorithm = PKIX
offsets.retention.minutes = 8640
max.connections.per.ip = 2147483647
replica.fetch.wait.max.ms = 500
metrics.num.samples = 2
port = 6667
offsets.retention.check.interval.ms = 60
log.cleaner.dedupe.buffer.size = 134217728
log.segment.bytes = 1073741824
group.min.session.timeout.ms = 6000
producer.purgatory.purge.interval.requests = 1
min.insync.replicas = 1
ssl.truststore.password = [hidden]
log.flush.scheduler.interval.ms = 9223372036854775807
socket.receive.buffer.bytes = 102400
leader.imbalance.per.broker.percentage = 10
num.io.threads = 8
zookeeper.connect = 
nats11-36-alzs-dgm10toeriedwngdha-s11-3.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-4.openstacklocal:2181,nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:2181
queued.max.requests = 500
offsets.topic.replication.factor = 3
replica.socket.timeout.ms = 3
offsets.topic.segment.bytes = 104857600
replica.high.watermark.checkpoint.interval.ms = 5000
broker.id = -1
ssl.keystore.location = /etc/security/serverKeys/keystore.jks
listeners = 
PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
log.flush.interval.messages = 9223372036854775807
principal.builder.class = class 
org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
log.retention.ms = null
offsets.commit.required.acks = -1
sasl.kerberos.principal.to.local.rules = [DEFAULT]
group.max.session.timeout.ms = 3
num.replica.fetchers = 1
advertised.listeners = 
PLAINTEXT://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:6667,SSL://nats11-36-alzs-dgm10toeriedwngdha-s11-1.openstacklocal:
replica.socket.receive.buffer.bytes = 65536
delete.topic.enable = false
log.index.interval.bytes = 4096
metric.reporters = []
compression.type = producer
log.cleanup.policy = delete
controlled.shutdown.max.retries = 3
log.cleaner.threads = 1
quota.window.size.seconds = 1
zookeeper.connection.timeout.ms = 25000
offsets.load.buffer.size = 5242880
zookeeper.session.timeout.ms = 3
ssl.cipher.suites = null
authorizer.class.name = 
org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.service.name = null
controlled.shutdown.enable = true
offsets.topic.num.partitions = 50
quota.window.num = 11
message.max.bytes = 100