Hi
I have a straightforward piece of code that creates a consumer (Kafka 0.9.0.0).
Properties props = new Properties();
props.put("zookeeper.connect", zookeeperServers);
props.put(org.apache.kafka.clients.consumer.ConsumerConfig.GROUP_ID_CONFIG,
groupId);
log.info("Starting consumer group for topic {} and group ID {}. Zookeeper
servers: {}", topic, groupId, zookeeperServers);
consumer = kafka.consumer.Consumer.createJavaConsumerConnector(new
ConsumerConfig(props));
log.info("Consumer group started for topic {} and group ID {}", topic, groupId);
We’ve run this countless times without any issues, but now we’re deploying a
new environment (AWS, just like the ones before) and it appears that the client
Java process dies entirely (without any logs/crash report/etc). This happens
right after logging the “Starting consumer group..”, so presumably when it
tries to createJavaConsumerConnector
Agreeably, this might be “environmental”, but even though we triple checked
everything (network setup, kafka logs, zookeeper logs, etc), we couldn’t
identify anything suspicious yet. So what I'd like to know is if there’s a way
to add further Kafka diagnosis/logging. Attached (trace-logging.txt) is further
logging after turning everything to TRACE, and at the top you can see the
message “Starting consumer…”, but with nothing really suspicious as far as I
can tell.
As an additional piece of information, Zookeeper does report the following when
this happens
2016-01-07 21:58:44,763 [myid:1] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
stream exception EndOfStreamException: Unable to read additional data from
client sessionid 0x1521e14797c0001, likely client has closed socket at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745) 2016-01-07 21:58:44,764 [myid:1] -
INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed
socket connection for client /10.100.101.159:41613 which had sessionid
0x1521e14797c0001
Any suggestions would be appreciated.
Thank you
Cosmin
