Re: when use kafka streams to(topic) method sometime throw error?

[Sharat Joshi]

unsubscribe

On Mon, Mar 5, 2018 at 7:23 PM, ? ?  wrote:

>
> hi:
> I meet a problem today.
> when I use kafka stream to consumer one topic and do mapValues() method,
> and to another topic then .sometimes throw an error
> this is code sample:
> new StreamsBuilder().stream(xxxtopic, Consumed.with(Serdes.String(),
> Serdes.String())).mapValus(method).to(newTopic).
> sometimes it's work well but sometime it's throw error
>
> to topic newTopic due to org.apache.kafka.common.errors.TimeoutException:
> Expiring 6 record(s) for newTopic-2: 30030 ms has passed since last attempt
> plus backoff time
>
> 
> funk...@live.com
>

About configuring Kafka 2.10-0.10.2 using SASL

[Sharat Joshi]

Hi Kafka Users,

I am trying to setup a Simple Authentication Mechanism for my Kafka
Instance running on my Virtual Box VM.
I am facing a lot of difficulty in starting the Zookeeper 3.4.10.

The scenario is like this...
I have a single admin User called sharjosh who's starting both the
Zookeeper and Kafka processes on the VM.

I tried to use the following command to start the zookeeper

~/zookeeper-3.4.10$ export
JVMFLAGS='-Djava.security.auth.login.config=/home/sharjosh/kafka_2.10-0.10.2.1/config/zk_server_jaas.conf';
bin/zkServer.sh start-foreground conf/zoo.cfg

and get

 at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:53)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:87)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:117)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:82)
at
org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:200)java.io.IOException:
No JAAS configuration section named 'Server' was foundin
'/home/sharjosh/kafka_2.10-0.10.2.1/config/zk_server_jaas.conf'.2017-06-04
22:43:31,324 [myid:] - ERROR [main:ZooKeeperServerMain@64] - Unexpected
exception, exiting abnormally2017-06-04 22:43:31,323 [myid:] - ERROR
[main:ServerCnxnFactory@199] - No JAAS configuration section named 'Server'
was foundin '/home/sharjosh/kafka_2.10-0.10.2.1/config/zk_server_jaas.conf'.

My JAAS Config file for Zookeeper is
Server {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
useTicketCache=true
keyTab="/home/sharjosh/kafkaSecurity/sharjosh.keytab"
principal="sharjosh@localhost"
};

My zoo.cfg file is
# The number of milliseconds of each tick
tickTime=2000

# The number of ticks that the initial
# synchronization phase can take
initLimit=10

# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5

# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/sharjosh/zookeeper-3.4.10/data

# the port at which the clients will connect
clientPort=2181

# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=500

# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
autopurge.snapRetainCount=3

# Purge task interval in hours
# Set to "0" to disable auto purge feature
autopurge.purgeInterval=1

#
https://cwiki.apache.org/confluence/display/ZOOKEEPER/Client-Server+mutual+authentication
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
jaasLoginRenew=360

My Kafka startup command is
export KAFKA_OPTS='-Djava.security.krb5.conf=/etc/krb5.conf
-Djava.security.auth.login.config=/home/sharjosh/kafka_2.10-0.10.2.1/config/kafka_server_jaas.conf
-Dzookeeper.sasl.client=zKClient'; ./kafka-server-start.sh
/home/sharjosh/kafka-0.10.2.1-src/config/server.properties

My Kafka server's JAAS file is
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
serviceName=kafka
useTicketCache=true
keyTab="/home/sharjosh/kafkaSecurity/sharjosh.keytab"
principal="sharjosh@localhost"
};

zKClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
serviceName=zookeeper
useTicketCache=true
keyTab="/home/sharjosh/kafkaSecurity/sharjosh.keytab"
principal="sharjosh@localhost"
};

My security settings in kafka's server.properties
listeners=SASL_PLAINTEXT://:9092
advertised.listeners=SASL_PLAINTEXT://:9092

security.inter.broker.protocol=SASL_PLAINTEXT
sasl.enabled.mechanisms=GSSAPI,PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN

zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

On the other hand in Kafka Broker startup also I get errors as there's no
Zookeeper available.

org.apache.kafka.common.KafkaException: Exception while loading Zookeeper
JAAS login context 'Client'
 at
org.apache.kafka.common.security.JaasUtils.isZkSecurityEnabled(JaasUtils.java:154)
 at kafka.server.KafkaServer.initZk(KafkaServer.scala:310)
 at kafka.server.KafkaServer.startup(KafkaServer.scala:187)
 at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
 at kafka.Kafka$.main(Kafka.scala:67)
 at kafka.Kafka.main(Kafka.scala)
Caused by: java.lang.SecurityException: java.io.IOException: Configuration
Error:
 Line 9: expected [option key]
 at 

Getting Lots of "Unreasonable Length" exceptions on Zk when I start my Kafka broker.

[Sharat Joshi]

Hi Experts,

I have just downloaded latest Confluent Kafka 2.0.1 and started playing
with it setting up a Single broker instance, a Zk and a Schema instance.
When I start Zk its fine, BUT when I start my kafka server I get a lot of
these...

[2016-05-12 11:16:23,343] INFO Accepted socket connection from /
192.168.0.9:37379 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-12 11:16:23,343] WARN Exception causing close of session 0x0 due
to java.io.IOException: Unreasonable length = 140473187
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-12 11:16:23,344] INFO Closed socket connection for client /
192.168.0.9:37379 (no session established for client)
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-12 11:16:23,645] INFO Accepted socket connection from /
192.168.0.9:37380 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-12 11:16:23,646] WARN Exception causing close of session 0x0 due
to java.io.IOException: Unreasonable length = 140473187
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-12 11:16:23,646] INFO Closed socket connection for client /
192.168.0.9:37380 (no session established for client)
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-12 11:16:23,949] INFO Accepted socket connection from /
192.168.0.9:37381 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-12 11:16:23,950] WARN Exception causing close of session 0x0 due
to java.io.IOException: Unreasonable length = 140473187
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-12 11:16:23,950] INFO Closed socket connection for client /
192.168.0.9:37381 (no session established for client)
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-12 11:16:24,252] INFO Accepted socket connection from /
192.168.0.9:37382 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-05-12 11:16:24,253] WARN Exception causing close of session 0x0 due
to java.io.IOException: Unreasonable length = 140473187
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-12 11:16:24,253] INFO Closed socket connection for client /
192.168.0.9:37382 (no session established for client)
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-05-12 11:16:24,404] INFO Processed session termination for sessionid:
0x154a627b3d6 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-05-12 11:16:24,419] INFO Closed socket connection for client /
192.168.0.9:37236 which had sessionid 0x154a627b3d6
(org.apache.zookeeper.server.NIOServerCnxn)


It just goes On and On and On...

I am using basic defaults on Zk, Server and Schema settings...


Zk Settings...
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=100


Server Settings...
broker.id=0
listeners=PLAINTEXT://:9092
advertised.host.name=192.168.0.9
advertised.port=2181
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=30
zookeeper.connect=192.168.0.9:2181
zookeeper.connection.timeout.ms=6000


Schema Registry Settings...
port=8081
kafkastore.connection.url=192.168.0.9:2181
kafkastore.topic=_schemas
debug=false



PLEASE Help me get around this BASIC error.
I am desperately waiting to tide over this one ASAP.

Thanks a Lot for your time.
Sharat