[ https://issues.apache.org/jira/browse/KAFKA-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthias J. Sax resolved KAFKA-7133. ------------------------------------ Resolution: Abandoned > DisconnectException every 5 minutes in single restore consumer thread > --------------------------------------------------------------------- > > Key: KAFKA-7133 > URL: https://issues.apache.org/jira/browse/KAFKA-7133 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 1.1.0 > Environment: Kafka Streams application in Kubernetes. > Kafka Server in Docker on machine in host mode > Reporter: Chris Schwarzfischer > Priority: Major > > One of our streams applications (and only this one) gets a > {{org.apache.kafka.common.errors.DisconnectException}} almost exactly every 5 > minutes. > The application has two of > KStream -> KGroupedStream -> KTable -> KGroupedTable -> KTable > aggregations. > Relevant config is in Streams: > {code:java} > this.properties.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, > StreamsConfig.AT_LEAST_ONCE); > //... > this.properties.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2); > this.properties.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 1024 * > 1024 * 500 /* 500 MB */ ); > this.properties.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 1024 * > 1024 * 100 /* 100 MB */); > this.properties.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, 1024 * 1024 * 50 > /* 50 MB */); > {code} > On the broker: > {noformat} > KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 > KAFKA_OFFSETS_RETENTION_MINUTES: 108000 > KAFKA_MIN_INSYNC_REPLICAS: 2 > KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3 > KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2 > KAFKA_TRANSACTIONAL_ID_EXPIRATION_MS: 2147483000 > KAFKA_LOG_RETENTION_HOURS: 2688 > KAFKA_OFFSETS_RETENTION_CHECK_INTERVAL_MS: 1200000 > KAFKA_ZOOKEEPER_SESSION_TIMEOUT_MS: 12000 > {noformat} > Logging gives us a single restore consumer thread that throws exceptions > every 5 mins: > > {noformat} > July 4th 2018, 15:38:51.560 dockertest03 2018-07-04T13:38:51,559Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) > to node 1: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:37:54.833 dockertest03 2018-07-04T13:37:54,832Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) > to node 3: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:37:54.833 dockertest03 2018-07-04T13:37:54,832Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) > to node 2: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:32:26.379 dockertest03 2018-07-04T13:32:26,378Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) > to node 1: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:32:01.926 dockertest03 2018-07-04T13:32:01,925Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) > to node 2: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:32:01.926 dockertest03 2018-07-04T13:32:01,925Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) > to node 3: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:26:53.886 dockertest03 2018-07-04T13:26:53,886Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) > to node 1: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:26:14.576 dockertest03 2018-07-04T13:26:14,575Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) > to node 3: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:26:14.575 dockertest03 2018-07-04T13:26:14,575Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) > to node 2: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:21:13.410 dockertest03 2018-07-04T13:21:13,409Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) > to node 1: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:20:39.618 dockertest03 2018-07-04T13:20:39,617Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) > to node 3: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:20:39.617 dockertest03 2018-07-04T13:20:39,617Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) > to node 2: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:15:31.454 dockertest03 2018-07-04T13:15:31,454Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) > to node 1: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:15:00.759 dockertest03 2018-07-04T13:15:00,758Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) > to node 3: org.apache.kafka.common.errors.DisconnectException. > July 4th 2018, 15:15:00.758 dockertest03 2018-07-04T13:15:00,758Z INFO > : > [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]: > FetchSessionHandler::handleError:440 - [Consumer > clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer, > groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) > to node 2: org.apache.kafka.common.errors.DisconnectException. > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)