[jira] [Created] (KAFKA-7133) DisconnectException every 5 minutes in single restore consumer thread

[Chris Schwarzfischer (JIRA)]

Chris Schwarzfischer created KAFKA-7133:
---

 Summary: DisconnectException every 5 minutes in single restore 
consumer thread
 Key: KAFKA-7133
 URL: https://issues.apache.org/jira/browse/KAFKA-7133
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.1.0
 Environment: Kafka Streams application in Kubernetes.
Kafka Server in Docker on machine in host mode
Reporter: Chris Schwarzfischer


One of our streams applications (and only this one) gets a 
{{org.apache.kafka.common.errors.DisconnectException}} almost exactly every 5 
minutes.
The application has two of
KStream -> KGroupedStream -> KTable -> KGroupedTable -> KTable
aggregations.

Relevant config is in Streams:
{code:java}
this.properties.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, 
StreamsConfig.AT_LEAST_ONCE);
//...
this.properties.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2);
this.properties.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 1024 * 1024 
* 500 /* 500 MB */ );
this.properties.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 1024 * 
1024 * 100 /* 100 MB */);
this.properties.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, 1024 * 1024 * 50 /* 
50 MB */);
{code}

On the broker:
{noformat}
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_OFFSETS_RETENTION_MINUTES: 108000
KAFKA_MIN_INSYNC_REPLICAS: 2
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
KAFKA_TRANSACTIONAL_ID_EXPIRATION_MS: 2147483000
KAFKA_LOG_RETENTION_HOURS: 2688
KAFKA_OFFSETS_RETENTION_CHECK_INTERVAL_MS: 120
KAFKA_ZOOKEEPER_SESSION_TIMEOUT_MS: 12000
{noformat}

Logging gives us a single restore consumer thread that throws exceptions every 
5 mins:
 
{noformat}
July 4th 2018, 15:38:51.560 dockertest032018-07-04T13:38:51,559Z INFO  
: 
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
 FetchSessionHandler::handleError:440 - [Consumer 
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
 groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to 
node 1: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:37:54.833 dockertest032018-07-04T13:37:54,832Z INFO  
: 
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
 FetchSessionHandler::handleError:440 - [Consumer 
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
 groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to 
node 3: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:37:54.833 dockertest032018-07-04T13:37:54,832Z INFO  
: 
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
 FetchSessionHandler::handleError:440 - [Consumer 
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
 groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to 
node 2: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:32:26.379 dockertest032018-07-04T13:32:26,378Z INFO  
: 
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
 FetchSessionHandler::handleError:440 - [Consumer 
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
 groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL) to 
node 1: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:32:01.926 dockertest032018-07-04T13:32:01,925Z INFO  
: 
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
 FetchSessionHandler::handleError:440 - [Consumer 
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
 groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL) to 
node 2: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:32:01.926 dockertest032018-07-04T13:32:01,925Z INFO  
: 
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
 FetchSessionHandler::handleError:440 - [Consumer 
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
 groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL) to 
node 3: org.apache.kafka.common.errors.DisconnectException.
July 4th 2018, 15:26:53.886 dockertest032018-07-04T13:26:53,886Z INFO  
: 
[testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
 FetchSessionHandler::handleError:440 - [Consumer 
clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restor

[jira] [Created] (KAFKA-6437) Streams does not warn about missing input topics, but hangs

[Chris Schwarzfischer (JIRA)]

Chris Schwarzfischer created KAFKA-6437:
---

 Summary: Streams does not warn about missing input topics, but 
hangs
 Key: KAFKA-6437
 URL: https://issues.apache.org/jira/browse/KAFKA-6437
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
 Environment: Single client on single node broker
Reporter: Chris Schwarzfischer
Priority: Minor


*Case*
Streams application with two input topics being used for a left join.
When the left side topic is missing upon starting the streams application, it 
hangs "in the middle" of the topology (at …9, see below). Only parts of the 
intermediate topics are created (up to …9)
When the missing input topic is created, the streams application resumes 
processing.

{noformat}
Topology:
StreamsTask taskId: 2_0
ProcessorTopology:
KSTREAM-SOURCE-11:
topics: 
[mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
children:   [KTABLE-AGGREGATE-12]
KTABLE-AGGREGATE-12:
states: 
[KTABLE-AGGREGATE-STATE-STORE-09]
children:   [KTABLE-TOSTREAM-20]
KTABLE-TOSTREAM-20:
children:   [KSTREAM-SINK-21]
KSTREAM-SINK-21:
topic:  faxout_udr_month_customer_aggregration
KSTREAM-SOURCE-17:
topics: 
[mystreams_app-KSTREAM-MAP-14-repartition]
children:   [KSTREAM-LEFTJOIN-18]
KSTREAM-LEFTJOIN-18:
states: 
[KTABLE-AGGREGATE-STATE-STORE-09]
children:   [KSTREAM-SINK-19]
KSTREAM-SINK-19:
topic:  data_UDR_joined
Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0, 
mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
{noformat}

*Why this matters*
The applications does quite a lot of preprocessing before joining with the 
missing input topic. This preprocessing won't happen without the topic, 
creating a huge backlog of data.

*Fix*
Issue an `warn` or `error` level message at start to inform about the missing 
topic and it's consequences.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)