[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16320675#comment-16320675
]
Chris Schwarzfischer commented on KAFKA-6437:
---------------------------------------------
Yep, I know it's by design and that doesn't need to change, of course.
"It hangs in the middle" means, that the application is actually starting and
processing data up to some intermediate topic. This makes it easy to overlook
that there are topics missing that prevent the application from running
correctly.
It would make it a lot easier to spot this error if there was an error
messaging saying that the topic is missing instead of simply switching to
"RUNNING" as if everything was ok…
> Streams does not warn about missing input topics, but hangs
> -----------------------------------------------------------
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 1.0.0
> Environment: Single client on single node broker
> Reporter: Chris Schwarzfischer
> Priority: Minor
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …00009, see below). Only parts of
> the intermediate topics are created (up to …00009)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-0000000011:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-0000000009-repartition]
> children: [KTABLE-AGGREGATE-0000000012]
> KTABLE-AGGREGATE-0000000012:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-0000000009]
> children: [KTABLE-TOSTREAM-0000000020]
> KTABLE-TOSTREAM-0000000020:
> children: [KSTREAM-SINK-0000000021]
> KSTREAM-SINK-0000000021:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-0000000017:
> topics:
> [mystreams_app-KSTREAM-MAP-0000000014-repartition]
> children: [KSTREAM-LEFTJOIN-0000000018]
> KSTREAM-LEFTJOIN-0000000018:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-0000000009]
> children: [KSTREAM-SINK-0000000019]
> KSTREAM-SINK-0000000019:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-0000000014-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-0000000009-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
