[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549951#comment-16549951
]
Guozhang Wang commented on KAFKA-6437:
--
Thanks to [~yuzhih...@gmail.com], this has been fixed as of
https://issues.apache.org/jira/browse/KAFKA-5037.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Assignee: Mariam John
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16524379#comment-16524379
]
Guozhang Wang commented on KAFKA-6437:
--
I think this is again correlated to
https://issues.apache.org/jira/browse/KAFKA-5037, I'll write the summary of
this general issue in 5037.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Assignee: Mariam John
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418207#comment-16418207
]
Matthias J. Sax commented on KAFKA-6437:
I mean that some topics are available but other are not (ie if there are
multiple input topics). There are case for which Kafka Streams would not fail
but just process the available topics atm.
I agree that KAFAK-6520 is different; however, it's somehow related (-> state
"RUNNING" is confusion and not really appropriate). Just wanted to point out
the relationship. Not sure, if we should introduce DISCONNECTED and IDLE or
just one state for both. I mentioned it to get a "global picture" only.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Assignee: Mariam John
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418185#comment-16418185
]
Mariam John commented on KAFKA-6437:
[~mjsax] what do you mean by partially available input topics? KAFKA-6520 deals
with when the kafka broker is down, then the Kafka streams apps connected to it
have the state as RUNNING. I think, like you suggested we could have an IDLE
state in both cases and log a different warning for the different cases. For
example, in this case, it would be because of missing input topics and in the
case of KAFKA-6520, it would be because it is unable to connect to the broker.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Assignee: Mariam John
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418131#comment-16418131
]
Matthias J. Sax commented on KAFKA-6437:
[~wojda] Thanks for follow up. With regard to the "RUNNING" status, it seems to
be related to KAFKA-6520.
About this ticket and KAFKA-6720: I agree that it is opinion based if failing
of logging is the right approach. Note, that Kafka Streams inherits its
behavior from KafkaConsumer: it also idles if the input topics don't exist.
Therefore, it might even be required to change the consumer, affecting even
more developers. Maybe [~guozhang] can shed some light, why the consumer is
designed in this way (I am sure, there are good reasons for it).
About adding a new config: might be a solution. However, from our experience we
learned that having too many configs can be confusion for users -- thus, we
tend to be conservative about adding new configs if there is better solution. I
am not saying, we should not introduce a "fail-on-missing-topic" configs, I am
just saying we should discuss it in detail before we make a decision. Do you
think that fixing KAFKA-6520 would be an acceptable alternative solution
instead of throwing an exception? Meaning, introducing an new "IDLE" state if
the input topics are missing? Question would be, how to handle partially
available input topics? (Just putting out ideas here...)
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Assignee: Mariam John
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418073#comment-16418073
]
Daniel Wojda commented on KAFKA-6437:
-
I would like to add my comment as a user of Kafka Streams and author of
KAFKA-6720.
Important missing information here is that if you start Kafka Streams
application without input topics created, it'll log a warning and stays in this
"idle" state until you create that topic(s) *AND* a rebalancing happens. If you
check the status of stream it will be "RUNNING". What is more, please correct
me if I'm wrong, checking consumer lag will not help, because lag will be 0
(number of messages in non-existing topic is 0).
As [~mjsax] already mentioned "it's well documented that you need to create all
input topics before you start your application", so in my opinion "stopping the
world and failing" is a better option than starting a "zombie" application.
I understand that Kafka Streams has many users, other developers can have a
different opinion than me, but in that case I'd suggest introducing a new
config. "fail-on-missing-topic"? WDYT?
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Assignee: Mariam John
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416494#comment-16416494
]
Guozhang Wang commented on KAFKA-6437:
--
This is a interesting reported issue in KAFKA-6720 that for join-involved
topics, if it does not exist yet a exception will be thrown. I think it is not
a complete duplicate of this ticket, and I'd like to summarize the
"inconsistent" behavior that we are facing today:
1) For join operation from user topics directly (i.e. no reshuffling added as
Streams assumes input topics already partitioned by key), we'd require user
topics pre-exist; and if not, we throw TopologyBuilderException.
2) For join operation from repartition topics, since they are note available at
assignment phase we "assume" the repartition topics will be created and become
available, hence we do not check if the source topics are available. When the
source topic is missing, and hence no data will be send to the repartition
topics at all, Streams will hang (this is what this JIRA reported).
3) For stateless operations, if a source topic was missing, Streams will
continue but generate a warning.
So I think the actual fix should be in two folds:
1) We can [collect all external topic's
num.partition|https://github.com/apache/kafka/blob/5bdfbd13524da667289cacb774bb92df36a253f2/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L425-L437]
at the very beginning of the assign() phase, and log a warning entry if some
of the topic's metadata cannot be found.
2) In step one we do not need to [query the
metadata|https://github.com/apache/kafka/blob/5bdfbd13524da667289cacb774bb92df36a253f2/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L341]
again but we can get directly from the collected available num.partitions map.
3) The finally in ensureCopartitioning, if the metadata cannot be found we skip
the [checking co-partition
phase|https://github.com/apache/kafka/blob/5bdfbd13524da667289cacb774bb92df36a253f2/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L665]
but log another warning that "since the topic is not found, we will skip the
co-partition validation .."
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Assignee: Mariam John
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411910#comment-16411910
]
Mariam John commented on KAFKA-6437:
I will do that [~mjsax]. I agree that a WARN message seems right in this case.
I gave some thought about the option of making this configurable but that might
not be easy given that we would want to provide a way to configure per topic
partition and replication factors.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411873#comment-16411873
]
Matthias J. Sax commented on KAFKA-6437:
Sounds good to me. Feel free to work on this, [~johnma]. I think we can put
this check into {{KafkaStreams}} constructor and log a warning if input/output
topics are missing.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410940#comment-16410940
]
Chris Schwarzfischer commented on KAFKA-6437:
-
Yes, I think, the current behavior is not an error.
A warning level message would help people on deployment when there is something
missing. Making the behavior configurable would be nice, but I‘m not sure
anybody would actually use that.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410522#comment-16410522
]
John Roesler commented on KAFKA-6437:
-
I sort of feel like the existing behavior is better than failing, and [~k1th]
was mostly complaining of a visibility problem, so it would be best to just
address the visibility of this situation.
Logging a warning seems appropriate. It's certainly not an error. It could also
be info. I guess someone with more context could advise as to whether it seems
like a warn vs. info.
A log message in conjunction with a lag metric (I'm planning to submit a KIP
for that...) would really have prevented this from biting them, so maybe we
just focus on exposing better information rather than failing fast in this case.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409886#comment-16409886
]
Matthias J. Sax commented on KAFKA-6437:
Sounds reasonable to me. The question is, what we want to do when topics are
missing? Should we stop the world and fail (ie, throw an exception) or just log
a warning? Or maybe make it even configurable? WDYT?
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409759#comment-16409759
]
Mariam John commented on KAFKA-6437:
In the KafkaStreams we can use the admin client to list all topics and go
through the source and sink topics to see which all topics are missing and
print an error message like a MissingTopicException. Is there something more we
want to do for a fix or is there more to this bug I am missing.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364884#comment-16364884
]
Matthias J. Sax commented on KAFKA-6437:
I think it's two different issues. KAFKA-5037 is if _all_ input topics are
unknown – for this case, Streams get stuck in rebalance. For this ticket,
rebalance does not hang but processing is only started partially.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364876#comment-16364876
]
Guozhang Wang commented on KAFKA-6437:
--
I think https://issues.apache.org/jira/browse/KAFKA-5037 can be marked as a
duplicate of this one? [~mjsax]
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320675#comment-16320675
]
Chris Schwarzfischer commented on KAFKA-6437:
-
Yep, I know it's by design and that doesn't need to change, of course.
"It hangs in the middle" means, that the application is actually starting and
processing data up to some intermediate topic. This makes it easy to overlook
that there are topics missing that prevent the application from running
correctly.
It would make it a lot easier to spot this error if there was an error
messaging saying that the topic is missing instead of simply switching to
"RUNNING" as if everything was ok…
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320641#comment-16320641
]
Matthias J. Sax commented on KAFKA-6437:
I cannot follow. What do you mean by "it handgs in the middle" ?
Also note, that the behavior you describe is "by design" because the used
consumer works this way. It's also well documented that you need to create all
input topics before you start your application. We can of course do more
logging, but this does not really "solve" the problem...
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-6437) Streams does not warn about missing input topics, but hangs
[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320487#comment-16320487
]
Bill Bejeck commented on KAFKA-6437:
[~k1th] thanks for reporting.
> Streams does not warn about missing input topics, but hangs
> ---
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 1.0.0
> Environment: Single client on single node broker
>Reporter: Chris Schwarzfischer
>Priority: Minor
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …9, see below). Only parts of
> the intermediate topics are created (up to …9)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-11:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition]
> children: [KTABLE-AGGREGATE-12]
> KTABLE-AGGREGATE-12:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KTABLE-TOSTREAM-20]
> KTABLE-TOSTREAM-20:
> children: [KSTREAM-SINK-21]
> KSTREAM-SINK-21:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-17:
> topics:
> [mystreams_app-KSTREAM-MAP-14-repartition]
> children: [KSTREAM-LEFTJOIN-18]
> KSTREAM-LEFTJOIN-18:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-09]
> children: [KSTREAM-SINK-19]
> KSTREAM-SINK-19:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-14-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-09-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
