[jira] [Resolved] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley resolved KAFKA-5825. - Resolution: Done > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170594#comment-16170594 ] Ryan Worsley commented on KAFKA-5825: - You're exactly right [~mjsax]! No wonder they pay you the big bucks... I just changed the code to insert a random UUID into the application ID _et voila_ it works as expected - well spotted! Really grateful for this as I can now progress our evaluation to the next phase. Much obliged. > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5825) Streams not processing when exactly once is set
[
https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170517#comment-16170517
]
Ryan Worsley commented on KAFKA-5825:
-
Hi [~apurva],
My understanding was that a streams app should only be reading in
READ_COMMITTED mode when exactly-once is set (see the code I've attached).
Indeed if I set the consumer config explicitly I get the following error;
{{Unexpected user-specified consumer config isolation.level; because
processing.guarantee is set to 'exactly_once' consumers will always read
committed data only.}}
Am I misunderstanding your comments above?
I'd like to share the actual data in my-topic-0 - how exactly can I go about
getting that for you?
I'm hopefully able to take some time tomorrow to run a regular READ_COMMITTED
consumer to check the behaviour for you - would appreciate your thoughts on the
above though.
> Streams not processing when exactly once is set
> ---
>
> Key: KAFKA-5825
> URL: https://issues.apache.org/jira/browse/KAFKA-5825
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
> Environment: EmbeddedKafka running on Windows. Relevant files
> attached.
>Reporter: Ryan Worsley
> Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala
>
>
> +Set-up+
> I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/]
> for ScalaTest.
> This spins up a single broker internally on a random port.
> I've written two tests - the first without transactions, the second with.
> They're nearly identical apart from the config and the transactional
> semantics. I've written the transactional version based on Neha's
> [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/]
> which is the closest thing I could find to instructions.
> The tests wait until a single message is processed by the streams topology,
> they use this message to complete a promise that the test is waiting on.
> Once the promise completes the test verifies the value of the promise as
> being the expected value of the message.
> +Observed behaviour+
> The first test passes fine, the second test times out, the stream processor
> never seems to read the transactional message.
> +Notes+
> I've attached my build.sbt, log4j.properties and my Tests.scala file in order
> to make it as easy as possible for someone to re-create. I'm running on
> Windows and using Scala as this reflects my workplace. I completely expect
> there to be some configuration issue that's causing this, but am unable to
> proceed at this time.
> Related information:
> https://github.com/manub/scalatest-embedded-kafka/issues/82
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170267#comment-16170267 ] Ryan Worsley commented on KAFKA-5825: - Hi [~guozhang] and [~mjsax] - I'd really appreciate any further suggestions on how to take this forward. The organisation I represent is really keen to prove out Kafka transactions. Really grateful for any help you can give. > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley updated KAFKA-5825: Attachment: (was: Tests.scala) > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley updated KAFKA-5825: Attachment: (was: log-output.txt) > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[
https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Worsley updated KAFKA-5825:
Attachment: log-output.txt
Tests.scala
Thanks for your detailed help [~guozhang].
I've made the change you suggested, but it hasn't worked, please see the new
log output that I've attached (I've also attached the latest version of
{{Tests.scala}}.)
I experimented with turning the replication factor down to 1 with;
{{transaction.state.log.replication.factor = 1}}
{{transaction.state.log.min.isr = 1}}
I also increased the test timeout to 5 minutes.
Unfortunately the effect is the same - however the log output is now
significantly smaller which may be helpful to you.
What can we try next? Is this a bug?
> Streams not processing when exactly once is set
> ---
>
> Key: KAFKA-5825
> URL: https://issues.apache.org/jira/browse/KAFKA-5825
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
> Environment: EmbeddedKafka running on Windows. Relevant files
> attached.
>Reporter: Ryan Worsley
> Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala
>
>
> +Set-up+
> I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/]
> for ScalaTest.
> This spins up a single broker internally on a random port.
> I've written two tests - the first without transactions, the second with.
> They're nearly identical apart from the config and the transactional
> semantics. I've written the transactional version based on Neha's
> [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/]
> which is the closest thing I could find to instructions.
> The tests wait until a single message is processed by the streams topology,
> they use this message to complete a promise that the test is waiting on.
> Once the promise completes the test verifies the value of the promise as
> being the expected value of the message.
> +Observed behaviour+
> The first test passes fine, the second test times out, the stream processor
> never seems to read the transactional message.
> +Notes+
> I've attached my build.sbt, log4j.properties and my Tests.scala file in order
> to make it as easy as possible for someone to re-create. I'm running on
> Windows and using Scala as this reflects my workplace. I completely expect
> there to be some configuration issue that's causing this, but am unable to
> proceed at this time.
> Related information:
> https://github.com/manub/scalatest-embedded-kafka/issues/82
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[
https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Worsley updated KAFKA-5825:
Attachment: log-output.txt
Thanks [~guozhang] I've no attached {{log-output.txt}} which is the output of
me running {{sbt test}}
The log-output is the broker output AND the streams application output. Please
note this.
You can see the first test succeeding as expected and the second test timing
out.
Does this output give you any insight? As mentioned before, look at the
{{build.sbt}} file in order to resolve any dependency issues that were giving
you compilation problems.
> Streams not processing when exactly once is set
> ---
>
> Key: KAFKA-5825
> URL: https://issues.apache.org/jira/browse/KAFKA-5825
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
> Environment: EmbeddedKafka running on Windows. Relevant files
> attached.
>Reporter: Ryan Worsley
> Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala
>
>
> +Set-up+
> I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/]
> for ScalaTest.
> This spins up a single broker internally on a random port.
> I've written two tests - the first without transactions, the second with.
> They're nearly identical apart from the config and the transactional
> semantics. I've written the transactional version based on Neha's
> [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/]
> which is the closest thing I could find to instructions.
> The tests wait until a single message is processed by the streams topology,
> they use this message to complete a promise that the test is waiting on.
> Once the promise completes the test verifies the value of the promise as
> being the expected value of the message.
> +Observed behaviour+
> The first test passes fine, the second test times out, the stream processor
> never seems to read the transactional message.
> +Notes+
> I've attached my build.sbt, log4j.properties and my Tests.scala file in order
> to make it as easy as possible for someone to re-create. I'm running on
> Windows and using Scala as this reflects my workplace. I completely expect
> there to be some configuration issue that's causing this, but am unable to
> proceed at this time.
> Related information:
> https://github.com/manub/scalatest-embedded-kafka/issues/82
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5825) Streams not processing when exactly once is set
[
https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160405#comment-16160405
]
Ryan Worsley edited comment on KAFKA-5825 at 9/10/17 4:50 PM:
--
Thanks [~guozhang] I've now attached {{log-output.txt}} which is the output of
me running {{sbt test}}
The log-output is the broker output AND the streams application output. Please
note this.
You can see the first test succeeding as expected and the second test timing
out.
Does this output give you any insight? As mentioned before, look at the
{{build.sbt}} file in order to resolve any dependency issues that were giving
you compilation problems.
was (Author: ryanworsley):
Thanks [~guozhang] I've no attached {{log-output.txt}} which is the output of
me running {{sbt test}}
The log-output is the broker output AND the streams application output. Please
note this.
You can see the first test succeeding as expected and the second test timing
out.
Does this output give you any insight? As mentioned before, look at the
{{build.sbt}} file in order to resolve any dependency issues that were giving
you compilation problems.
> Streams not processing when exactly once is set
> ---
>
> Key: KAFKA-5825
> URL: https://issues.apache.org/jira/browse/KAFKA-5825
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.0
> Environment: EmbeddedKafka running on Windows. Relevant files
> attached.
>Reporter: Ryan Worsley
> Attachments: build.sbt, log4j.properties, log-output.txt, Tests.scala
>
>
> +Set-up+
> I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/]
> for ScalaTest.
> This spins up a single broker internally on a random port.
> I've written two tests - the first without transactions, the second with.
> They're nearly identical apart from the config and the transactional
> semantics. I've written the transactional version based on Neha's
> [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/]
> which is the closest thing I could find to instructions.
> The tests wait until a single message is processed by the streams topology,
> they use this message to complete a promise that the test is waiting on.
> Once the promise completes the test verifies the value of the promise as
> being the expected value of the message.
> +Observed behaviour+
> The first test passes fine, the second test times out, the stream processor
> never seems to read the transactional message.
> +Notes+
> I've attached my build.sbt, log4j.properties and my Tests.scala file in order
> to make it as easy as possible for someone to re-create. I'm running on
> Windows and using Scala as this reflects my workplace. I completely expect
> there to be some configuration issue that's causing this, but am unable to
> proceed at this time.
> Related information:
> https://github.com/manub/scalatest-embedded-kafka/issues/82
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16159834#comment-16159834 ] Ryan Worsley commented on KAFKA-5825: - Hi [~guozhang] did you use the SBT file to satisfy the dependencies? I'll post the full server logs as soon as I can. > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154827#comment-16154827 ] Ryan Worsley commented on KAFKA-5825: - Thanks [~guozhang] I don't get any obvious streams issues logged. The code is attached to the ticket - it's the simplest example I could come up with. Would appreciate someone glancing at it and maybe running it? > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5825) Streams not processing when exactly once is set
[
https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151923#comment-16151923
]
Ryan Worsley edited comment on KAFKA-5825 at 9/3/17 7:21 PM:
-
I've updated the tests to use the {{EmbeddedKafkaCluster}} and removed the
dependency on the Kafka test library that I was using - it took a while to
solve various library dependency issues, but I can now start a 3 node cluster.
[~mjsax] it's exhibiting the same behaviour unfortunately, there's no errors in
the broker logs, just a lot of this:
{quote}
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,26] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Recorded replica 0 log end
offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,26] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-26 unblocked 0
producer requests.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-11 unblocked 0
producer requests.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-11 unblocked 0
producer requests.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Recorded replica 0 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-8 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-8 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Recorded replica 0 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-41 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,17] on broker 2: Skipping update high watermark
since new hw 3 [0 : 474] is not larger than old hw 3 [0 : 474].All LEOs are 3
[0 : 474]
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-41 unblocked 0
producer requests.
20:15:38.716 [kafka-request-hand
[jira] [Comment Edited] (KAFKA-5825) Streams not processing when exactly once is set
[
https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151923#comment-16151923
]
Ryan Worsley edited comment on KAFKA-5825 at 9/3/17 7:20 PM:
-
I've updated the tests to use the {{EmbeddedKafkaCluster}} and removed the
dependency on the Kafka test library that I was using - it took a while to
solve various library dependency issues, but I can now start a 3 node cluster.
[~mjsax] it's exhibiting the same behaviour unfortunately, there's no errors in
the broker logs, just a lot of this:
{{20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,26] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Recorded replica 0 log end
offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,26] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-26 unblocked 0
producer requests.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-11 unblocked 0
producer requests.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-11 unblocked 0
producer requests.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Recorded replica 0 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-8 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-8 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Recorded replica 0 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-41 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,17] on broker 2: Skipping update high watermark
since new hw 3 [0 : 474] is not larger than old hw 3 [0 : 474].All LEOs are 3
[0 : 474]
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-41 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-6]
[jira] [Commented] (KAFKA-5825) Streams not processing when exactly once is set
[
https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151923#comment-16151923
]
Ryan Worsley commented on KAFKA-5825:
-
I've updated the tests to use the {{EmbeddedKafkaCluster}} and removed the
dependency on the Kafka test library that I was using - it took a while to
solve various library dependency issues, but I can now start a 3 node cluster.
[~mjsax] it's exhibiting the same behaviour unfortunately, there's no errors in
the broker logs, just a lot of this:
{{
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,26] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Recorded replica 0 log end
offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,26] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-26 unblocked 0
producer requests.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-11 unblocked 0
producer requests.
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,11] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.715 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-11 unblocked 0
producer requests.
20:15:38.715 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Recorded replica 0 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-8 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,8] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-8 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Recorded replica 0 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-41 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Skipping update high watermark
since new hw 0 [0 : 0] is not larger than old hw 0 [0 : 0].All LEOs are 0 [0 :
0]
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,41] on broker 2: Recorded replica 1 log end
offset (LEO) position 0.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [__transaction_state,17] on broker 2: Skipping update high watermark
since new hw 3 [0 : 474] is not larger than old hw 3 [0 : 474].All LEOs are 3
[0 : 474]
20:15:38.716 [kafka-request-handler-3] DEBUG kafka.server.ReplicaManager -
[Replica Manager on Broker 2]: Request key __transaction_state-41 unblocked 0
producer requests.
20:15:38.716 [kafka-request-handler-6] DEBUG kafka.cluster.Partition -
Partition [_
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley updated KAFKA-5825: Attachment: (was: build.sbt) > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley updated KAFKA-5825: Attachment: (was: Tests.scala) > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley updated KAFKA-5825: Attachment: Tests.scala build.sbt Updated to reflect suggestion by [~mjsax]. > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, build.sbt, log4j.properties, Tests.scala, > Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151850#comment-16151850 ] Ryan Worsley edited comment on KAFKA-5825 at 9/3/17 4:19 PM: - Hi [~mjsax] thank you for your reply. In that test I explicitly set the replication factor for the transaction log to 1... Are you saying that this doesn't have the desired effect? If that's true then a warning would be helpful rather than a seeming application hang? was (Author: ryanworsley): Hi [~mjsax] thank you for your reply. In that test I explicitly set the replication factor for the transaction log to 1... Are you saying that this doesn't have the desired effect? If > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151850#comment-16151850 ] Ryan Worsley commented on KAFKA-5825: - Hi [~mjsax] thank you for your reply. In that test I explicitly set the replication factor for the transaction log to 1... Are you saying that this doesn't have the desired effect? If > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley updated KAFKA-5825: Description: +Set-up+ I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] for ScalaTest. This spins up a single broker internally on a random port. I've written two tests - the first without transactions, the second with. They're nearly identical apart from the config and the transactional semantics. I've written the transactional version based on Neha's [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] which is the closest thing I could find to instructions. The tests wait until a single message is processed by the streams topology, they use this message to complete a promise that the test is waiting on. Once the promise completes the test verifies the value of the promise as being the expected value of the message. +Observed behaviour+ The first test passes fine, the second test times out, the streams processor never seems to read the transactional message. +Notes+ I've attached my build.sbt, log4j.properties and my Tests.scala file in order to make it as easy as possible for someone to re-create. I'm running on Windows and using Scala as this reflects my workplace. I completely expect there to be some configuration issue that's causing this, but am unable to proceed at this time. Related information: https://github.com/manub/scalatest-embedded-kafka/issues/82 was: +Set-up+ I'm using [EmbeddedKafka https://github.com/manub/scalatest-embedded-kafka/] for ScalaTest. This spins up a single broker internally on a random port. I've written two tests - the first without transactions, the second with. They're nearly identical apart from the config and the transactional semantics. I've written the transactional version based on Neha's [blog https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] which is the closest thing I could find to instructions. The tests wait until a single message is processed by the streams topology, they use this message to complete a promise that the test is waiting on. Once the promise completes the test verifies the value of the promise as being the expected value of the message. +Observed behaviour+ The first test passes fine, the second test times out, the streams processor never seems to read the transactional message. +Notes+ I've attached my build.sbt, log4j.properties and my Tests.scala file in order to make it as easy as possible for someone to re-create. I'm running on Windows and using Scala as this reflects my workplace. I completely expect there to be some configuration issue that's causing this, but am unable to proceed at this time. Related information: https://github.com/manub/scalatest-embedded-kafka/issues/82 > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the streams processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5825) Streams not processing when exactly once is set
[ https://issues.apache.org/jira/browse/KAFKA-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Worsley updated KAFKA-5825: Description: +Set-up+ I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] for ScalaTest. This spins up a single broker internally on a random port. I've written two tests - the first without transactions, the second with. They're nearly identical apart from the config and the transactional semantics. I've written the transactional version based on Neha's [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] which is the closest thing I could find to instructions. The tests wait until a single message is processed by the streams topology, they use this message to complete a promise that the test is waiting on. Once the promise completes the test verifies the value of the promise as being the expected value of the message. +Observed behaviour+ The first test passes fine, the second test times out, the stream processor never seems to read the transactional message. +Notes+ I've attached my build.sbt, log4j.properties and my Tests.scala file in order to make it as easy as possible for someone to re-create. I'm running on Windows and using Scala as this reflects my workplace. I completely expect there to be some configuration issue that's causing this, but am unable to proceed at this time. Related information: https://github.com/manub/scalatest-embedded-kafka/issues/82 was: +Set-up+ I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] for ScalaTest. This spins up a single broker internally on a random port. I've written two tests - the first without transactions, the second with. They're nearly identical apart from the config and the transactional semantics. I've written the transactional version based on Neha's [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] which is the closest thing I could find to instructions. The tests wait until a single message is processed by the streams topology, they use this message to complete a promise that the test is waiting on. Once the promise completes the test verifies the value of the promise as being the expected value of the message. +Observed behaviour+ The first test passes fine, the second test times out, the streams processor never seems to read the transactional message. +Notes+ I've attached my build.sbt, log4j.properties and my Tests.scala file in order to make it as easy as possible for someone to re-create. I'm running on Windows and using Scala as this reflects my workplace. I completely expect there to be some configuration issue that's causing this, but am unable to proceed at this time. Related information: https://github.com/manub/scalatest-embedded-kafka/issues/82 > Streams not processing when exactly once is set > --- > > Key: KAFKA-5825 > URL: https://issues.apache.org/jira/browse/KAFKA-5825 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 0.11.0.0 > Environment: EmbeddedKafka running on Windows. Relevant files > attached. >Reporter: Ryan Worsley > Attachments: build.sbt, log4j.properties, Tests.scala > > > +Set-up+ > I'm using [EmbeddedKafka|https://github.com/manub/scalatest-embedded-kafka/] > for ScalaTest. > This spins up a single broker internally on a random port. > I've written two tests - the first without transactions, the second with. > They're nearly identical apart from the config and the transactional > semantics. I've written the transactional version based on Neha's > [blog|https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] > which is the closest thing I could find to instructions. > The tests wait until a single message is processed by the streams topology, > they use this message to complete a promise that the test is waiting on. > Once the promise completes the test verifies the value of the promise as > being the expected value of the message. > +Observed behaviour+ > The first test passes fine, the second test times out, the stream processor > never seems to read the transactional message. > +Notes+ > I've attached my build.sbt, log4j.properties and my Tests.scala file in order > to make it as easy as possible for someone to re-create. I'm running on > Windows and using Scala as this reflects my workplace. I completely expect > there to be some configuration issue that's causing this, but am unable to > proceed at this time. > Related information: > https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KAFKA-5825) Streams not processing when exactly once is set
Ryan Worsley created KAFKA-5825: --- Summary: Streams not processing when exactly once is set Key: KAFKA-5825 URL: https://issues.apache.org/jira/browse/KAFKA-5825 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 0.11.0.0 Environment: EmbeddedKafka running on Windows. Relevant files attached. Reporter: Ryan Worsley Attachments: build.sbt, log4j.properties, Tests.scala +Set-up+ I'm using [EmbeddedKafka https://github.com/manub/scalatest-embedded-kafka/] for ScalaTest. This spins up a single broker internally on a random port. I've written two tests - the first without transactions, the second with. They're nearly identical apart from the config and the transactional semantics. I've written the transactional version based on Neha's [blog https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/] which is the closest thing I could find to instructions. The tests wait until a single message is processed by the streams topology, they use this message to complete a promise that the test is waiting on. Once the promise completes the test verifies the value of the promise as being the expected value of the message. +Observed behaviour+ The first test passes fine, the second test times out, the streams processor never seems to read the transactional message. +Notes+ I've attached my build.sbt, log4j.properties and my Tests.scala file in order to make it as easy as possible for someone to re-create. I'm running on Windows and using Scala as this reflects my workplace. I completely expect there to be some configuration issue that's causing this, but am unable to proceed at this time. Related information: https://github.com/manub/scalatest-embedded-kafka/issues/82 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
