[jira] [Created] (KAFKA-6749) TopologyTestDriver fails when topoloy under test uses EXACTLY_ONCE

2018-04-05 Thread Frederic Arno (JIRA) 
Frederic Arno created KAFKA-6749:


 Summary: TopologyTestDriver fails when topoloy under test uses 
EXACTLY_ONCE
 Key: KAFKA-6749
 URL: https://issues.apache.org/jira/browse/KAFKA-6749
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.1.0
Reporter: Frederic Arno


Stream processing topologies which are configured to use `EXACTLY_ONCE` 
processing guarantee cannot be tested with the `TopologyTestDriver`. Tests 
usually crash with `java.lang.IllegalStateException: MockProducer hasn't been 
initialized for transactions` within the second call to 
`TopologyTestDriver.pipeInput()`, the first call works fine.

Changing the processing guarantee to `AT_LEAST_ONCE` makes tests pass.

This is a problem because it is expected that proper processor topologies can 
be successfully tested using `TopologyTestDriver`, however `TopologyTestDriver` 
can't handle `EXACTLY_ONCE` and crashes during tests. To a developer, this 
usually means that there is something wrong with their processor topologies.

Kafka developpers can reproduce this by adding:
{code:java}
put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, 
StreamsConfig.EXACTLY_ONCE);{code}

 to line 88 of TopologyTestDriverTest: 
streams/test-utils/src/test/java/org/apache/kafka/streams/TopologyTestDriverTest.java

Originally reported here: 
[http://mail-archives.apache.org/mod_mbox/kafka-users/201804.mbox/%3C54ab29ad-44e1-35bd-9c16-c1d8d68a88db%40gmail.com%3E]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6748) Scheduler cannot be cancelled from Punctuator

2018-04-05 Thread Frederic Arno (JIRA) 
Frederic Arno created KAFKA-6748:


 Summary: Scheduler cannot be cancelled from Punctuator
 Key: KAFKA-6748
 URL: https://issues.apache.org/jira/browse/KAFKA-6748
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.1, 1.1.0
Reporter: Frederic Arno
Assignee: Frederic Arno
 Fix For: 1.0.2, 1.1.1


A Scheduler cannot be cancelled from within the scheduled punctuator, I will 
post a test case to illustrate the problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6323) punctuate with WALL_CLOCK_TIME triggered immediately

2017-12-06 Thread Frederic Arno (JIRA) 
Frederic Arno created KAFKA-6323:


 Summary: punctuate with WALL_CLOCK_TIME triggered immediately
 Key: KAFKA-6323
 URL: https://issues.apache.org/jira/browse/KAFKA-6323
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.0
Reporter: Frederic Arno
 Fix For: 1.0.1


When working on a custom Processor from which I am scheduling a punctuation 
using WALL_CLOCK_TIME. I've noticed that whatever the punctuation interval I 
set, a call to my Punctuator is always triggered immediately.

Having a quick look at kafka-streams' code, I could find that all 
PunctuationSchedule's timestamps are matched against the current time in order 
to decide whether or not to trigger the punctuator 
(org.apache.kafka.streams.processor.internals.PunctuationQueue#mayPunctuate). 
However, I've only seen code that initializes PunctuationSchedule's timestamp 
to 0, which I guess is what is causing an immediate punctuation.

At least when using WALL_CLOCK_TIME, shouldn't the PunctuationSchedule's 
timestamp be initialized to current time + interval?




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)