Re: [VOTE] KIP-623: Add "internal-topics" option to streams application reset tool

[Joel Wee]

Thanks all for voting! 

I am closing the vote as accepted with three binding +1 votes (Boyang, 
Guozhang, John).

Thanks John for the suggestion. I think that makes sense. I have updated the 
KIP to say that only topics flagged as internal by the tool can be deleted, and 
have also rephrased the option description to make this clearer:

> --internal-topics   Comma-separated list of internal topics 
>  
> to delete. Must be a subset of the
>  
> internal topics marked for deletion 
> by 
> the default behaviour. To view these  
>  
> topics, do a dry-run without this 
>  
> option.  


Thanks Guozhang for the suggestion. The updated option description should 
address your first point. By the “topology description” script are you 
referring to bin/kafka-topics.sh? It currently has the option to display all 
topics (including internal topics). Were you thinking about adding something to 
view just internal topics?

Thanks Bruno for the suggestion. I will close this vote for now, and we can 
continue the discussion on another thread. (P.S. apologies for misspelling your 
name previously)

- Joel

> On 1 Jul 2020, at 3:04 AM, Boyang Chen  wrote:
> 
> Hey Bruno,
> 
> I agree adding a prompt would be a nice precaution, but it is not backward
> compatible as you suggested and could make the automation harder to
> achieve.
> 
> If you want, we may consider starting a separate ticket to discuss whether
> adding a prompt to let users be aware of the topics that are about to
> delete. However, this is also inverting the assumptions we made about
> `--dry-run` mode, which would become useless to me once we are adding a
> prompt asking users whether they want to remove these topics completely, or
> do nothing at all.
> 
> Back to KIP-623, I think this discussion could be held in orthogonal, which
> applies to more general considerations of reducing human errors, etc.
> 
> Boyang
> 
> On Tue, Jun 30, 2020 at 12:55 AM Bruno Cadonna  wrote:
> 
>> Hi,
>> 
>> I have already brought this up in the discussion thread.
>> 
>> Should we not run a dry-run in any case to avoid inadvertently
>> deleting topics of other applications?
>> 
>> I know it is a backward incompatible change if users use it in
>> scripts, but I think it is still worth discussing it. I would to hear
>> what committers think about it.
>> 
>> Best,
>> Bruno
>> 
>> On Tue, Jun 30, 2020 at 5:33 AM Boyang Chen 
>> wrote:
>>> 
>>> Thanks John for the great suggestion. +1 for enforcing the prefix check
>> for
>>> the `--internal-topics` list.
>>> 
>>> On Mon, Jun 29, 2020 at 5:11 PM John Roesler 
>> wrote:
>>> 
 Oh, I meant to say, if that’s the case, then I’m +1 (binding) also :)
 
 Thanks again,
 John
 
 On Mon, Jun 29, 2020, at 19:09, John Roesler wrote:
> Thanks for the KIP, Joel!
> 
> It seems like a good pattern to document would be to first run with
> —dry-run and without —internal-topics to list all potential topics,
>> and
> then to use —internal-topics if needed to limit the internal topics
>> to
> delete.
> 
> Just to make sure, would we have a sanity check to forbid including
> arbitrary topics? I.e., it seems like —internal-topics should require
> all the topics to be prefixed with the app id. Is that right?
> 
> Thanks,
> John
> 
> On Mon, Jun 29, 2020, at 18:25, Guozhang Wang wrote:
>> Thanks Joel, the KIP lgtm.
>> 
>> A minor suggestion is to explain where users can get the list of
 internal
>> topics of a given application, and maybe also add it as part of the
 helper
>> scripts, for example via topology description.
>> 
>> Overall, I'm +1 as well (binding).
>> 
>> Guozhang
>> 
>> 
>> On Sat, Jun 27, 2020 at 4:33 AM Joel Wee 
>> wrote:
>> 
>>> Thanks Boyang, I think what you’ve said makes sense. I’ve made
>> the
>>> motivation clearer now:
>>> 
>>> 
>>> Users may want to specify which internal topics should be
>> deleted. At
>>> present, the streams reset tool deletes all topics that start
>> with "<
>>> application.id>-" and there are no
>> options to
>>> control it.
>>> 
>>> The `--internal-topics` option is especially useful when there
>> are
 prefix
>>> conflicts between applications, e.g. "app" and "app-v2". In this
 case, if
>>> we want to reset "app", the reset tool’s default behaviour will
 delete both
>>> the internal topics of "app" and "app-v2" (since both are
>> prefixed by
>>> "app-"). With the `--internal-topics` option, we can provide
 internal topic
>>> names for "app" and delete the internal topics for "app" without
 deleting
>>> the internal topics for "app-v2".
>>> 

Re: [DISCUSS] KIP-510: Metrics library upgrade

[Ismael Juma]

Hi Mario,

Thanks for submitting the KIP. With Apache Kafka 3.0 approaching due to
KIP-500, it's a good time to tackle a metrics upgrade that may have
compatibility implications. Having said that, it would be good to avoid
unnecessary breaking changes. Is there a way to avoid changing the metric
names?

Ismael

On Fri, Aug 30, 2019 at 10:02 AM Mario Molina  wrote:

> Could we start the discussion?
>
> On Wed, 21 Aug 2019 at 08:57, Mario Molina  wrote:
>
> > Hi there,
> >
> > I've written KIP-510
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-510%3A+Metrics+library+upgrade
> >
> > proposing the upgrade of the metrics library which Kafka is using.
> >
> > Please, have a look and let me know what you think,
> > Thanks,
> > Mario
> >
>

Build failed in Jenkins: kafka-trunk-jdk11 #1618

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10209: Fix connect_rest_test.py after the introduction of new


--
[...truncated 3.19 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

[jira] [Created] (KAFKA-10234) The key/value deserializer used by ConsoleConsumer is not closed

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-10234:
--

 Summary: The key/value deserializer used by ConsoleConsumer is not 
closed
 Key: KAFKA-10234
 URL: https://issues.apache.org/jira/browse/KAFKA-10234
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


We instantiate, configure and use them but them are never closed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-10232) MirrorMaker2 internal topics Formatters

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-10232.

Fix Version/s: 2.7.0
   Resolution: Fixed

> MirrorMaker2 internal topics Formatters
> ---
>
> Key: KAFKA-10232
> URL: https://issues.apache.org/jira/browse/KAFKA-10232
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect, mirrormaker
>Reporter: Mickael Maison
>Assignee: Mickael Maison
>Priority: Major
> Fix For: 2.7.0
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-597%3A+MirrorMaker2+internal+topics+Formatters



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10232) MirrorMaker2 internal topics Formatters

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-10232:
--

 Summary: MirrorMaker2 internal topics Formatters
 Key: KAFKA-10232
 URL: https://issues.apache.org/jira/browse/KAFKA-10232
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect, mirrormaker
Reporter: Mickael Maison
Assignee: Mickael Maison


https://cwiki.apache.org/confluence/display/KAFKA/KIP-597%3A+MirrorMaker2+internal+topics+Formatters



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10231) Broken Kafka Connect node to node communication if invalid hostname is in place

[Pere Urbon-Bayes (Jira)]

Pere Urbon-Bayes created KAFKA-10231:


 Summary: Broken Kafka Connect node to node communication if 
invalid hostname is in place
 Key: KAFKA-10231
 URL: https://issues.apache.org/jira/browse/KAFKA-10231
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.4.1, 2.5.0, 2.3.1, 2.4.0
Reporter: Pere Urbon-Bayes


As a Kafka Connect operator I would expect a more definitive error when the 
internal node to node communication can't happen.

If the hostname contains an invalid character according to the [RFC1123 section 
2.1|[https://tools.ietf.org/html/rfc1123#page-13]], the error raised by the 
Kafka Connect worker node look like:

 
{quote}{{[2020-06-30 10:38:49,990] ERROR Uncaught exception in REST call to 
/connectors 
(org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper)}}{{java.lang.IllegalArgumentException:
 Invalid URI host: null (authority: kafka_connect-0.cdh-dev-2:8083)}}{{
at org.eclipse.jetty.client.HttpClient.checkHost(HttpClient.java:506)}}{{   
 at org.eclipse.jetty.client.HttpClient.newHttpRequest(HttpClient.java:491)}}{{ 
   at 
org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:449)}}{{
at org.eclipse.jetty.client.HttpClient.newRequest(HttpClient.java:438)}}{{  
  at 
org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:83)}}{{
at 
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.completeOrForwardRequest(ConnectorsResource.java:309)}}{{
at 
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResource.createConnector(ConnectorsResource.java:138)}}
{quote}
 

it would be much nicer for operators that such situations are detected in 
similar, or improved, version as the JVM is doing in the [IDN 
class|[https://github.com/JetBrains/jdk8u_jdk/blob/master/src/share/classes/java/net/IDN.java#L291]].

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10233) KafkaConsumer polls in a tight loop if group is not authorized

[Rajini Sivaram (Jira)]

Rajini Sivaram created KAFKA-10233:
--

 Summary: KafkaConsumer polls in a tight loop if group is not 
authorized
 Key: KAFKA-10233
 URL: https://issues.apache.org/jira/browse/KAFKA-10233
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 2.7.0


Consumer propagates GroupAuthorizationException from poll immediately when 
trying to find coordinator even though it is a retriable exception. If the 
application polls in a loop, ignoring retriable exceptions, the consumer tries 
to find coordinator in a tight loop without any backoff. We should apply retry 
backoff in this case to avoid overloading brokers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-10209) Fix connect_rest_test.py after the introduction of new connector configs

[Konstantine Karantasis (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantine Karantasis resolved KAFKA-10209.

Resolution: Fixed

> Fix connect_rest_test.py after the introduction of new connector configs
> 
>
> Key: KAFKA-10209
> URL: https://issues.apache.org/jira/browse/KAFKA-10209
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Minor
> Fix For: 2.6.0
>
>
> There are two new configs introduced by 
> [371f14c3c12d2e341ac96bd52393b43a10acfa84|https://github.com/apache/kafka/commit/371f14c3c12d2e341ac96bd52393b43a10acfa84]
>  and 
> [1c4eb1a5757df611735cfac9b709e0d80d0da4b3|https://github.com/apache/kafka/commit/1c4eb1a5757df611735cfac9b709e0d80d0da4b3]
>  so we have to update the expected configs in connect_rest_test.py



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: kafka-trunk-jdk8 #4692

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10209: Fix connect_rest_test.py after the introduction of new


--
[...truncated 3.17 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 

Build failed in Jenkins: kafka-trunk-jdk14 #267

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10209: Fix connect_rest_test.py after the introduction of new


--
[...truncated 3.19 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 

[VOTE] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

[Jorge Esteban Quilcate Otoya]

Hola everyone,

I'd like to start a new thread to vote for KIP-617 as there have been
significant changes since the previous vote started.

KIP wiki page:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-617%3A+Allow+Kafka+Streams+State+Stores+to+be+iterated+backwards

Many thanks!

Jorge.

Re: [DISCUSS] KIP-622 Add currentSystemTimeMs and currentStreamTimeMs to ProcessorContext

[Piotr Smoliński]

1. Makes sense; let me propose something

2. That's not testing-only. The goal is to use the same API to access the time
in deployment and testing environments. The major driver is 
System.currentTimeMillis(),
which a) cannot be used in tests b) could go in specific cases back c) is not 
compatible
with punctuator call. The idea is that we could access clock using uniform API. 
For completness we should have same API for system and stream time.

3. There aren't that many subclasses. Two important ones are 
ProcessorContextImpl and 
MockProcessorContext (and third one: ForwardingDisableProcessorContext). If 
given
implementation does not support schedule() call, there is no reason to support 
clock access. 
The default implementation should just throw UnsupportedOperationException just 
to prevent
from compilation errors in possible subclasses.

On 2020/07/01 02:24:43, Boyang Chen  wrote: 
> Thanks Will for the KIP. A couple questions and suggestions:
> 
> 1. I think for new APIs to make most sense, we should add a minimal example
> demonstrating how it could be useful to structure unit tests w/o the new
> APIs.
> 2. If this is a testing-only feature, could we only add it
> to MockProcessorContext?
> 3. Regarding the API, since this will be added to the ProcessorContext with
> many subclasses, does it make sense to provide default implementations as
> well?
> 
> Boyang
> 
> On Tue, Jun 30, 2020 at 6:56 PM William Bottrell 
> wrote:
> 
> > Thanks, John! I made the change. How much longer should I let there be
> > discussion before starting a VOTE?
> >
> > On Sat, Jun 27, 2020 at 6:50 AM John Roesler  wrote:
> >
> > > Thanks, Will,
> > >
> > > That looks good to me. I would only add "cached" or something
> > > to indicate that it wouldn't just transparently look up the current
> > > System.currentTimeMillis every time.
> > >
> > > For example:
> > > /**
> > >  * Returns current cached wall-clock system timestamp in milliseconds.
> > >  *
> > >  * @return the current cached wall-clock system timestamp in milliseconds
> > >  */
> > > long currentSystemTimeMs();
> > >
> > > I don't want to give specific information about _when_ exactly the
> > > timestamp cache will be updated, so that we can adjust it in the
> > > future, but it does seem important to make people aware that they
> > > won't see the timestamp advance during the execution of
> > > Processor.process(), for example.
> > >
> > > With that modification, I'll be +1 on this proposal.
> > >
> > > Thanks again for the KIP!
> > > -John
> > >
> > > On Thu, Jun 25, 2020, at 02:32, William Bottrell wrote:
> > > > Thanks, John! I appreciate you adjusting my lingo. I made the change to
> > > the
> > > > KIP. I will add the note about system time to the javadoc.
> > > >
> > > > On Wed, Jun 24, 2020 at 6:52 PM John Roesler 
> > > wrote:
> > > >
> > > > > Hi Will,
> > > > >
> > > > > This proposal looks good to me overall. Thanks for the contribution!
> > > > >
> > > > > Just a couple of minor notes:
> > > > >
> > > > > The system time method would return a cached timestamp that Streams
> > > looks
> > > > > up once when it starts processing a record. This may be confusing, so
> > > it
> > > > > might be good to state it in the javadoc.
> > > > >
> > > > > I thought the javadoc for the stream time might be a bit confusing.
> > We
> > > > > normally talk about “Tasks” not “partition groups” in the public api.
> > > Maybe
> > > > > just saying that it’s “the maximum timestamp of any record yet
> > > processed by
> > > > > the task” would be both high level and accurate.
> > > > >
> > > > > Thanks again!
> > > > > -John
> > > > >
> > > > > On Mon, Jun 22, 2020, at 02:10, William Bottrell wrote:
> > > > > > Thanks, Bruno. I updated the KIP, so hopefully it makes more sense.
> > > > > Thanks
> > > > > > to Matthias J. Sax and Piotr Smolinski for helping with details.
> > > > > >
> > > > > > I welcome more feedback. Let me know if something doesn't make
> > sense
> > > or I
> > > > > > need to provide more detail. Also, feel free to enlighten me.
> > Thanks!
> > > > > >
> > > > > > On Thu, Jun 11, 2020 at 1:11 PM Bruno Cadonna 
> > > > > wrote:
> > > > > >
> > > > > > > Hi Will,
> > > > > > >
> > > > > > > Thank you for the KIP.
> > > > > > >
> > > > > > > 1. Could you elaborate a bit more on the motivation in the KIP?
> > An
> > > > > > > example would make the motivation clearer.
> > > > > > >
> > > > > > > 2. In section "Proposed Changes" you do not need to show the
> > > > > > > implementation and describe internals. A description of the
> > > expected
> > > > > > > behavior of the newly added methods should suffice.
> > > > > > >
> > > > > > > 3. In "Compatibility, Deprecation, and Migration Plan" you should
> > > > > > > state that the change is backward compatible because the two
> > > methods
> > > > > > > will be added and no other method will be changed or removed.
> > > > > > >
> > > > > > > Best,
> > > > > > > Bruno
> > > > > > >
> > > > > > > On Wed, Jun 

Build failed in Jenkins: kafka-trunk-jdk11 #1617

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10232: MirrorMaker2 internal topics Formatters KIP-597 (#8604)


--
[...truncated 3.19 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualWithNullForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

Build failed in Jenkins: kafka-trunk-jdk8 #4691

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10232: MirrorMaker2 internal topics Formatters KIP-597 (#8604)


--
[...truncated 3.16 MB...]
org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 

Build failed in Jenkins: kafka-trunk-jdk14 #266

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10232: MirrorMaker2 internal topics Formatters KIP-597 (#8604)


--
[...truncated 3.19 MB...]

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:testClasses
> Task :streams:upgrade-system-tests-0101:checkstyleTest
> Task :streams:upgrade-system-tests-0101:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:test
> Task 

Re: [DISCUSS] KIP-617: Allow Kafka Streams State Stores to be iterated backwards

[Jorge Esteban Quilcate Otoya]

Hi John,

Thanks for the feedback.

I'd be happy to take the third option and consider moving methods to
ReadOnlySessionStore as part of the KIP.
Docs is updated to reflect these changes.

Cheers,
Jorge.

On Thu, Jul 2, 2020 at 10:06 PM John Roesler  wrote:

> Hey Jorge,
>
> Thanks for the details. That sounds like a mistake to me on both counts.
>
> I don’t think you need to worry about those depreciations. If the
> interface methods aren’t deprecated, then the methods are not deprecated.
> We should remove the annotations, but it doesn’t need to be in the kip.
>
> I think any query methods should have been in the ReadOnly interface. I
> guess it’s up to you whether you want to:
> 1. Add the reverse methods next to the existing methods (what you have in
> the kip right now)
> 2. Partially fix it by adding your new methods to the ReadOnly interface
> 3. Fully fix the problem by moving the existing methods as well as your
> new ones. Since  SessionStore extends ReadOnlySessionStore, it’s ok just to
> move the definitions.
>
> I’m ok with whatever you prefer.
>
> Thanks,
> John
>
> On Thu, Jul 2, 2020, at 11:29, Jorge Esteban Quilcate Otoya wrote:
> > (sorry for the spam)
> >
> > Actually `findSessions` are only deprecated on `InMemorySessionStore`,
> > which seems strange as RocksDB and interfaces haven't marked these
> methods
> > as deprecated.
> >
> > Any hint on how to handle this?
> >
> > Cheers,
> >
> > On Thu, Jul 2, 2020 at 4:57 PM Jorge Esteban Quilcate Otoya <
> > quilcate.jo...@gmail.com> wrote:
> >
> > > @John: I can see there are some deprecations in there as well. Will
> update
> > > the KIP.
> > >
> > > Thanks,
> > > Jorge
> > >
> > >
> > > On Thu, Jul 2, 2020 at 3:29 PM Jorge Esteban Quilcate Otoya <
> > > quilcate.jo...@gmail.com> wrote:
> > >
> > >> Thanks John.
> > >>
> > >> > it looks like there’s a revision error in which two methods are
> > >> proposed for SessionStore, but seem like they should be in
> > >> ReadOnlySessionStore. Do I read that right?
> > >>
> > >> Yes, I've opted to keep the new methods alongside the existing ones.
> In
> > >> the case of SessionStore, `findSessions` are in `SessionStore`, and
> `fetch`
> > >> are in `ReadOnlySessionStore`. If it makes more sense, I can move all
> of
> > >> them to ReadOnlySessionStore.
> > >> Let me know what you think.
> > >>
> > >> Thanks,
> > >> Jorge.
> > >>
> > >> On Thu, Jul 2, 2020 at 2:36 PM John Roesler 
> wrote:
> > >>
> > >>> Hi Jorge,
> > >>>
> > >>> Thanks for the update. I think this is a good plan.
> > >>>
> > >>> I just took a look at the KIP again, and it looks like there’s a
> > >>> revision error in which two methods are proposed for SessionStore,
> but seem
> > >>> like they should be in ReadOnlySessionStore. Do I read that right?
> > >>>
> > >>> Otherwise, I’m happy with your proposal.
> > >>>
> > >>> Thanks,
> > >>> John
> > >>>
> > >>> On Wed, Jul 1, 2020, at 17:01, Jorge Esteban Quilcate Otoya wrote:
> > >>> > Quick update: KIP is updated with latest changes now.
> > >>> > Will leave it open this week while working on the PR.
> > >>> >
> > >>> > Hope to open a new vote thread over the next few days if no
> additional
> > >>> > feedback is provided.
> > >>> >
> > >>> > Cheers,
> > >>> > Jorge.
> > >>> >
> > >>> > On Mon, Jun 29, 2020 at 11:30 AM Jorge Esteban Quilcate Otoya <
> > >>> > quilcate.jo...@gmail.com> wrote:
> > >>> >
> > >>> > > Thanks, John!
> > >>> > >
> > >>> > > Make sense to reconsider the current approach. I was heading in a
> > >>> similar
> > >>> > > direction while drafting the implementation. Metered, Caching,
> and
> > >>> other
> > >>> > > layers will also have to get duplicated to build up new methods
> in
> > >>> `Stores`
> > >>> > > factory, and class casting issues would appear on stores created
> by
> > >>> DSL.
> > >>> > >
> > >>> > > I will draft a proposal with new methods (move methods from
> proposed
> > >>> > > interfaces to existing ones) with default implementation in a KIP
> > >>> update
> > >>> > > and wait for Matthias to chime in to validate this approach.
> > >>> > >
> > >>> > > Jorge.
> > >>> > >
> > >>> > >
> > >>> > > On Sat, Jun 27, 2020 at 4:01 PM John Roesler <
> vvcep...@apache.org>
> > >>> wrote:
> > >>> > >
> > >>> > >> Hi Jorge,
> > >>> > >>
> > >>> > >> Sorry for my silence, I've been absorbed with the 2.6 and 2.5.1
> > >>> releases.
> > >>> > >>
> > >>> > >> The idea to separate the new methods into "mixin" interfaces
> seems
> > >>> > >> like a good one, but as we've discovered in KIP-614, it doesn't
> work
> > >>> > >> out that way in practice. The problem is that the store
> > >>> implementations
> > >>> > >> are just the base layer that get composed with other layers in
> > >>> Streams
> > >>> > >> before they can be accessed in the DSL. This is extremely
> subtle, so
> > >>> > >> I'm going to put everyone to sleep with a detailed explanation:
> > >>> > >>
> > >>> > >> For example, this is the mechanism by which all KeyValueStore
> > >>> > >>