Build failed in Jenkins: Kafka » kafka-2.5-jdk8 #11

[Apache Jenkins Server]

See 


Changes:

[github] Backport Jenkinsfile to 2.5 (#9327)


--
[...truncated 3.11 MB...]

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithCustomTimestampAndIncrementsAndNotAdvanceTime
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithTimestampWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKeyAndDefaultTimestamp
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairsWithTimestampAndIncrements PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

Re: [DISCUSS] Release Deadlines

[Matthias J. Sax]

I added one week between KIP freeze and feature freeze as my observation
was that we tend to vote some last minutes KIPs and than rush the
implementation within one week. Having two weeks to implement last
minute KIPs should lead the a more stable release branch when we cut it
after feature freeze, with hopefully fewer last minute critical/blocker
bugs.

I also added one more week after code freeze, because this is the period
we need to just stabilize the release branch. We had many blockers in
the last releases that delayed the whole process. Thus, accommodating
one more week seems to be an adjustment to reality. And because we only
merge blocker fixes, it should not introduce any new risks.

Let me know if this reasoning make sense. In the end, it's a proposal --
I am more than happy to consider other ideas. I just wanted to put out
something concrete, instead of a vague "we should change something"
statement.


-Matthias


On 9/30/20 8:38 PM, Ismael Juma wrote:
> Thanks for proposing this Matthias. A couple of questions:
> 
> 1. How did you decide where to increase the time?
> 2. Do you think there's a risk that having more time won't necessarily
> help, we will just try to fit more things? I've seen that happen in similar
> circumstances.
> 
> Ismael
> 
> On Tue, Sep 29, 2020, 7:29 PM Matthias J. Sax  wrote:
> 
>> Hi,
>>
>> when we introduced time based releases, we added certain deadlines to
>> streamline the release process and to make sure we can ship the release
>> on time. Based on early experience, we adjusted those deadlines and
>> introduced new deadlines which improved the situation.
>>
>> However, we still have the issue that it often takes very long to
>> stabilize a release branch and the release was delayed by several weeks.
>>
>> Thus, I am wondering if we should adjust those deadlines again.
>> Currently, we have
>>
>>  - KIP freeze
>>  - Feature freeze (+1 week)
>>  - Code freeze (+2 weeks)
>>  - Target release date (+2 weeks)
>>
>> I would like to propose to extend the deadlines as follows:
>>
>>  - KIP freeze
>>  - Feature freeze (+2 weeks)
>>  - Code freeze (+2 weeks)
>>  - Target release date (+3 weeks)
>>
>> This would give us 2 more weeks. Note, that we would not put the target
>> release date 2 week later, but put KIP freeze 2 weeks earlier.
>>
>> It does of course not come for free. In particular, having 2 weeks
>> (instead of 1 week) between feature freeze and code freeze implies a
>> longer period when PR needs to be double committed. However, from my
>> personal experience, I don't think that this burden on committers it too
>> high.
>>
>> Looking forward to your feedback.
>>
>>
>> -Matthias
>>
> 

Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema

[Gwen Shapira]

Thank you, this will be quite helpful.

+1 (binding)

On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela  wrote:
>
> Hi everyone,
>
> Thanks again for the discussion. I'd like to start the vote for this KIP.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
>
> Thanks,
> Anastasia



-- 
Gwen Shapira
Engineering Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

[Guozhang Wang]

Hello Jose,

Thanks for the replies and the KIP updates. Just want to clarify one more
thing regarding my previous comment 3): I understand that when a snapshot
has completed loading, then we can use it in our handling logic of vote
request. And I understand that:

1) Before a snapshot has been completely received (e.g. if we've only
received a subset of the "chunks"), then we just handle vote requests "as
like" there's no snapshot yet.

2) After a snapshot has been completely received and loaded into main
memory, we can handle vote requests "as of" the received snapshot.

What I'm wondering if, in between of these two synchronization barriers,
after all the snapshot chunks have been received but before it has been
completely parsed and loaded into the memory's metadata cache, if we
received a request (note they may be handled by different threads, hence
concurrently), what should we do? Or are you proposing that the
fetchSnapshot request would also be handled in that single-threaded raft
client loop so it is in order with all other requests, if that's the case
then we do not have any concurrency issues to worry, but then on the other
hand the reception of the last snapshot chunk and loading them to main
memory may also take long time during which a client may not be able to
handle any other requests.


Guozhang

On Wed, Sep 30, 2020 at 10:57 AM Jun Rao  wrote:

> Hi, Jose,
>
> Thanks for the updated KIP. A few more comments below.
>
> 40. LBO: Code wise, logStartOffset is used in so many places like Log,
> ReplicaManager, LogCleaner, ReplicaFetcher, checkpoint files, etc. I am not
> if it's worth renaming in all those places. If the main concern is to
> avoid confusion, we can just always spell out logStartOffset.
>
> 41. metadata.snapshot.min.cleanable.ratio: Since the snapshot could be
> empty initially, it's probably better to define the ratio as new_data_size
> / (new_data_size + snapshot_size). This avoids the dividing by zero issue
> and is also consistent with the cleaner ratio definition for compacted
> topics.
>
> 42. FetchSnapshotRequest: Since this is designed to fetch more than one
> partition, it seems it's useful to have a per-partition maxBytes, in
> addition to the request level maxBytes, just like a Fetch request?
>
> 43. FetchSnapshotResponse:
> 43.1 I think the confusing part for OFFSET_OUT_OF_RANGE is
> that FetchSnapshotRequest includes EndOffset. So, OFFSET_OUT_OF_RANGE seems
> to suggest that the provided EndOffset is wrong, which is not the intention
> for the error code.
> 43.1 Position field seems to be the same as the one in
> FetchSnapshotRequest. If we have both, should the requester verify the
> consistency between two values and what should the requester do if the two
> values don't match?
>
> 44. metric: Would a metric that captures the lag in offset between the last
> snapshot and the logEndOffset be useful?
>
> 45. It seems the KIP assumes that every voter (leader and follower) and
> observer has a local replicated log for __cluster_metadata. It would be
> useful to make that clear in the overview section.
>
> 46. Does this KIP cover upgrading from older versions of Kafka? If so, do
> we need IBP to guard the usage of modified FetchRequest and the new
> FetchSnapshotRequest? If not, could we make it clear that upgrading will be
> covered somewhere else?
>
> Thanks,
>
> Jun
>
> On Mon, Sep 28, 2020 at 9:25 PM Jose Garcia Sancio 
> wrote:
>
> > Hi Guozhang,
> >
> > Thanks for your feedback. It was very helpful. See my comments below.
> >
> > Changes to the KIP:
> >
> >
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=28=27
> >
> > On Sun, Sep 27, 2020 at 9:02 PM Guozhang Wang 
> wrote:
> > >
> > > Hello Jose,
> > >
> > > Thanks for the KIP. Overall it looks great. I have a few meta / minor
> > > question, or maybe just clarifications below:
> > >
> > > Meta:
> > >
> > > 1. I want to clarify that if only the active controller would generate
> > > snapshots, OR would any voter in the quorum would generate snapshots,
> OR
> > > would even observers generate snapshots? Originally I thought it was
> the
> > > latter case, but I think reading through the doc I got confused by some
> > > paragraphs. E.g. you mentioned snapshots are generated by the
> Controller
> > > module, and observers would not have that module.
> >
> > Sorry for the confusion and inconsistency here. Every replica of the
> > cluster metadata topic partition will generate a snapshot. That
> > includes the voters (leader and followers) and observers. In this KIP
> > the leader is the Active Controller, the voters are the Kafka
> > Controllers and the observers are the Metadata Cache.
> >
> > I went through the KIP again and made sure to enumerate both Kafka
> > Controllers and Metadata Cache when talking about snapshot generation
> > and loading.
> >
> > I renamed the new configurations to be prefixed by metadata instead of
> > controller.
> >
> > I moved the terminology 

Re: [DISCUSS] Release Deadlines

[Ismael Juma]

Thanks for proposing this Matthias. A couple of questions:

1. How did you decide where to increase the time?
2. Do you think there's a risk that having more time won't necessarily
help, we will just try to fit more things? I've seen that happen in similar
circumstances.

Ismael

On Tue, Sep 29, 2020, 7:29 PM Matthias J. Sax  wrote:

> Hi,
>
> when we introduced time based releases, we added certain deadlines to
> streamline the release process and to make sure we can ship the release
> on time. Based on early experience, we adjusted those deadlines and
> introduced new deadlines which improved the situation.
>
> However, we still have the issue that it often takes very long to
> stabilize a release branch and the release was delayed by several weeks.
>
> Thus, I am wondering if we should adjust those deadlines again.
> Currently, we have
>
>  - KIP freeze
>  - Feature freeze (+1 week)
>  - Code freeze (+2 weeks)
>  - Target release date (+2 weeks)
>
> I would like to propose to extend the deadlines as follows:
>
>  - KIP freeze
>  - Feature freeze (+2 weeks)
>  - Code freeze (+2 weeks)
>  - Target release date (+3 weeks)
>
> This would give us 2 more weeks. Note, that we would not put the target
> release date 2 week later, but put KIP freeze 2 weeks earlier.
>
> It does of course not come for free. In particular, having 2 weeks
> (instead of 1 week) between feature freeze and code freeze implies a
> longer period when PR needs to be double committed. However, from my
> personal experience, I don't think that this burden on committers it too
> high.
>
> Looking forward to your feedback.
>
>
> -Matthias
>

[jira] [Resolved] (KAFKA-10326) Both serializer and deserializer should be able to see the generated client id

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-10326.
---
Fix Version/s: 2.7.0
   Resolution: Fixed

> Both serializer and deserializer should be able to see the generated client id
> --
>
> Key: KAFKA-10326
> URL: https://issues.apache.org/jira/browse/KAFKA-10326
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 2.7.0
>
>
> Producer and consumer generate client id when users don't define it. the 
> generated client id is passed to all configurable components (for example, 
> metrics reporter) except for serializer/deseriaizer.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Release Deadlines

[John Roesler]

Thanks for this, Matthias!

I’m in favor. I’m glad we have all been working hard to stabilize our releases 
and offer a high quality project, even if it means delays. However, the last 
several releases have demonstrated that we consistently need more time than we 
have allocated. Rather than risk building an expectation that our releases will 
always be delayed, we should just move up the deadlines.

Thanks,
John

On Tue, Sep 29, 2020, at 21:29, Matthias J. Sax wrote:
> Hi,
> 
> when we introduced time based releases, we added certain deadlines to
> streamline the release process and to make sure we can ship the release
> on time. Based on early experience, we adjusted those deadlines and
> introduced new deadlines which improved the situation.
> 
> However, we still have the issue that it often takes very long to
> stabilize a release branch and the release was delayed by several weeks.
> 
> Thus, I am wondering if we should adjust those deadlines again.
> Currently, we have
> 
>  - KIP freeze
>  - Feature freeze (+1 week)
>  - Code freeze (+2 weeks)
>  - Target release date (+2 weeks)
> 
> I would like to propose to extend the deadlines as follows:
> 
>  - KIP freeze
>  - Feature freeze (+2 weeks)
>  - Code freeze (+2 weeks)
>  - Target release date (+3 weeks)
> 
> This would give us 2 more weeks. Note, that we would not put the target
> release date 2 week later, but put KIP freeze 2 weeks earlier.
> 
> It does of course not come for free. In particular, having 2 weeks
> (instead of 1 week) between feature freeze and code freeze implies a
> longer period when PR needs to be double committed. However, from my
> personal experience, I don't think that this burden on committers it too
> high.
> 
> Looking forward to your feedback.
> 
> 
> -Matthias
>

Re: [DISCUSS] KIP-674: API to Aggregate Metrics in Kafka Streams

[Guozhang Wang]

Hello Bruno,

Thanks for the proposed KIP! Here are some thoughts:

1) Regarding Sophie's comment 1), I think there may be one merit to
defining the aggregated metric on a different level (though only on a
higher level would generally make sense). Although in order for the
aggregated metric to report any values, we are still bound to the reporting
level of its underlying metrics, but for the metrics reporter we can
potentially leverage on the level, to define e.g. that "only collect
metrics at the INFO level". Today since the reporting level is not exposed
from Metric yet, we cannot do that. Without this meric, I feel it indeed
makes less sense to allow users defining the level of the aggregated metric
to be different from the underlying metrics' level.

2) Regarding the rejected alternative, personally I'm still a bit hesitant
to put a metrics-related API at the entry class KafkaStreams. WDYT about
adding an API to expose StreamsMetrics in KafkaStreams, and add this
function in StreamsMetricsImpl, which would check if the state of the
client is in CREATED, otherwise throw an illegal state exception to avoid
concurrency? We can pass in the KafkaStreams reference to the
StreamsMetricsImpl constructor so that its state can be accessed in this
function.


Guozhang



On Wed, Sep 30, 2020 at 5:34 PM Sophie Blee-Goldman 
wrote:

> Thanks for the KIP! This looks really useful
>
> 1)
>
> > If the recording level for the application is set to INFO, a DEBUG-level
> > metric that should be aggregated will
> > not record values even if the metrics that records the aggregation is on
> > recording level INFO
>
>
> I think I'm missing something here: how is it possible for there to be a
> "DEBUG-level metric
> that should be aggregated" and yet "the metrics that records the
> aggregation is on INFO" ?
> I thought that each metric to be aggregated would be uniquely defined by
> the combination
> of metricName + metricGroup, so there wouldn't be both an INFO and DEBUG
> level
> metric  to distinguish between.
>
> It seems like if a user tries to aggregate a DEBUG-level metric but passes
> in the recording
> level as INFO, this would be a misconfiguration. Can you give an example to
> help me
> understand this?
>
> 2) Why do we require users to supply an aggregator and initializer?
> Shouldn't
> that be determined solely by the actual metric being aggregated? For
> example
> if a user wants to aggregate the `commit-latency-avg`, then the only
> sensible
> way to aggregate this is by averaging. Similarly the only sensible
> aggregation
> for `commit-latency-max` is to take the max, a `-total` metric should be
> summed,
> and so on.
>
> If it's not always possible to infer the aggregation type, then WDYT about
> providing
> a few aggregation options rather than a fully pluggable BiFunction that
> users have
> to implement themselves? It seems like the vast majority of aggregations
> would be one of {avg, min, max, sum}, so we should just give users those
> options
> directly. For. example instead of the `initalAggregate` and
> `aggregateFunction` parameters,
> we just have an enum with possible values AVERAGE, MIN, MAX, and SUM
> (am I missing any in that list?)
>
> If a user really wants to do something more complicated, they can always
> roll it
> up themselves. And if people ask for it we can also go back and add the
> pluggable
> BiFunction option as well. I'd just rather keep things as simple for users
> as possible,
> until we've heard actual feedback that more complicated options are
> desired.
>
> 3) nit: I feel the name `tagLabels` is a bit subtle, what about
> `tagsToAggregate`?
> (just a suggestion, feel free to ignore)
>
> 4) Also a bit of a nit: why put all the configs in the
> MetricsAggregationConfig class,
> except for the groupOfMetricsToAggregate and nameOfMetricsToAggregate? It
> seems like they are just as much a config as the `tagLabels`, for example.
>
> WDYT?
>
> Sophie
>
> On Wed, Sep 30, 2020 at 4:49 AM Bruno Cadonna  wrote:
>
> > Hi,
> >
> > I would like to propose the following KIP to add an API to the Kafka
> > Streams client to aggregate metrics.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-674%3A+API+to+Aggregate+Metrics+in+Kafka+Streams
> >
> > Best,
> > Bruno
> >
>


-- 
-- Guozhang

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #101

[Apache Jenkins Server]

See 


Changes:

[cshapi] KAFKA-10390; Remove ignore case option when grep process info to be 
more specific

[github] KAFKA-10277: Allow null keys with non-null mappedKey in 
KStreamKGlobalTable join (#9186)

[github] KAFKA-10205: Documentation and handling of non deterministic 
Topologies (#9064)

[github] KAFKA-9274: Revert deprecation of `retries` for producer and admin 
clients (#9333)


--
[...truncated 2.31 MB...]

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

Re: [DISCUSS] KIP-674: API to Aggregate Metrics in Kafka Streams

[Sophie Blee-Goldman]

Thanks for the KIP! This looks really useful

1)

> If the recording level for the application is set to INFO, a DEBUG-level
> metric that should be aggregated will
> not record values even if the metrics that records the aggregation is on
> recording level INFO


I think I'm missing something here: how is it possible for there to be a
"DEBUG-level metric
that should be aggregated" and yet "the metrics that records the
aggregation is on INFO" ?
I thought that each metric to be aggregated would be uniquely defined by
the combination
of metricName + metricGroup, so there wouldn't be both an INFO and DEBUG
level
metric  to distinguish between.

It seems like if a user tries to aggregate a DEBUG-level metric but passes
in the recording
level as INFO, this would be a misconfiguration. Can you give an example to
help me
understand this?

2) Why do we require users to supply an aggregator and initializer?
Shouldn't
that be determined solely by the actual metric being aggregated? For example
if a user wants to aggregate the `commit-latency-avg`, then the only
sensible
way to aggregate this is by averaging. Similarly the only sensible
aggregation
for `commit-latency-max` is to take the max, a `-total` metric should be
summed,
and so on.

If it's not always possible to infer the aggregation type, then WDYT about
providing
a few aggregation options rather than a fully pluggable BiFunction that
users have
to implement themselves? It seems like the vast majority of aggregations
would be one of {avg, min, max, sum}, so we should just give users those
options
directly. For. example instead of the `initalAggregate` and
`aggregateFunction` parameters,
we just have an enum with possible values AVERAGE, MIN, MAX, and SUM
(am I missing any in that list?)

If a user really wants to do something more complicated, they can always
roll it
up themselves. And if people ask for it we can also go back and add the
pluggable
BiFunction option as well. I'd just rather keep things as simple for users
as possible,
until we've heard actual feedback that more complicated options are
desired.

3) nit: I feel the name `tagLabels` is a bit subtle, what about
`tagsToAggregate`?
(just a suggestion, feel free to ignore)

4) Also a bit of a nit: why put all the configs in the
MetricsAggregationConfig class,
except for the groupOfMetricsToAggregate and nameOfMetricsToAggregate? It
seems like they are just as much a config as the `tagLabels`, for example.

WDYT?

Sophie

On Wed, Sep 30, 2020 at 4:49 AM Bruno Cadonna  wrote:

> Hi,
>
> I would like to propose the following KIP to add an API to the Kafka
> Streams client to aggregate metrics.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-674%3A+API+to+Aggregate+Metrics+in+Kafka+Streams
>
> Best,
> Bruno
>

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #103

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10351: Add tests for IOExceptions for 
GlobalStateManagerImpl/OffsetCheckpoint (#9121)

[cshapi] KAFKA-10390; Remove ignore case option when grep process info to be 
more specific

[github] KAFKA-10277: Allow null keys with non-null mappedKey in 
KStreamKGlobalTable join (#9186)

[github] KAFKA-10205: Documentation and handling of non deterministic 
Topologies (#9064)

[github] KAFKA-9274: Revert deprecation of `retries` for producer and admin 
clients (#9333)


--
[...truncated 2.33 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureSinkTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #135

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10351: Add tests for IOExceptions for 
GlobalStateManagerImpl/OffsetCheckpoint (#9121)

[cshapi] KAFKA-10390; Remove ignore case option when grep process info to be 
more specific

[github] KAFKA-10277: Allow null keys with non-null mappedKey in 
KStreamKGlobalTable join (#9186)

[github] KAFKA-10205: Documentation and handling of non deterministic 
Topologies (#9064)

[github] KAFKA-9274: Revert deprecation of `retries` for producer and admin 
clients (#9333)


--
[...truncated 2.33 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

Jenkins build is back to normal : Kafka » kafka-trunk-jdk8 #100

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-631: The Quorum-based Kafka Controller

[Colin McCabe]

On Tue, Sep 29, 2020, at 17:43, Jason Gustafson wrote:
> Hey Colin,
> 
> Thanks for the hard work on this proposal.
> 
> I'm gradually coming over to the idea of the controllers having separate
> IDs. One of the benefits is that it allows us to separate the notion of
> controller liveness from broker liveness, which has always been a tricky
> detail. I think it's fair to say that the liveness of the controller is
> tracked through the Raft protocol while the liveness of the broker is
> tracked by the heartbeating mechanism in this KIP. This saves from having
> to deal with tricky cases like electing a controller which is not actually
> live from the perspective of heartbeating. I suspect you are right that it
> will be simpler internally if we treat them distinctly, though it will take
> some getting used to for users. It also seemingly makes it easier to
> migrate to a dedicated controller setup once a cluster gets large enough
> for it.
> 
> With that said, one detail which is not very clear to me is how these two
> IDs interact with the metadata quorum. Let's say we have a broker which is
> running as both a broker and a controller. I think it must send fetches
> using `controller.id` so that the leader can track its progress and it can
> be eligible for election. Will it also fetch using `broker.id`? If we could
> avoid replicating the metadata twice, that would be preferable, but it
> seems like that will break the isolation between the controller and broker
> at some level.
>

Hi Jason,

Thanks for taking a look.

I agree that it would be good to avoid replicating the metadata twice on disk.  
This is one place where I think it's OK to break the isolation between the 
controller and the broker.  This is also one benefit of separating the metadata 
fetches from the heartbeats -- a broker which is co-located with a controller 
will still make broker heartbeats to the controller quorum, but won't perform 
metadata fetches... since the controller part of the process is already doing 
that and sharing the results...

> 
> There is another option I was thinking about for the sake of discussion.
> Suppose we say that controllers always persist the metadata log and brokers
> never do. A broker would always start from the latest snapshot, but we can
> allow it to fetch from the "nearest" controller (which might be local if
> `process.roles` is set to both controller and broker). If users want to
> have the metadata log replicated to all nodes, then they can make each node
> a controller. It's fine to have controllers that are not voters since they
> can be observers. They replicate and persist the metadata log, but they
> do not take part in elections, though they would be available for observer
> promotion when we get around to completing the raft reassignment work.
> 

That's an interesting idea.  However, I don't think you would want every node 
to maintain the controller data structures in memory.  Those data structures 
are optimized for taking over as the active controller if necessary.  That's 
different than the MetadataCache on the brokers, which is optimized for being 
accessed concurrently by lots of request handler threads.

>
> On a related note, I'm still slightly in favor of unifying the controller
> listener into the existing `listeners` configuration. I think we would
> agree that separating the controller listener should at least be considered
> a best practice in a secure configuration, but I am not sure about the case
> for mandating it. I'm sympathetic to the idea that we should be opinionated
> about this, but it would be helpful if the KIP documented what exactly we
> are getting out of it. My concern is basically that it hurts usability.
> 

I don't think having separate ports hurts usability.  Anyone using a quickstart 
docker image or default configuration can just leave this as the default, which 
will be something like 0.0.0.0:9093.  Users are already setting controller.id 
and node.roles differently on controller nodes in any case.

Sharing the same port doesn't really work.  You wouldn't know whether anyone 
contacting wanted to connect to the broker or the controller.  Then you can't 
have tighter security on the controller than on the broker (or indeed any 
security differences on the controller vs. the broker).  And so on, and so on, 
for metrics, backpressure, quotas, etc.

There is also the issue with clients flooding the controller port.  There's no 
reason to expose ourselves to that.  It's just a trap for novice admins, which 
is actually a negative for usability, I think.

>
> By the way, in KIP-595 we tended to favor `quorum` as the prefix for
> configurations related to the management of the metadata quorum. None of
> these configs have been exposed yet, so we can still change them. I think
> `controller` is probably more descriptive, so if you're in agreement, we
> can amend KIP-595 so that it uses the `controller` prefix. However, I think
> it's important to 

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #102

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-6585: Consolidate duplicated logic on reset tools (#9255)


--
[...truncated 6.70 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithPersistentStore PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutAllWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldPutIfAbsentWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldForwardInit 
STARTED

[jira] [Created] (KAFKA-10561) Support microseconds precision for Timestamps

[Daniel Petisme (Jira)]

Daniel Petisme created KAFKA-10561:
--

 Summary: Support microseconds precision for Timestamps
 Key: KAFKA-10561
 URL: https://issues.apache.org/jira/browse/KAFKA-10561
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Daniel Petisme


Kafka connect model Timestamp logical type has a milliseconds precision where 
AVRO provide both milli and micro seconds precision (for both Time and 
Timestamp).

 

I faced the issue when using Qlik Replicate (former Attunity) Change Data 
Capture (CDC)tool. The CDC serializes timestamp with a microseconds precision 
and provide the official `timestamp-micros` logical type.

Due to the lack of support, SMTs like TimestampConverter fallback to a Long 
representation (since internally the `timestamp-micros`) does not exists and 
the conversion to java.utilDate is wrong.

I can't check myself but I heard that IBM CDC was also using `timestamp-micros`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10560) Request for AK bin command to get AK cluster ID

[Yeva Byzek (Jira)]

Yeva Byzek created KAFKA-10560:
--

 Summary: Request for AK bin command to get AK cluster ID
 Key: KAFKA-10560
 URL: https://issues.apache.org/jira/browse/KAFKA-10560
 Project: Kafka
  Issue Type: Improvement
Reporter: Yeva Byzek


This Jira is a request for AK bin command to _cleanly_ get AK cluster ID, 
without going to ZooKeeper.

 

Looking for an equivalent to `describeCluster` functionality in the AdminAPI, 
but from an AK command.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-516: Topic Identifiers

[Jun Rao]

Hi, Justine,

Thanks for the summary. Just a few minor comments blow.

30.  {"name": "id", "type": "string", "doc": "version id"}}: The doc should
say UUID. The issue seems unfixed.

40. Since UUID is public facing, could you include its definition?

41. StopReplicaResponse still includes the topic field.

42. "It is unnecessary to include the name of the topic in the following
Request/Response calls" It would be useful to include all modified requests
(e.g. produce) in the list below.

Thanks,

Jun


On Wed, Sep 30, 2020 at 10:17 AM Justine Olshan 
wrote:

> Hello again,
>
> I've taken some time to discuss some of the remaining points brought up by
> the previous emails offline. Here are some of the conclusions.
>
> 1. Directory Structure:
> There was some talk about whether the directory structure should be changed
> to replace all topic names with topic IDs.
> This will be a useful change, but will prevent downgrades. It will be best
> to wait until a major release, likely alongside KIP-500 changes that will
> prevent downgrades. I've updated the KIP to include this change with the
> note about migration and deprecation.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-log.dirlayout
>
> 2. Partition Metadata file
> There was some disagreement about the inclusion of the partition metadata
> file.
> This file will be needed to persist the topic ID, especially while we still
> have the old directory structure. Even after the changes, this file can be
> useful for debugging and recovery.
> Creating a single mapping file from topic names to topic IDs was
> considered, but ultimately scrapped as it would not be as easy to maintain.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-PartitionMetadatafile
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-PersistingTopicIDs
>
> 3. Produce Protocols
> After some further discussion, this replacing the topic name with topic ID
> in these protocols has been added to the KIP.
> This will cut down on the size of the protocol. Since changes to fetch are
> included, it does make sense to update these protocols.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-Produce
>
> 4. KIP-500 Compatibility
> After some discussion with those involved with KIP-500, it seems best to
> use a sentinel topic ID for the metadata topic that is used before the
> first controller election.
> We had an issue where this metadata topic may not be assigned an ID before
> utilizing the Vote and Fetch protocols. It was decided to reserve a unique
> ID for this topic to be used until the controller could give the topic a
> unique ID.
> Having the topic keep the sentinel ID (not be reassigned to a unique ID)
> was considered, but it seemed like a good idea to leave the option open for
> the metadata topic to have a unique ID in cases where it would need to be
> differentiated from other clusters' metadata topics. (ex. tiered storage)
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-CompatibilitywithKIP-500
>
> I've also split up the KIP into sub-tasks on the JIRA. Hopefully this will
> provide a better idea about what tasks we have, and eventually provide a
> place to see what's done and what is left.
> If there is a task I am missing, please let me know!
> https://issues.apache.org/jira/browse/KAFKA-8872
>
> Of course, these decisions are not set in stone, and I would love to hear
> any feedback.
>
> Thanks,
> Justine
>
> On Mon, Sep 28, 2020 at 11:38 AM Justine Olshan 
> wrote:
>
> > Hello all,
> >
> > I just wanted to follow up on this discussion. Did we come to an
> > understanding about the directory structure?
> >
> > I think the biggest question here is what is acceptable to leave out due
> > to scope vs. what is considered to be too much tech debt.
> > This KIP is already pretty large with the number of changes, but it also
> > makes sense to do things correctly, so I'd love to hear everyone's
> thoughts.
> >
> > Thanks,
> > Justine
> >
> > On Fri, Sep 25, 2020 at 8:19 AM Lucas Bradstreet 
> > wrote:
> >
> >> Hi Ismael,
> >>
> >> If you do not store it in a metadata file or in the directory structure
> >> would you then
> >> require the LeaderAndIsrRequest following startup to give you some
> notion
> >> of
> >> topic name in memory? We would still need this information for the older
> >> protocols, but
> >> perhaps this is what's meant by tech debt.
> >>
> >> Once we're free of the old non-topicID requests then I think you
> wouldn't
> >> need to retain the topic name.
> >> I think the ability to easily look up topic names associated with
> >> partition
> >> directories would still be missed
> >> when diagnosing issues, though maybe it wouldn't be a deal breaker with
> >> the
> >> right tooling.
> >>
> >> 

Re: [DISCUSS] KIP-508: Make Suppression State Queriable - rebooted.

[John Roesler]

Thanks Dongjin,

It typically is nicer to be able to see usage examples, so
I'd certainly be in favor if you're willing to add it to the
KIP.

I'm wondering if it's possible to implement the whole
ReadOnlyKeyValueStore interface as proposed, if we really go
ahead and just internally query into the suppression buffer
as well as using the upstream ValueGetter. The reason is
twofold:
1. The suppression buffer is ordered by arrival time, not by
key. There is a by-key index, but it is also not ordered the
same way that in-memory stores are ordered. Thus, we'd have
a hard time implementing key-based range scans.
2. The internal ValueGetter interface only supports get-by-
key lookups, so it would also need to be expanded to support
range scans on the parent table.

Neither of these problems are insurmountable, but I'm
wondering if we _want_ to surmount them right now. Or should
we instead just throw an UnsupportedOperationException on
any API call that's inconvenient to implement right now?
Then, we could get incremental value by first supporting
(eg) key-based lookups and adding scans later.

Or does this mean that our design so far is invalid, and we
should really just make people provision a separate
Materialized downstream? To pull this off, we'd need to
first address KIP-300's challenges, though.

I'm honestly not sure what the right call is here.

Thanks,
-John

On Thu, 2020-10-01 at 01:50 +0900, Dongjin Lee wrote:
> > It seems like it must be a ReadOnlyKeyValueStore. Does that sound right?
> 
> Yes, it is. Would it be better to add a detailed description of how this
> feature effects interactive query, with examples?
> 
> Best,
> Dongjin
> 
> On Tue, Sep 29, 2020 at 10:31 AM John Roesler  wrote:
> 
> > Hi Dongjin,
> > 
> > Thanks! Sorry, I missed your prior message. The proposed API looks good to
> > me.
> > 
> > I’m wondering if we should specify what kind of store view would be
> > returned when querying the operation result. It seems like it must be a
> > ReadOnlyKeyValueStore. Does that sound right?
> > 
> > Thanks!
> > John
> > 
> > On Mon, Sep 28, 2020, at 10:06, Dongjin Lee wrote:
> > > Hi John,
> > > 
> > > I updated the KIP with the discussion above. The 'Public Interfaces'
> > > section describes the new API, and the 'Rejected Alternatives' section
> > > describes the reasoning about why we selected this API design and
> > rejected
> > > the other alternatives.
> > > 
> > > Please have a look when you are free. And please note that the KIP freeze
> > > for 2.7.0 is imminent.
> > > 
> > > Thanks,
> > > Dongjin
> > > 
> > > On Mon, Sep 21, 2020 at 11:35 PM Dongjin Lee  wrote:
> > > 
> > > > Hi John,
> > > > 
> > > > I updated the PR applying the API changes we discussed above. I am now
> > > > updating the KIP document.
> > > > 
> > > > Thanks,
> > > > Dongjin
> > > > 
> > > > On Sat, Sep 19, 2020 at 10:42 AM John Roesler 
> > wrote:
> > > > > Hi Dongjin,
> > > > > 
> > > > > Yes, that’s right. My the time of KIP-307, we had no choice but to
> > add a
> > > > > second name. But we do have a choice with Suppress.
> > > > > 
> > > > > Thanks!
> > > > > -John
> > > > > 
> > > > > On Thu, Sep 17, 2020, at 13:14, Dongjin Lee wrote:
> > > > > > Hi John,
> > > > > > 
> > > > > > I just reviewed KIP-307. As far as I understood, ...
> > > > > > 
> > > > > > 1. There was Materialized name initially.
> > > > > > 2. With KIP-307, Named Operations were added.
> > > > > > 3. Now we have two options for materializing suppression. If we take
> > > > > > Materialized name here, we have two names for the same operation,
> > which
> > > > > is
> > > > > > not feasible.
> > > > > > 
> > > > > > Do I understand correctly?
> > > > > > 
> > > > > > > Do you have a use case in mind for having two separate names for
> > the
> > > > > > operation and the view?
> > > > > > 
> > > > > > No. I am now entirely convinced with your suggestion.
> > > > > > 
> > > > > > I just started to update the draft implementation. If I understand
> > > > > > correctly, please notify me; I will update the KIP by adding the
> > > > > discussion
> > > > > > above.
> > > > > > 
> > > > > > Best,
> > > > > > Dongjin
> > > > > > 
> > > > > > On Thu, Sep 17, 2020 at 11:06 AM John Roesler 
> > > > > wrote:
> > > > > > > Hi Dongjin,
> > > > > > > 
> > > > > > > Thanks for the reply. Yes, that’s correct, we added that method to
> > > > > name
> > > > > > > the operation. But the operation seems synonymous with the view
> > > > > produced
> > > > > > > the operation, right?
> > > > > > > 
> > > > > > > During KIP-307, I remember thinking that it’s unfortunate the we
> > had
> > > > > to
> > > > > > > have two different “name” concepts for the same thing just because
> > > > > setting
> > > > > > > the name on Materialized is equivalent both to making it
> > queriable and
> > > > > > > actually materializing it.
> > > > > > > 
> > > > > > > If we were to reconsider the API, it would be nice to treat these
> > > > > three as
> > > > > > > orthogonal:
> > 

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #21

[Apache Jenkins Server]

See 


Changes:

[John Roesler] MINOR: Do not swallow exception when collecting PIDs (#8914)


--
[...truncated 6.31 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > 

[jira] [Created] (KAFKA-10559) Don't shutdown the entire app upon TimeoutException during internal topic validation

[Sophie Blee-Goldman (Jira)]

Sophie Blee-Goldman created KAFKA-10559:
---

 Summary: Don't shutdown the entire app upon TimeoutException 
during internal topic validation
 Key: KAFKA-10559
 URL: https://issues.apache.org/jira/browse/KAFKA-10559
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Sophie Blee-Goldman
 Fix For: 2.7.0


During some of the KIP-572 work, we made things pretty brittle by changing the 
StreamsPartitionAssignor to send the `INCOMPLETE_SOURCE_TOPIC_METADATA` error 
code and shut down the entire application if a TimeoutException is hit during 
the internal topic creation/validation.

Internal topic validation occurs during every rebalance, and we have seen it 
time out on topic discovery in unstable environments. So shutting down the 
entire application seems like a step in the wrong direction, and antithetical 
to the goal of KIP-572 (improving the resiliency of Streams in the face of 
TimeoutExceptions)

I'm not totally sure what the previous behavior was, but it seems to me we have 
three options:
 # Rethrow the TimeoutException and allow it to kill the thread
 # Swallow the TimeoutException and retry the rebalance indefinitely
 # Some combination of the above: swallow the TimeoutException but don't retry 
indefinitely:
 ## Start a timer and allow retrying rebalances for up the configured 
task.timeout.ms, the timeout config introduced in KIP-572
 ## Retry for some constant number of rebalances

I think if we go with option 3, then shutting down the entire application is 
relatively more palatable, as we have given the environment a chance to 
stabilize.

But, killing the thread still seems preferable, given the two new features that 
are coming out soon: the ability to start up new threads, and the improved 
exception handler that allows the user to choose to shut down the entire 
application if that's really what they want. Once users have this level of 
control over the application, we should allow them to decide how they want to 
handle exceptional cases like this, rather than forcing an option on them (eg 
shutdown everything) 

 

Imo we should fix this before 2.7 comes out, even if it's just a partial fix 
(eg we do option 1 in 2.7, but plan to implement option 3 eventually)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #99

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10469: Resolve logger levels hierarchically (#9266)


--
[...truncated 6.65 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPassRecordHeadersIntoSerializersAndDeserializers[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 

Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema

[Lucas Bradstreet]

Hi Anastasia,

This looks like a great change that will make debugging cluster behavior
much easier.

+1 non binding.

Lucas

On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela  wrote:

> Hi everyone,
>
> Thanks again for the discussion. I'd like to start the vote for this KIP.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
>
> Thanks,
> Anastasia
>

[VOTE] KIP-673: Emit JSONs with new auto-generated schema

[Anastasia Vela]

Hi everyone,

Thanks again for the discussion. I'd like to start the vote for this KIP.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema

Thanks,
Anastasia

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #101

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10469: Resolve logger levels hierarchically (#9266)


--
[...truncated 6.71 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureInternalTopicNamesIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldWorkWithInMemoryStore PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithLogging PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching STARTED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 
shouldFailWithCaching PASSED

org.apache.kafka.streams.test.wordcount.WindowedWordCountProcessorTest > 

[jira] [Created] (KAFKA-10558) Fetch Session Cache Performance Improvement

[Alyssa Huang (Jira)]

Alyssa Huang created KAFKA-10558:


 Summary: Fetch Session Cache Performance Improvement
 Key: KAFKA-10558
 URL: https://issues.apache.org/jira/browse/KAFKA-10558
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Alyssa Huang


Make kafka.server.FetchSessionCache implementation faster to help mitigate high 
lock contention as detailed in KAFKA-9401, and to allow for increased cache 
sizes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-630: Kafka Raft Snapshot

[Jun Rao]

Hi, Jose,

Thanks for the updated KIP. A few more comments below.

40. LBO: Code wise, logStartOffset is used in so many places like Log,
ReplicaManager, LogCleaner, ReplicaFetcher, checkpoint files, etc. I am not
if it's worth renaming in all those places. If the main concern is to
avoid confusion, we can just always spell out logStartOffset.

41. metadata.snapshot.min.cleanable.ratio: Since the snapshot could be
empty initially, it's probably better to define the ratio as new_data_size
/ (new_data_size + snapshot_size). This avoids the dividing by zero issue
and is also consistent with the cleaner ratio definition for compacted
topics.

42. FetchSnapshotRequest: Since this is designed to fetch more than one
partition, it seems it's useful to have a per-partition maxBytes, in
addition to the request level maxBytes, just like a Fetch request?

43. FetchSnapshotResponse:
43.1 I think the confusing part for OFFSET_OUT_OF_RANGE is
that FetchSnapshotRequest includes EndOffset. So, OFFSET_OUT_OF_RANGE seems
to suggest that the provided EndOffset is wrong, which is not the intention
for the error code.
43.1 Position field seems to be the same as the one in
FetchSnapshotRequest. If we have both, should the requester verify the
consistency between two values and what should the requester do if the two
values don't match?

44. metric: Would a metric that captures the lag in offset between the last
snapshot and the logEndOffset be useful?

45. It seems the KIP assumes that every voter (leader and follower) and
observer has a local replicated log for __cluster_metadata. It would be
useful to make that clear in the overview section.

46. Does this KIP cover upgrading from older versions of Kafka? If so, do
we need IBP to guard the usage of modified FetchRequest and the new
FetchSnapshotRequest? If not, could we make it clear that upgrading will be
covered somewhere else?

Thanks,

Jun

On Mon, Sep 28, 2020 at 9:25 PM Jose Garcia Sancio 
wrote:

> Hi Guozhang,
>
> Thanks for your feedback. It was very helpful. See my comments below.
>
> Changes to the KIP:
>
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158864763=28=27
>
> On Sun, Sep 27, 2020 at 9:02 PM Guozhang Wang  wrote:
> >
> > Hello Jose,
> >
> > Thanks for the KIP. Overall it looks great. I have a few meta / minor
> > question, or maybe just clarifications below:
> >
> > Meta:
> >
> > 1. I want to clarify that if only the active controller would generate
> > snapshots, OR would any voter in the quorum would generate snapshots, OR
> > would even observers generate snapshots? Originally I thought it was the
> > latter case, but I think reading through the doc I got confused by some
> > paragraphs. E.g. you mentioned snapshots are generated by the Controller
> > module, and observers would not have that module.
>
> Sorry for the confusion and inconsistency here. Every replica of the
> cluster metadata topic partition will generate a snapshot. That
> includes the voters (leader and followers) and observers. In this KIP
> the leader is the Active Controller, the voters are the Kafka
> Controllers and the observers are the Metadata Cache.
>
> I went through the KIP again and made sure to enumerate both Kafka
> Controllers and Metadata Cache when talking about snapshot generation
> and loading.
>
> I renamed the new configurations to be prefixed by metadata instead of
> controller.
>
> I moved the terminology section to the top.
>
> >
> > 2. Following on Jun's previous comment: currently the __consumer_metadata
> > log is replicated on ALL brokers since all voters and observers would
> > replicate that topic. I know this may be out of the scope of this KIP
> but I
> > think maybe only letting the voters to replicate (and periodically
> > truncate) the log while observers only maintain the in-memory state and
> > snapshots is a good trade-off here, assuming snapshot loading is
> relatively
> > fast.
>
> This is a good idea and optimization. It would save a write. I think
> we need to think about the implication to KIP-642, the dynamic quorum
> reassignment KIP, if we end up allowing observers to get "promoted" to
> voters.
>
> >
> > 3. When a raft client is in the middle of loading a snapshot, should it
> > reject any vote / begin-/end-/describe-quorum requests at the time? More
> > generally, while a snapshot is being loaded, how should we treat the
> > current state of the client when handling Raft requests.
>
> Re: requesting votes and granting votes.
>
> In the section "Changes to Leader Election", I think this section was
> improved since your review. I mentioned that the raft client needs to
> look at:
>
> 1. latest/largest snapshot epoch and end offset
> 2. the LEO of the replicated log
>
> The voters should use the latest/largest of these two during the
> election process.
>
> Re: quorum state
>
> For KIP-595 and KIP-630 the snapshot doesn't include any quorum
> information. This may change in KIP-642.
>
> >
> > Minor:
> >
> > 

Re: [DISCUSS] KIP-516: Topic Identifiers

[Justine Olshan]

Hello again,

I've taken some time to discuss some of the remaining points brought up by
the previous emails offline. Here are some of the conclusions.

1. Directory Structure:
There was some talk about whether the directory structure should be changed
to replace all topic names with topic IDs.
This will be a useful change, but will prevent downgrades. It will be best
to wait until a major release, likely alongside KIP-500 changes that will
prevent downgrades. I've updated the KIP to include this change with the
note about migration and deprecation.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-log.dirlayout

2. Partition Metadata file
There was some disagreement about the inclusion of the partition metadata
file.
This file will be needed to persist the topic ID, especially while we still
have the old directory structure. Even after the changes, this file can be
useful for debugging and recovery.
Creating a single mapping file from topic names to topic IDs was
considered, but ultimately scrapped as it would not be as easy to maintain.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-PartitionMetadatafile

https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-PersistingTopicIDs

3. Produce Protocols
After some further discussion, this replacing the topic name with topic ID
in these protocols has been added to the KIP.
This will cut down on the size of the protocol. Since changes to fetch are
included, it does make sense to update these protocols.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-Produce

4. KIP-500 Compatibility
After some discussion with those involved with KIP-500, it seems best to
use a sentinel topic ID for the metadata topic that is used before the
first controller election.
We had an issue where this metadata topic may not be assigned an ID before
utilizing the Vote and Fetch protocols. It was decided to reserve a unique
ID for this topic to be used until the controller could give the topic a
unique ID.
Having the topic keep the sentinel ID (not be reassigned to a unique ID)
was considered, but it seemed like a good idea to leave the option open for
the metadata topic to have a unique ID in cases where it would need to be
differentiated from other clusters' metadata topics. (ex. tiered storage)
https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers#KIP516:TopicIdentifiers-CompatibilitywithKIP-500

I've also split up the KIP into sub-tasks on the JIRA. Hopefully this will
provide a better idea about what tasks we have, and eventually provide a
place to see what's done and what is left.
If there is a task I am missing, please let me know!
https://issues.apache.org/jira/browse/KAFKA-8872

Of course, these decisions are not set in stone, and I would love to hear
any feedback.

Thanks,
Justine

On Mon, Sep 28, 2020 at 11:38 AM Justine Olshan 
wrote:

> Hello all,
>
> I just wanted to follow up on this discussion. Did we come to an
> understanding about the directory structure?
>
> I think the biggest question here is what is acceptable to leave out due
> to scope vs. what is considered to be too much tech debt.
> This KIP is already pretty large with the number of changes, but it also
> makes sense to do things correctly, so I'd love to hear everyone's thoughts.
>
> Thanks,
> Justine
>
> On Fri, Sep 25, 2020 at 8:19 AM Lucas Bradstreet 
> wrote:
>
>> Hi Ismael,
>>
>> If you do not store it in a metadata file or in the directory structure
>> would you then
>> require the LeaderAndIsrRequest following startup to give you some notion
>> of
>> topic name in memory? We would still need this information for the older
>> protocols, but
>> perhaps this is what's meant by tech debt.
>>
>> Once we're free of the old non-topicID requests then I think you wouldn't
>> need to retain the topic name.
>> I think the ability to easily look up topic names associated with
>> partition
>> directories would still be missed
>> when diagnosing issues, though maybe it wouldn't be a deal breaker with
>> the
>> right tooling.
>>
>> Thanks,
>>
>> Lucas
>>
>> On Fri, Sep 25, 2020 at 7:55 AM Ismael Juma  wrote:
>>
>> > Hi Lucas,
>> >
>> > Why would you include the name and id? I think you'd want to remove the
>> > name from the directory name right? Jason's suggestion was that if you
>> > remove the name from the directory, then why would you need the id name
>> > mapping file?
>> >
>> > Ismael
>> >
>> > On Thu, Sep 24, 2020 at 4:24 PM Lucas Bradstreet 
>> > wrote:
>> >
>> > > > 2. Part of the usage of the file is to have persistent storage of
>> the
>> > > topic
>> > > ID and use it to compare with the ID supplied in the LeaderAndIsr
>> > Request.
>> > > There is some discussion in the KIP about changes to the directory
>> > > structure, but I believe directory changes were 

[jira] [Resolved] (KAFKA-6585) Consolidate duplicated logic on reset tools

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-6585.

Fix Version/s: 2.7.0
   Resolution: Fixed

> Consolidate duplicated logic on reset tools
> ---
>
> Key: KAFKA-6585
> URL: https://issues.apache.org/jira/browse/KAFKA-6585
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Guozhang Wang
>Assignee: Mani Jindal
>Priority: Minor
>  Labels: newbie
> Fix For: 2.7.0
>
>
> The consumer reset tool and streams reset tool today shares lot of common 
> logics such as resetting to a datetime etc. We can consolidate them into a 
> common class which directly depend on admin client at simply let these tools 
> to use the class.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-508: Make Suppression State Queriable - rebooted.

[Dongjin Lee]

> It seems like it must be a ReadOnlyKeyValueStore. Does that sound right?

Yes, it is. Would it be better to add a detailed description of how this
feature effects interactive query, with examples?

Best,
Dongjin

On Tue, Sep 29, 2020 at 10:31 AM John Roesler  wrote:

> Hi Dongjin,
>
> Thanks! Sorry, I missed your prior message. The proposed API looks good to
> me.
>
> I’m wondering if we should specify what kind of store view would be
> returned when querying the operation result. It seems like it must be a
> ReadOnlyKeyValueStore. Does that sound right?
>
> Thanks!
> John
>
> On Mon, Sep 28, 2020, at 10:06, Dongjin Lee wrote:
> > Hi John,
> >
> > I updated the KIP with the discussion above. The 'Public Interfaces'
> > section describes the new API, and the 'Rejected Alternatives' section
> > describes the reasoning about why we selected this API design and
> rejected
> > the other alternatives.
> >
> > Please have a look when you are free. And please note that the KIP freeze
> > for 2.7.0 is imminent.
> >
> > Thanks,
> > Dongjin
> >
> > On Mon, Sep 21, 2020 at 11:35 PM Dongjin Lee  wrote:
> >
> > > Hi John,
> > >
> > > I updated the PR applying the API changes we discussed above. I am now
> > > updating the KIP document.
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Sat, Sep 19, 2020 at 10:42 AM John Roesler 
> wrote:
> > >
> > >> Hi Dongjin,
> > >>
> > >> Yes, that’s right. My the time of KIP-307, we had no choice but to
> add a
> > >> second name. But we do have a choice with Suppress.
> > >>
> > >> Thanks!
> > >> -John
> > >>
> > >> On Thu, Sep 17, 2020, at 13:14, Dongjin Lee wrote:
> > >> > Hi John,
> > >> >
> > >> > I just reviewed KIP-307. As far as I understood, ...
> > >> >
> > >> > 1. There was Materialized name initially.
> > >> > 2. With KIP-307, Named Operations were added.
> > >> > 3. Now we have two options for materializing suppression. If we take
> > >> > Materialized name here, we have two names for the same operation,
> which
> > >> is
> > >> > not feasible.
> > >> >
> > >> > Do I understand correctly?
> > >> >
> > >> > > Do you have a use case in mind for having two separate names for
> the
> > >> > operation and the view?
> > >> >
> > >> > No. I am now entirely convinced with your suggestion.
> > >> >
> > >> > I just started to update the draft implementation. If I understand
> > >> > correctly, please notify me; I will update the KIP by adding the
> > >> discussion
> > >> > above.
> > >> >
> > >> > Best,
> > >> > Dongjin
> > >> >
> > >> > On Thu, Sep 17, 2020 at 11:06 AM John Roesler 
> > >> wrote:
> > >> >
> > >> > > Hi Dongjin,
> > >> > >
> > >> > > Thanks for the reply. Yes, that’s correct, we added that method to
> > >> name
> > >> > > the operation. But the operation seems synonymous with the view
> > >> produced
> > >> > > the operation, right?
> > >> > >
> > >> > > During KIP-307, I remember thinking that it’s unfortunate the we
> had
> > >> to
> > >> > > have two different “name” concepts for the same thing just because
> > >> setting
> > >> > > the name on Materialized is equivalent both to making it
> queriable and
> > >> > > actually materializing it.
> > >> > >
> > >> > > If we were to reconsider the API, it would be nice to treat these
> > >> three as
> > >> > > orthogonal:
> > >> > > * specify a name
> > >> > > * flag to make the view queriable
> > >> > > * flag to materialize the view
> > >> > >
> > >> > > That was the context behind my suggestion. Do you have a use case
> in
> > >> mind
> > >> > > for having two separate names for the operation and the view?
> > >> > >
> > >> > > Thanks,
> > >> > > John
> > >> > >
> > >> > > On Wed, Sep 16, 2020, at 11:43, Dongjin Lee wrote:
> > >> > > > Hi John,
> > >> > > >
> > >> > > > It seems like the available alternatives in this point is clear:
> > >> > > >
> > >> > > > 1. Pass queriable name as a separate parameter (i.e.,
> > >> > > > `KTable#suppress(Suppressed, String)`)
> > >> > > > 2. Make use of the Suppression processor name as a queryable
> name by
> > >> > > adding
> > >> > > > `enableQuery` optional flag to `Suppressed`.
> > >> > > >
> > >> > > > However, I doubt the second approach a little bit; As far as I
> > >> know, the
> > >> > > > processor name is introduced in KIP-307[^1] to make debugging
> > >> topology
> > >> > > easy
> > >> > > > and understandable. Since the processor name is an independent
> > >> concept
> > >> > > with
> > >> > > > the materialization, I feel the first approach is more natural
> and
> > >> > > > consistent. Is there any specific reason that you prefer the
> second
> > >> > > > approach?
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Dongjin
> > >> > > >
> > >> > > > [^1]:
> > >> > > >
> > >> > >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Sep 16, 2020 at 11:48 PM John Roesler <
> vvcep...@apache.org>
> > >> > > wrote:
> > >> > > >
> > >> > 

[jira] [Created] (KAFKA-10557) Missing docs when describing topic configs including documentation

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-10557:
--

 Summary: Missing docs when describing topic configs including 
documentation
 Key: KAFKA-10557
 URL: https://issues.apache.org/jira/browse/KAFKA-10557
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 2.6.0, 2.7.0
Reporter: Mickael Maison
Assignee: Mickael Maison


When describing topic or broker configs with the AdminClient, we can request 
the documentation of configuration settings to be included.

This does not work with topic configs. The issue lies in this line:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/AdminManager.scala#L767

This uses a KafkaConfig object to look up topic configs. Hence for 
configurations that have different names between KafkaConfig and LogConfig, no 
documentation is found!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-406: GlobalStreamThread should honor custom reset policy

[Matthias J. Sax]

I guess we need to have some cleanup mechanism for this case anyway,
because, the global thread can enter RESTORING state at any point in
time, and thus, even if we set a flag to pause processing on the
StreamThreads we are subject to a race condition.

Beside that, on a high level I am fine with either "busy waiting" (ie,
just lock the global-store and retry) or setting a flag. However, there
are some trade-offs to consider:

As we need a cleanup mechanism anyway, it might be ok to just use a
single mechanism. -- We should consider the impact in EOS though, as we
might need to wipe out the store of regular tasks for this case. Thus,
setting a flag might actually help to prevent that we repeatably wipe
the store on retries... On the other hand, we plan to avoid wiping the
store in case of error for EOS anyway, and if we get this improvement,
we might not need the flag.

For the client state machine: I would actually prefer to have a
RESTORING state and I would also prefer to pause _all_ tasks. This might
imply that we want a flag. In the past, we allowed to interleave restore
and processing in StreamThread (for regular tasks) what slowed down
restoring and we changed it back to not process any tasks until all
tasks are restored). Of course, in our case we have two different
threads (not a single one). However, the network is still shared, so it
might be desirable to give the full network bandwidth to the global
consumer to restore as fast as possible (maybe an improvement we could
add to `StreamThreads` too, if we have multiple threads)? And as a side
effect, it does not muddy the waters what each client state means.

Thus, overall, I tend to prefer a flag on `StreamThread` as it seems to
provide a cleaner end-to-end solution (and we avoid the dependency to
improve EOS state management).

Btw: I am not sure if we actually need to preserve compatibility for the
state machine? To me, it seems not to be a strict contract, and I would
personally be ok to just change it.


-Matthias


On 9/22/20 11:08 PM, Navinder Brar wrote:
> Thanks a lot John for these suggestions. @Matthias can share your thoughts on 
> the last two comments made in this chain.
> 
> Thanks,Navinder 
> 
> On Monday, 14 September, 2020, 09:03:32 pm IST, John Roesler 
>  wrote:  
>  
>  Hi Navinder,
> 
> Thanks for the reply.
> 
> I wasn't thinking of an _exponential_ backoff, but
> otherwise, yes, that was the basic idea. Note, the mechanism
> would be similar (if not the same) to what Matthias is
> implementing for KIP-572:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-572%3A+Improve+timeouts+and+retries+in+Kafka+Streams
> 
> Regarding whether we'd stay in RUNNING during global
> restoration or not, I can see your point. It seems like we
> have three choices with how we set the state during global
> restoration:
> 1. stay in RUNNING: Users might get confused, since
> processing could get stopped for some tasks. On the other
> hand, processing for tasks not blocked by the global
> restoration could proceed (if we adopt the other idea), so
> maybe it still makes sense.
> 2. transition to REBALANCING: Users might get confused,
> since there is no actual rebalance. However, the current
> state for Kafka Streams during state restoration is actually
> REBALANCING, so it seems people already should understand
> that REBALANCING really means REBALANCING|RESTORING. This
> choice would preseve the existing state machine as well as
> the existing meaning of all states
> 3. add RESTORING: This could clarify the state machine, at
> the expense of breaking compatibility. We could implement a
> migration path by adding a new "state listener" interface
> for the new state machine.
> 
> It seems like option 3 results in the most sensible system,
> but I'm not sure if it's worth the hassle. It certainly
> seems orthogonal to the goal of this KIP. Option 2 is
> probably the best practical choice.
> 
> 
> Regarding _how_ the global state restoration could set a
> flag preventing access to the store... This is indeed the
> central challenge to this new idea. Just throwing out one
> possibility: Once the global thread marks the store for
> restoration, it would throw an exception, such as
> "StoreIsRestoringException" on any access. The processor
> would _not_ catch this exception. Instead, the StreamThread
> would catch it, put this record/task on ice, and re-try it
> later.
> 
> That last mechanism is actually pretty complicated. For
> example, what if the record is already partially processed
> in the topology? We'd have to remember which ProcessorNode
> to resume from when we re-try later.
> 
> This is really where the spiritual overlap with KIP-572
> comes in. Maybe Matthias can share some thoughts.
> 
> Thanks,
> -John
> 
> On Sun, 2020-09-13 at 07:50 +, Navinder Brar wrote:
>>   
>> Hi John,
>>
>>
>>
>>
>>
>>
>>
>> If I understand this correctly, you are proposing to use exponential backoff
>>
>> in globalStore.get() to keep polling the 

[jira] [Created] (KAFKA-10556) NPE if sasl.mechanism is unrecognized

[Ron Dagostino (Jira)]

Ron Dagostino created KAFKA-10556:
-

 Summary: NPE if sasl.mechanism is unrecognized
 Key: KAFKA-10556
 URL: https://issues.apache.org/jira/browse/KAFKA-10556
 Project: Kafka
  Issue Type: Task
Reporter: Ron Dagostino
Assignee: Ron Dagostino


If a client sets an unknown sasl.mechanism value (e.g. mistakenly setting 
"PLAN" instead of "PLAIN") the client sees a NullPointerException that only 
indirectly indicates the nature of the problem.  For example:

java.lang.NullPointerException
at 
org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.sendSaslClientToken(SaslClientAuthenticator.java:430)

It is better to see an exception that directly states what the issue is.  For 
example, the initial version of this PR would provide the following information:

Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed 
to create SaslClient with mechanism PLAN




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10555) Improve client state machine

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-10555:
---

 Summary: Improve client state machine
 Key: KAFKA-10555
 URL: https://issues.apache.org/jira/browse/KAFKA-10555
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


The KafkaStreams client exposes its state to the user for monitoring purpose 
(ie, RUNNING, REBALANCING etc). The state of the client depends on the state(s) 
of the internal StreamThreads that have their own states.

Furthermore, the client state has impact on what the user can do with the 
client. For example, active task can only be queried in RUNNING state and 
similar.

With KIP-671 and KIP-663 we improved error handling capabilities and allow to 
add/remove stream thread dynamically. We allow adding/removing threads only in 
RUNNING and REBALANCING state. This puts us in a "weird" position, because if 
we enter ERROR state (ie, if the last thread dies), we cannot add new threads 
and longer. However, if we have multiple threads and one dies, we don't enter 
ERROR state and do allow to recover the thread.

Before the KIPs the definition of ERROR state was clear, however, with both 
KIPs it seem that we should revisit its semantics.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

[Viktor Somogyi-Vass]

Hi Daniel,

I think in this sense we can use the precedence set with the
KAfkaAdminClient. It has *Result and *Options classes which in this
interpretation are similar in versioning and usage as they transform and
convey the responses of the protocol in a minimalistic API.
I've modified the KIP a bit and created some examples for these event
classes. For now as the implementation I think we can treat this similarly
to KIP-4 (AdminClient) which didn't push implementation for everything but
rather pushed implementing everything to subsequent KIPs as the
requirements become important. In this first KIP we can create the more
important ones (listed in the "Default Implementation") section if that is
fine.

Regarding the response passing: to be honest I feel like that it's not that
strictly related to auditing but I think it's a good idea and could fit
into this API. I think that we should design this current API with this in
mind. Did you have any specific ideas about the implementation?

Viktor

On Tue, Sep 22, 2020 at 9:05 AM Dániel Urbán  wrote:

> An example I had in mind was the ProduceResponse - the auditor might need
> access to the new end offset of the partitions.
> The event-based approach sounds good - new events and fields can be added
> on-demand. Do we need the same versioning strategy we use with the
> requests/responses?
>
> Daniel
>
> Viktor Somogyi-Vass  ezt írta (időpont: 2020.
> szept. 21., H, 14:08):
>
> > Hi Daniel,
> >
> > > If the auditor needs access to the details of the action, one could
> argue
> > that even the response should be passed down to the auditor.
> > At this point I don't think we need to include responses into the
> interface
> > but if you have a use-case we can consider doing that.
> >
> > > Is it feasible to convert the Java requests and responses to public
> API?
> > Well I think that in this case we would need to actually transform a lot
> of
> > classes and that might be a bit too invasive. Although since the protocol
> > itself *is* a public API it might make sense to have some kind of Java
> > representation as a public API as well.
> >
> > > If not, do we have another option to access this info in the auditor?
> > I think one option would be to do what the original KIP-567 was
> > implemented. Basically we could have an AuditEvent interface that would
> > contain request specific data. Its obvious drawback is that it has to be
> > implemented for most of the 40 something protocols but on the upside
> these
> > classes shouldn't be complicated. I can try to do a PoC with this to see
> > how it looks like and whether it solves the problem. To be honest I think
> > it would be better than publishing the request classes as an API because
> > here we're restricting access to only what is necessary.
> >
> > Thanks,
> > Viktor
> >
> >
> >
> > On Fri, Sep 18, 2020 at 8:37 AM Dániel Urbán 
> > wrote:
> >
> > > Hi,
> > >
> > > Thanks for the KIP.
> > >
> > > If the auditor needs access to the details of the action, one could
> argue
> > > that even the response should be passed down to the auditor.
> > > Is it feasible to convert the Java requests and responses to public
> API?
> > > If not, do we have another option to access this info in the auditor?
> > > I know that the auditor could just send proper requests through the API
> > to
> > > the brokers, but that seems like an awful lot of overhead, and could
> > > introduce timing issues as well.
> > >
> > > Daniel
> > >
> > >
> > > Viktor Somogyi-Vass  ezt írta (időpont: 2020.
> > > szept. 16., Sze, 17:17):
> > >
> > > > One more after-thought on your second point (AbstractRequest): the
> > > reason I
> > > > introduced it in the first place was that this way implementers can
> > > access
> > > > request data. A use case can be if they want to audit a change in
> > > > configuration or client quotas but not just acknowledge the fact that
> > > such
> > > > an event happened but also capture the change itself by peeking into
> > the
> > > > request. Sometimes it can be useful especially when people want to
> > trace
> > > > back the order of events and what happened when and not just
> > acknowledge
> > > > that there was an event of a certain kind. I also recognize that this
> > > might
> > > > be a very loose interpretation of auditing as it's not strictly
> related
> > > to
> > > > authorization but rather a way of tracing the admin actions within
> the
> > > > cluster. It even could be a different API therefore but because of
> the
> > > > variety of the Kafka APIs it's very hard to give a method that fits
> > all,
> > > so
> > > > it's easier to pass down the AbstractRequest and the implementation
> can
> > > do
> > > > the extraction of valuable info. So that's why I added this in the
> > first
> > > > place and I'm interested in your thoughts.
> > > >
> > > > On Wed, Sep 16, 2020 at 4:41 PM Viktor Somogyi-Vass <
> > > > viktorsomo...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > 

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

[John Roesler]

Thanks, Matthias!

I can certainly document it. I didn't bother because the old
Processor, Supplier, and Context will themselves be
deprecated, so any method that handles them won't be able to
avoid the deprecation warning. Nevertheless, it doesn't hurt
just to explicitly deprecated those methods.

Thanks,
-John

On Wed, 2020-09-30 at 08:10 -0700, Matthias J. Sax wrote:
> Thanks John. I like the proposal.
> 
> Btw: I was just going over the KIP and realized that we add new methods
> to `StreamBuilder`, `Topology`, and `KStream` that take the new
> `ProcessorSupplier` class -- should we also deprecate the corresponding
> existing ones that take the old `ProcessorSupplier`?
> 
> 
> -Matthias
> 
> 
> On 9/30/20 7:46 AM, John Roesler wrote:
> > Thanks Paul and Sophie,
> > 
> > Your feedback certainly underscores the need to be explicit
> > in the javadoc about why that parameter is Optional. Getting
> > this kind of feedback before the release is exactly the kind
> > of outcome we hope to get from the KIP process!
> > 
> > Thanks,
> > -John
> > 
> > On Tue, 2020-09-29 at 22:32 -0500, Paul Whalen wrote:
> > > John, I totally agree that adding a method to Processor is cumbersome and
> > > not a good path.  I was imagining maybe a separate interface that could be
> > > used in the appropriate context, but I don't think that makes too much
> > > sense - it's just too far away from what Kafka Streams is.  I was
> > > originally more interested in the "why" Optional than the "how" (I think 
> > > my
> > > original reply overplayed the "optional as an argument" concern).  But
> > > you've convinced me that there is a perfectly legitimate "why".  We should
> > > make sure that it's clear why it's Optional, but I suppose that goes
> > > without saying.  It's a nice opportunity to make the API reflect more what
> > > is actually going on under the hood.
> > > 
> > > Thanks!
> > > Paul
> > > 
> > > On Tue, Sep 29, 2020 at 10:05 PM Sophie Blee-Goldman 
> > > wrote:
> > > 
> > > > FWIW, while I'm really not a fan of Optional in general, I agree that 
> > > > its
> > > > usage
> > > > here seems appropriate. Even for those rare software developers who
> > > > carefully
> > > > read all the docs several times over, I think it wouldn't be too hard to
> > > > miss a
> > > > note about the RecordMetadata possibly being null.
> > > > 
> > > > Especially because it's not that obvious why at first glance, and takes 
> > > > a
> > > > bit of
> > > > thinking to realize that records originating from a Punctuator wouldn't
> > > > have a
> > > > "current record". This  is something that has definitely confused users
> > > > today.
> > > > 
> > > > It's on us to improve the education here -- and an 
> > > > Optional
> > > > would naturally raise awareness of this subtlety
> > > > 
> > > > On Tue, Sep 29, 2020 at 7:40 PM Sophie Blee-Goldman 
> > > > 
> > > > wrote:
> > > > 
> > > > > Does my reply address your concerns?
> > > > > 
> > > > > 
> > > > > Yes; also, I definitely misread part of the proposal earlier and 
> > > > > thought
> > > > > you had put
> > > > > the timestamp field in RecordMetadata. Sorry for not giving things a
> > > > > closer look
> > > > > before responding! I'm not sure my original message made much sense 
> > > > > given
> > > > > the misunderstanding, but thanks for responding anyway :P
> > > > > 
> > > > > Having given the proposal a second pass, I agree, it's very elegant. 
> > > > > +1
> > > > > 
> > > > > On Tue, Sep 29, 2020 at 6:50 PM John Roesler 
> > > > wrote:
> > > > > > Thanks for the reply, Sophie,
> > > > > > 
> > > > > > I think I may have summarized too much in my prior reply.
> > > > > > 
> > > > > > In the currently proposed KIP, any caller of forward() must
> > > > > > supply a Record, which consists of:
> > > > > > * key
> > > > > > * value
> > > > > > * timestamp
> > > > > > * headers (with a convenience constructor that sets empty
> > > > > > headers)
> > > > > > 
> > > > > > These aren't what I was referring to as potentially being
> > > > > > undefined downstream, since thanks to the introduction of
> > > > > > Record, they are, as you're advocating, required to be
> > > > > > defined everywhere, even when forwarding from a punctuator.
> > > > > > 
> > > > > > So to be clear, the intent of this change is actually to
> > > > > > _enforce_ that timestamp would never be undefined (which it
> > > > > > currently can be). Also, since punctuators _are_ going to
> > > > > > have to "make up" a timestamp going forward, we should note
> > > > > > that the "punctuate" method currently passes in a good
> > > > > > timestamp that they can use: for system-time punctuations,
> > > > > > they receive the current system time, and for stream-time
> > > > > > punctuations, they get the current stream time.
> > > > > > 
> > > > > > The potentially undefined RecordMetadata only contains these
> > > > > > fields:
> > > > > > * topic
> > > > > > * partition
> > > > > > * offset
> > > > > > 
> > > > > > 

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

[Matthias J. Sax]

Thanks John. I like the proposal.

Btw: I was just going over the KIP and realized that we add new methods
to `StreamBuilder`, `Topology`, and `KStream` that take the new
`ProcessorSupplier` class -- should we also deprecate the corresponding
existing ones that take the old `ProcessorSupplier`?


-Matthias


On 9/30/20 7:46 AM, John Roesler wrote:
> Thanks Paul and Sophie,
> 
> Your feedback certainly underscores the need to be explicit
> in the javadoc about why that parameter is Optional. Getting
> this kind of feedback before the release is exactly the kind
> of outcome we hope to get from the KIP process!
> 
> Thanks,
> -John
> 
> On Tue, 2020-09-29 at 22:32 -0500, Paul Whalen wrote:
>> John, I totally agree that adding a method to Processor is cumbersome and
>> not a good path.  I was imagining maybe a separate interface that could be
>> used in the appropriate context, but I don't think that makes too much
>> sense - it's just too far away from what Kafka Streams is.  I was
>> originally more interested in the "why" Optional than the "how" (I think my
>> original reply overplayed the "optional as an argument" concern).  But
>> you've convinced me that there is a perfectly legitimate "why".  We should
>> make sure that it's clear why it's Optional, but I suppose that goes
>> without saying.  It's a nice opportunity to make the API reflect more what
>> is actually going on under the hood.
>>
>> Thanks!
>> Paul
>>
>> On Tue, Sep 29, 2020 at 10:05 PM Sophie Blee-Goldman 
>> wrote:
>>
>>> FWIW, while I'm really not a fan of Optional in general, I agree that its
>>> usage
>>> here seems appropriate. Even for those rare software developers who
>>> carefully
>>> read all the docs several times over, I think it wouldn't be too hard to
>>> miss a
>>> note about the RecordMetadata possibly being null.
>>>
>>> Especially because it's not that obvious why at first glance, and takes a
>>> bit of
>>> thinking to realize that records originating from a Punctuator wouldn't
>>> have a
>>> "current record". This  is something that has definitely confused users
>>> today.
>>>
>>> It's on us to improve the education here -- and an Optional
>>> would naturally raise awareness of this subtlety
>>>
>>> On Tue, Sep 29, 2020 at 7:40 PM Sophie Blee-Goldman 
>>> wrote:
>>>
 Does my reply address your concerns?


 Yes; also, I definitely misread part of the proposal earlier and thought
 you had put
 the timestamp field in RecordMetadata. Sorry for not giving things a
 closer look
 before responding! I'm not sure my original message made much sense given
 the misunderstanding, but thanks for responding anyway :P

 Having given the proposal a second pass, I agree, it's very elegant. +1

 On Tue, Sep 29, 2020 at 6:50 PM John Roesler 
>>> wrote:
> Thanks for the reply, Sophie,
>
> I think I may have summarized too much in my prior reply.
>
> In the currently proposed KIP, any caller of forward() must
> supply a Record, which consists of:
> * key
> * value
> * timestamp
> * headers (with a convenience constructor that sets empty
> headers)
>
> These aren't what I was referring to as potentially being
> undefined downstream, since thanks to the introduction of
> Record, they are, as you're advocating, required to be
> defined everywhere, even when forwarding from a punctuator.
>
> So to be clear, the intent of this change is actually to
> _enforce_ that timestamp would never be undefined (which it
> currently can be). Also, since punctuators _are_ going to
> have to "make up" a timestamp going forward, we should note
> that the "punctuate" method currently passes in a good
> timestamp that they can use: for system-time punctuations,
> they receive the current system time, and for stream-time
> punctuations, they get the current stream time.
>
> The potentially undefined RecordMetadata only contains these
> fields:
> * topic
> * partition
> * offset
>
> These fields aren't required (or even used) in a Sink, and
> it doesn't seem like they would be important to many
> applications. Furthermore, it doesn't _seem_ like you'd even
> want to set these fields. They seem purely informational and
> only useful in the context when you are actually processing
> a real input record. It doesn't sound like you were asking
> for it, but just to put it on the record, I think if we were
> to require values for the metadata from punctuators, people
> would mostly just make up their own dummy values, to no
> one's benefit.
>
> I should also note that with the current
> Record/RecordMetadata split, we will have the freedom to
> move fields into the Record class (or even add new fields)
> if we want them to become "data" as opposed to "metadata" in
> the future.
>
> Thanks for your reply; I was similarly 

Re: [VOTE] KIP-478 Strongly Typed Streams Processor API

[John Roesler]

Thanks Paul and Sophie,

Your feedback certainly underscores the need to be explicit
in the javadoc about why that parameter is Optional. Getting
this kind of feedback before the release is exactly the kind
of outcome we hope to get from the KIP process!

Thanks,
-John

On Tue, 2020-09-29 at 22:32 -0500, Paul Whalen wrote:
> John, I totally agree that adding a method to Processor is cumbersome and
> not a good path.  I was imagining maybe a separate interface that could be
> used in the appropriate context, but I don't think that makes too much
> sense - it's just too far away from what Kafka Streams is.  I was
> originally more interested in the "why" Optional than the "how" (I think my
> original reply overplayed the "optional as an argument" concern).  But
> you've convinced me that there is a perfectly legitimate "why".  We should
> make sure that it's clear why it's Optional, but I suppose that goes
> without saying.  It's a nice opportunity to make the API reflect more what
> is actually going on under the hood.
> 
> Thanks!
> Paul
> 
> On Tue, Sep 29, 2020 at 10:05 PM Sophie Blee-Goldman 
> wrote:
> 
> > FWIW, while I'm really not a fan of Optional in general, I agree that its
> > usage
> > here seems appropriate. Even for those rare software developers who
> > carefully
> > read all the docs several times over, I think it wouldn't be too hard to
> > miss a
> > note about the RecordMetadata possibly being null.
> > 
> > Especially because it's not that obvious why at first glance, and takes a
> > bit of
> > thinking to realize that records originating from a Punctuator wouldn't
> > have a
> > "current record". This  is something that has definitely confused users
> > today.
> > 
> > It's on us to improve the education here -- and an Optional
> > would naturally raise awareness of this subtlety
> > 
> > On Tue, Sep 29, 2020 at 7:40 PM Sophie Blee-Goldman 
> > wrote:
> > 
> > > Does my reply address your concerns?
> > > 
> > > 
> > > Yes; also, I definitely misread part of the proposal earlier and thought
> > > you had put
> > > the timestamp field in RecordMetadata. Sorry for not giving things a
> > > closer look
> > > before responding! I'm not sure my original message made much sense given
> > > the misunderstanding, but thanks for responding anyway :P
> > > 
> > > Having given the proposal a second pass, I agree, it's very elegant. +1
> > > 
> > > On Tue, Sep 29, 2020 at 6:50 PM John Roesler 
> > wrote:
> > > > Thanks for the reply, Sophie,
> > > > 
> > > > I think I may have summarized too much in my prior reply.
> > > > 
> > > > In the currently proposed KIP, any caller of forward() must
> > > > supply a Record, which consists of:
> > > > * key
> > > > * value
> > > > * timestamp
> > > > * headers (with a convenience constructor that sets empty
> > > > headers)
> > > > 
> > > > These aren't what I was referring to as potentially being
> > > > undefined downstream, since thanks to the introduction of
> > > > Record, they are, as you're advocating, required to be
> > > > defined everywhere, even when forwarding from a punctuator.
> > > > 
> > > > So to be clear, the intent of this change is actually to
> > > > _enforce_ that timestamp would never be undefined (which it
> > > > currently can be). Also, since punctuators _are_ going to
> > > > have to "make up" a timestamp going forward, we should note
> > > > that the "punctuate" method currently passes in a good
> > > > timestamp that they can use: for system-time punctuations,
> > > > they receive the current system time, and for stream-time
> > > > punctuations, they get the current stream time.
> > > > 
> > > > The potentially undefined RecordMetadata only contains these
> > > > fields:
> > > > * topic
> > > > * partition
> > > > * offset
> > > > 
> > > > These fields aren't required (or even used) in a Sink, and
> > > > it doesn't seem like they would be important to many
> > > > applications. Furthermore, it doesn't _seem_ like you'd even
> > > > want to set these fields. They seem purely informational and
> > > > only useful in the context when you are actually processing
> > > > a real input record. It doesn't sound like you were asking
> > > > for it, but just to put it on the record, I think if we were
> > > > to require values for the metadata from punctuators, people
> > > > would mostly just make up their own dummy values, to no
> > > > one's benefit.
> > > > 
> > > > I should also note that with the current
> > > > Record/RecordMetadata split, we will have the freedom to
> > > > move fields into the Record class (or even add new fields)
> > > > if we want them to become "data" as opposed to "metadata" in
> > > > the future.
> > > > 
> > > > Thanks for your reply; I was similarly floored when I
> > > > realized the true nature of the current situation. Does my
> > > > reply address your concerns?
> > > > 
> > > > Thanks,
> > > > -John
> > > > 
> > > > On Tue, 2020-09-29 at 18:34 -0700, Sophie Blee-Goldman
> > > > wrote:
> > > > > > 

Re: [VOTE] KIP-671: Add method to Shutdown entire Streams Application

[Walker Carlson]

Bruno Cadonna 
4:51 AM (2 hours ago)
to dev
Thank you all for voting!

This KIP is accepted with +3 binding (Guozhang, Bill, Matthias) and +2
non-binding (Bruno, Leah).

Matthias, we will take care of  the global threads, and for the replacement
that will depend on Kip-663.

Best,

On Wed, Sep 30, 2020 at 4:59 AM Bruno Cadonna  wrote:

> Thanks a lot Walker!
>
> +1 (non-binding)
>
> Best,
> Bruno
>
> On 30.09.20 03:10, Matthias J. Sax wrote:
> > Thanks Walker. The proposed API changes LGTM.
> >
> > +1 (binding)
> >
> > One minor nit: you should also mention the global-thread that also needs
> > to be shutdown if requested by the user.
> >
> > Minor side question: should we actually terminate a thread and create a
> > new one, or instead revive the existing thread (reusing its existing ID)?
> >
> >
> > -Matthias
> >
> > On 9/29/20 2:39 PM, Bill Bejeck wrote:
> >> Thanks for the KIP Walker.
> >>
> >> +1 (binding)
> >>
> >> -Bill
> >>
> >> On Tue, Sep 29, 2020 at 4:59 PM Guozhang Wang 
> wrote:
> >>
> >>> +1 again on the KIP.
> >>>
> >>> On Tue, Sep 29, 2020 at 1:51 PM Leah Thomas 
> wrote:
> >>>
>  Hey Walker,
> 
>  Thanks for the KIP! I'm +1, non-binding.
> 
>  Cheers,
>  Leah
> 
>  On Tue, Sep 29, 2020 at 1:56 PM Walker Carlson  >
>  wrote:
> 
> > Hello all,
> >
> > I made some changes to the KIP the descriptions are on the discussion
> > thread. If you have already voted I would ask you to confirm your
> vote.
> >
> > Otherwise please vote so we can get this feature in.
> >
> > Thanks,
> > Walker
> >
> > On Thu, Sep 24, 2020 at 4:36 PM John Roesler 
>  wrote:
> >
> >> Thanks for the KIP, Walker!
> >>
> >> I’m +1 (binding)
> >>
> >> -John
> >>
> >> On Mon, Sep 21, 2020, at 17:04, Guozhang Wang wrote:
> >>> Thanks for finalizing the KIP. +1 (binding)
> >>>
> >>>
> >>> Guozhang
> >>>
> >>> On Mon, Sep 21, 2020 at 1:38 PM Walker Carlson <
>  wcarl...@confluent.io>
> >>> wrote:
> >>>
>  Hello all,
> 
>  I would like to start a thread to vote for KIP-671 to add a
> >>> method
>  to
> >> close
>  all clients in a kafka streams application.
> 
>  KIP:
> 
> 
> >>
> >
> 
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-671%3A+Shutdown+Streams+Application+when+appropriate+exception+is+thrown
> 
>  Discussion thread: *here
>  <
> 
> >>
> >
> 
> >>>
> https://mail-archives.apache.org/mod_mbox/kafka-dev/202009.mbox/%3CCAC55fuh3HAGCxz-PzxTJraczy6T-os2oiCV328PBeuJQSVYASg%40mail.gmail.com%3E
> > *
> 
>  Thanks,
>  -Walker
> 
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
> >
> 
> >>>
> >>>
> >>> --
> >>> -- Guozhang
> >>>
> >>
>

Re: [VOTE] KIP-671: Add method to Shutdown entire Streams Application

[Bruno Cadonna]

Thanks a lot Walker!

+1 (non-binding)

Best,
Bruno

On 30.09.20 03:10, Matthias J. Sax wrote:

Thanks Walker. The proposed API changes LGTM.

+1 (binding)

One minor nit: you should also mention the global-thread that also needs
to be shutdown if requested by the user.

Minor side question: should we actually terminate a thread and create a
new one, or instead revive the existing thread (reusing its existing ID)?


-Matthias

On 9/29/20 2:39 PM, Bill Bejeck wrote:

Thanks for the KIP Walker.

+1 (binding)

-Bill

On Tue, Sep 29, 2020 at 4:59 PM Guozhang Wang  wrote:


+1 again on the KIP.

On Tue, Sep 29, 2020 at 1:51 PM Leah Thomas  wrote:


Hey Walker,

Thanks for the KIP! I'm +1, non-binding.

Cheers,
Leah

On Tue, Sep 29, 2020 at 1:56 PM Walker Carlson 
wrote:


Hello all,

I made some changes to the KIP the descriptions are on the discussion
thread. If you have already voted I would ask you to confirm your vote.

Otherwise please vote so we can get this feature in.

Thanks,
Walker

On Thu, Sep 24, 2020 at 4:36 PM John Roesler 

wrote:



Thanks for the KIP, Walker!

I’m +1 (binding)

-John

On Mon, Sep 21, 2020, at 17:04, Guozhang Wang wrote:

Thanks for finalizing the KIP. +1 (binding)


Guozhang

On Mon, Sep 21, 2020 at 1:38 PM Walker Carlson <

wcarl...@confluent.io>

wrote:


Hello all,

I would like to start a thread to vote for KIP-671 to add a

method

to

close

all clients in a kafka streams application.

KIP:









https://cwiki.apache.org/confluence/display/KAFKA/KIP-671%3A+Shutdown+Streams+Application+when+appropriate+exception+is+thrown


Discussion thread: *here
<








https://mail-archives.apache.org/mod_mbox/kafka-dev/202009.mbox/%3CCAC55fuh3HAGCxz-PzxTJraczy6T-os2oiCV328PBeuJQSVYASg%40mail.gmail.com%3E

*


Thanks,
-Walker




--
-- Guozhang










--
-- Guozhang

Re: [VOTE] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

[Bruno Cadonna]

Thank you all for voting!

This KIP is accepted with 3 binding +1 (Guozhang, John, Matthias).

Best,
Bruno

On 29.09.20 22:24, Matthias J. Sax wrote:

+1 (binding)

I am not super happy with the impact on the client state. For example, I
don't understand why it's ok to scale out if we lose one thread out of
four, but why it's not ok to scale out if we lose one thread out of one
(for this case, we would enter ERROR state and cannot add new threads
afterwards).

However, this might be an issue for a follow up KIP.


-Matthias

On 9/29/20 7:20 AM, John Roesler wrote:

Thanks, Bruno, this sounds good to me.
-John

On Tue, Sep 29, 2020, at 03:13, Bruno Cadonna wrote:

Hi all,

I did two minor modifications to the KIP.

- I removed the rather strict guarantee "Dead stream threads are removed
from a Kafka Streams client at latest after the next call to
KafkaStreams#addStreamThread() or KafkaStreams#removeStreamThread()
following the transition to state DEAD."
Dead stream threads will be still removed, but the behavior will be less
strict.

- Added a sentence that states that the Kafka Streams client will
transit to ERROR if the last alive stream thread dies exceptionally.
This corresponds to the current behavior.

I will not restart voting and keep the votes so far.

Best,
Bruno

On 22.09.20 01:19, John Roesler wrote:

I’m +1 also. Thanks, Bruno!
-John

On Mon, Sep 21, 2020, at 17:08, Guozhang Wang wrote:

Thanks Bruno. I'm +1 on the KIP.

On Mon, Sep 21, 2020 at 2:49 AM Bruno Cadonna  wrote:


Hi,

I would like to restart from zero the voting on KIP-663 that proposes to
add methods to the Kafka Streams client to add and remove stream threads
during execution.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-663%3A+API+to+Start+and+Shut+Down+Stream+Threads

Matthias, if you are still +1, please vote again.

Best,
Bruno

On 04.09.20 23:12, John Roesler wrote:

Hi Sophie,

Uh, oh, it's never a good sign when the discussion moves
into the vote thread :)

I agree with you, it seems like a good touch for
removeStreamThread() to return the name of the thread that
got removed, rather than a boolean flag. Maybe the return
value would be `null` if there is no thread to remove.

If we go that way, I'd suggest that addStreamThread() also
return the name of the newly created thread, or null if no
thread can be created right now.

I'm not completely sure if I think that callers of this
method would know exactly how many threads there are. Sure,
if a human being is sitting there looking at the metrics or
logs and decides to call the method, it would work out, but
I'd expect this kind of method to find its way into
automated tooling that reacts to things like current system
load or resource saturation. Those kinds of toolchains often
are part of a distributed system, and it's probably not that
easy to guarantee that the thread count they observe is
fully consistent with the number of threads that are
actually running. Therefore, an in-situ `int
numStreamThreads()` method might not be a bad idea. Then
again, it seems sort of optional. A caller can catch an
exception or react to a `null` return value just the same
either way. Having both add/remove methods behave similarly
is probably more valuable.

Thanks,
-John


On Thu, 2020-09-03 at 12:15 -0700, Sophie Blee-Goldman
wrote:

Hey, sorry for the late reply, I just have one minor suggestion. Since

we

don't
make any guarantees about which thread gets removed or allow the user to
specify, I think we should return either the index or full name of the
thread
that does get removed by removeThread().

I know you just updated the KIP to return true/false if there

are/aren't any

threads to be removed, but I think this would be more appropriate as an
exception than as a return type. I think it's reasonable to expect

users to

have some sense to how many threads are remaining, and not try to remove
a thread when there is none left. To me, that indicates something wrong
with the user application code and should be treated as an exceptional

case.

I don't think the same code clarify argument applies here as to the
addStreamThread() case, as there's no reason for an application to be
looping and retrying removeStreamThread()  since if that fails, it's

because

there are no threads left and thus it will continue to always fail. And

if

the
user actually wants to shut down all threads, they should just close the
whole application rather than call removeStreamThread() in a loop.

While I generally think it should be straightforward for users to track

how

many stream threads they have running, maybe it would be nice to add
a small utility method that does this for them. Something like

// Returns the number of currently alive threads
boolean runningStreamThreads();

On Thu, Sep 3, 2020 at 7:41 AM Matthias J. Sax 

wrote:



+1 (binding)

On 9/3/20 6:16 AM, Bruno Cadonna wrote:

Hi,

I would like to start the voting on KIP-663 that proposes to add

methods

to the Kafka Streams 

[DISCUSS] KIP-674: API to Aggregate Metrics in Kafka Streams

[Bruno Cadonna]

Hi,

I would like to propose the following KIP to add an API to the Kafka 
Streams client to aggregate metrics.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-674%3A+API+to+Aggregate+Metrics+in+Kafka+Streams

Best,
Bruno

[jira] [Created] (KAFKA-10554) Perform follower truncation based on epoch offsets returned in Fetch response

[Rajini Sivaram (Jira)]

Rajini Sivaram created KAFKA-10554:
--

 Summary: Perform follower truncation based on epoch offsets 
returned in Fetch response
 Key: KAFKA-10554
 URL: https://issues.apache.org/jira/browse/KAFKA-10554
 Project: Kafka
  Issue Type: Task
  Components: replication
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 2.7.0


KAFKA-10435 updated fetch protocol for KIP-595 to return diverging epoch and 
offset as part of fetch response. We can use this to truncate logs in followers 
while processing fetch responses.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-673: Emit JSONs with new auto-generated schema

[Tom Bentley]

Hi Anastasia,

Is the concern that deserialization would have no guarantee as to the order
> it appears?
>

No, it's simply that when reading the logs screeds of unformatted JSON can
be hard to read, especially when more important fields come later in an
object (or after an array). This is also a problem with the existing format
too, of course. And there's now the option to put it through `jq` to indent
it. So I don't think it's worth obsessing over.


> I understand the performance concern. However, this would only be logged
> when the trace level logging is turned on, so this logging isn't done
> otherwise. Plus, the existing text-based logging is already very expensive,
> so it might not be that significant of a change.
>

Yeah, it's an implementation detail, so could always be optimised later
when it's known to be a problem.

Thanks again for the KIP, LGTM.

Tom