Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Vahid Hashemian]

Thank you all.

Regards,
--Vahid

On Wed, Jan 15, 2020 at 9:15 AM Colin McCabe  wrote:

> Thanks, everyone!
>
> best,
> Colin
>
> On Wed, Jan 15, 2020, at 07:50, Sean Glover wrote:
> > Congratulations Colin, Vahid and Manikumar and thank you for all your
> > excellent work on Apache Kafka!
> >
> > On Wed, Jan 15, 2020 at 8:42 AM Ron Dagostino  wrote:
> >
> > > Congratulations!
> > >
> > > > On Jan 15, 2020, at 5:04 AM, Viktor Somogyi-Vass <
> > > viktorsomo...@gmail.com> wrote:
> > > >
> > > > Congrats to you guys, it's a great accomplishment! :)
> > > >
> > > >> On Wed, Jan 15, 2020 at 10:20 AM David Jacot 
> > > wrote:
> > > >>
> > > >> Congrats!
> > > >>
> > > >>> On Wed, Jan 15, 2020 at 12:00 AM James Cheng  >
> > > wrote:
> > > >>>
> > > >>> Congrats Colin, Vahid, and Manikumar!
> > > >>>
> > > >>> -James
> > > >>>
> > >  On Jan 14, 2020, at 10:59 AM, Tom Bentley 
> > > wrote:
> > > 
> > >  Congratulations!
> > > 
> > >  On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram <
> > > >> rajinisiva...@gmail.com>
> > >  wrote:
> > > 
> > > > Congratulations Colin, Vahid and Manikumar!
> > > >
> > > > Regards,
> > > > Rajini
> > > >
> > > > On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison <
> > > >>> mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> Congrats Colin, Vahid and Manikumar!
> > > >>
> > > >> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma 
> > > >> wrote:
> > > >>>
> > > >>> Congratulations Colin, Vahid and Manikumar!
> > > >>>
> > > >>> Ismael
> > > >>>
> > > >>> On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira <
> g...@confluent.io>
> > > > wrote:
> > > >>>
> > >  Hi everyone,
> > > 
> > >  I'm happy to announce that Colin McCabe, Vahid Hashemian and
> > > > Manikumar
> > >  Reddy are now members of Apache Kafka PMC.
> > > 
> > >  Colin and Manikumar became committers on Sept 2018 and Vahid
> on
> > > Jan
> > >  2019. They all contributed many patches, code reviews and
> > > > participated
> > >  in many KIP discussions. We appreciate their contributions
> and are
> > >  looking forward to many more to come.
> > > 
> > >  Congrats Colin, Vahid and Manikumar!
> > > 
> > >  Gwen, on behalf of Apache Kafka PMC
> > > 
> > > >>
> > > >
> > > >>>
> > > >>>
> > > >>
> > >
> >
>


-- 

Thanks!
--Vahid

[jira] [Resolved] (KAFKA-9410) Make groupId Optional in KafkaConsumer

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9410.

Resolution: Fixed

> Make groupId Optional in KafkaConsumer
> --
>
> Key: KAFKA-9410
> URL: https://issues.apache.org/jira/browse/KAFKA-9410
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: David Mollitor
>Priority: Minor
>
> ... because it is.
>  
> [https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Role of CompositeReadOnlyKeyValueStore for point queries

[Navinder Brar]

Hi all,
Can someone explain to me the thoughts behind having 
CompositeReadOnlyKeyValueStore. java class while serving data via APIs in Kafka 
Streams. It fetches the list of stores for all the running tasks on the machine 
and then looks for a key one by one in each of the stores. When we already know 
the key belongs to a particular partition(with the help of partitioner) we can 
just query that particular partition's store right?
I am thinking of overloading the get() function as get(int partition) and 
sending just the store for that single partition from 
QueryableStoreProvider.java so that all the stores are needed to be iterated 
through to fetch a key.
Regards,
Navinder

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[Navinder Brar]

Thanks, Vinoth and John for making the last minute improvements. I have gone 
through the PR and looks good to me. 

On Thursday, 16 January, 2020, 12:42:09 am IST, Guozhang Wang 
 wrote:  
 
 Thanks for the update of the PR John! I have taken a look at 7962 and it
looks good to me overall.


Guozhang

On Wed, Jan 15, 2020 at 9:35 AM John Roesler  wrote:

> Hello again all,
>
> I had a bit of inspiration last night and realized that it's not necessary
> (and maybe even inappropriate) for StreamThreadStateStoreProvider
> and WrappingStoreProvider to implement the public StateStoreProvider
> interface.
>
> By breaking this dependency, I was able to implement the flag without
> touching
> any public interfaces except adding the new overload to KafkaStreams as
> originally
> discussed.
>
> You can take a look at https://github.com/apache/kafka/pull/7962 for the
> details.
>
> Since there was no objection to that new overload, I'll go ahead and
> update the KIP
> and we can proceed with a final round of code reviews on
> https://github.com/apache/kafka/pull/7962
>
> Thanks, all,
> -John
>
> On Tue, Jan 14, 2020, at 22:52, Matthias J. Sax wrote:
> > Thanks. SGTM.
> >
> > -Matthias
> >
> > On 1/14/20 4:52 PM, John Roesler wrote:
> > > Hey Matthias,
> > >
> > > Thanks for taking a look! I felt a little uneasy about it, but didn’t
> think about the case you pointed out. Throwing an exception would indeed be
> safer.
> > >
> > > Given a choice between throwing in the default method or defining a
> new interface and throwing if the wrong interface is implemented, it seems
> nicer for everyone to go the default method route. Since we’re not
> referencing the other method anymore, I should probably deprecate it, too.
> > >
> > > Thanks again for your help. I really appreciate it.
> > >
> > > -John
> > >
> > > On Tue, Jan 14, 2020, at 18:15, Matthias J. Sax wrote:
> > >> Thanks for the PR. That helps a lot.
> > >>
> > >> I actually do have a concern: the proposed default method, would
> disable
> > >> the new feature to allow querying an active task during restore
> > >> automatically. Hence, if a user has an existing custom store type, and
> > >> would use the new
> > >>
> > >> KafkaStreams.store(..., true)
> > >>
> > >> method to enable querying during restore it would not work, and it
> would
> > >> be unclear why. It would even be worth if there are two developers and
> > >> one provide the store type while the other one just uses it.
> > >>
> > >> Hence, the default implementation should maybe throw an exception by
> > >> default? Or maybe, we would introduce a new interface that extends
> > >> `QueryableStoreType` and add this new method. For this case, we could
> > >> check within
> > >>
> > >> KafkaStreams.store(..., true)
> > >>
> > >> if the StoreType implements the new interface and if not, throw an
> > >> exception there.
> > >>
> > >> Those exceptions would be more descriptive (ie, state store does not
> > >> support querying during restore) and give the user a chance to figure
> > >> out what's wrong.
> > >>
> > >> Not sure if overwriting a default method or a new interface is the
> > >> better choice to let people opt-into the feature.
> > >>
> > >>
> > >>
> > >> -Matthias
> > >>
> > >> On 1/14/20 3:22 PM, John Roesler wrote:
> > >>> Hi again all,
> > >>>
> > >>> I've sent a PR including this new option, and while implementing it,
> I
> > >>> realized we also have to make a (source-compatible) addition to the
> > >>> QueryableStoreType interface, so that the IQ store wrapper can pass
> the
> > >>> flag down to the composite store provider.
> > >>>
> > >>> See https://github.com/apache/kafka/pull/7962
> > >>> In particular:
> https://github.com/apache/kafka/pull/7962/files#diff-d0242b7289f4e0886490351a5a803d41
> > >>>
> > >>> If there are no objections to these additions, I'll update the KIP
> tomorrow.
> > >>>
> > >>> Thanks,
> > >>> -John
> > >>>
> > >>> On Tue, Jan 14, 2020, at 14:11, John Roesler wrote:
> >  Thanks for calling this out, Matthias. You're correct that this
> looks like a
> >  harmful behavioral change. I'm fine with adding the new overload you
> >  mentioned, just a simple boolean flag to enable the new behavior.
> > 
> >  I'd actually propose that we call this flag "queryStaleData", with
> a default
> >  of "false". The meaning of this would be to preserve exactly the
> existing
> >  semantics. I.e., that the store must be both active and running to
> be
> >  included.
> > 
> >  It seems less severe to just suddenly start returning queries from
> standbys,
> >  but in the interest of safety, the easiest thing is just flag the
> whole feature.
> > 
> >  If you, Navinder, and Vinoth agree, we can just update the KIP. It
> seems like
> >  a pretty uncontroversial amendment to avoid breaking query
> consistency
> >  semantics.
> > 
> >  Thanks,
> >  -John
> > 
> > 
> >  On Tue, Jan 14, 2020, at 13:21, 

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[John Roesler]

Hi Vito,

Haha, your archive game is on point!

What Matthias said in that email is essentially what I figured was the 
rationale. It makes sense, but the point I was making is that this really 
doesn’t seem like a good way to structure a production app. On the other hand, 
considering the exception fatal has a good chance of avoiding a frustrating 
debug session if you just forgot to call start. 

Nevertheless, if we omit the categorization, it’s moot.

It would be easy to add a categorization layer later if we want it, but not 
very easy to change it if we get it wrong. 

Thanks for your consideration!
-John

On Wed, Jan 15, 2020, at 21:14, Vito Jeng wrote:
> Hi John,
> 
> About `StreamsNotStartedException is strange` --
> The original idea came from Matthias, two years ago. :)
> You can reference here:
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc5...@confluent.io%3e
> 
> About omitting the categorization --
> It looks reasonable. I'm fine with omitting the categorization but not very
> sure it is a good choice.
> Does any other folks provide opinion?
> 
> 
> Hi, folks,
> 
> Just update the KIP-216, please take a look.
> 
> ---
> Vito
> 
> 
> On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng  wrote:
> 
> >
> > Hi, folks,
> >
> > Thank you suggestion, really appreciate it. :)
> > I understand your concern. I'll merge StreamsNotRunningException and
> > StateStoreNotAvailableException.
> >
> >
> > ---
> > Vito
> >
> >
> > On Thu, Jan 16, 2020 at 6:22 AM John Roesler  wrote:
> >
> >> Hey Vito,
> >>
> >> Yes, thanks for the KIP. Sorry the discussion has been so long.
> >> Hopefully, we can close it out soon.
> >>
> >> I agree we can drop StreamsNotRunningException in favor of
> >> just StateStoreNotAvailableException.
> >>
> >> Unfortunately, I have some higher-level concerns. The value
> >> of these exceptions is that they tell you how to handle the
> >> various situations that can arise while querying a distributed
> >> data store.
> >>
> >> Ideally, as a caller, I should be able to just catch "retriable" or
> >> "fatal" and handle them appropriately. Otherwise, there's no
> >> point in having categories, and we should just have all the
> >> exceptions extend InvalidStateStoreException.
> >>
> >> Presently, it's not possible to tell from just the
> >> "retriable"/"fatal" distinction what to do. You  can tell
> >> from the descriptions of the various exceptions. E.g.:
> >>
> >> Retriable:
> >>  * StreamsRebalancingException: the exact same call
> >> should just be retried until the rebalance is complete
> >>  * StateStoreMigratedException: the store handle is
> >> now invalid, so you need to re-discover the instance
> >> and get a new handle on that instance. In other words,
> >> the query itself may be valid, but the particular method
> >> invocation on this particular instance has encountered
> >> a fatal exception.
> >>
> >> Fatal:
> >>  * UnknownStateStoreException: this is truly fatal. No amount
> >> of retrying or re-discovering is going to get you a handle on a
> >> store that doesn't exist in the cluster.
> >>  * StateStoreNotAvailableException: this is actually recoverable,
> >> since the store might exist in the cluster, but isn't available on
> >> this particular instance (which is shut down or whatever).
> >>
> >> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> >> with omitting the categorization and just having 5 subclasses
> >> of InvalidStateStoreException. Each of them would tell you
> >> how to handle them, and it's not too many to really
> >> understand and handle each one.
> >>
> >> If you really want to have a middle tier, I'd recommend:
> >> * RetryableStateStoreException: the exact same call
> >> should be repeated.
> >> * RecoverableStateStoreException: the store handle
> >> should be discarded and the caller should re-discover
> >> the location of the store and repeat the query on the
> >> correct instance.
> >> * FatalStateStoreException: the query/request is totally
> >> invalid and will never succeed.
> >>
> >> However, attempting to categorize the proposed exceptions
> >> reveals even problems with this categorization:
> >> Retriable:
> >> * StreamsRebalancingException
> >> Recoverable:
> >> * StateStoreMigratedException
> >> * StreamsNotRunningException
> >> Fatal:
> >> * UnknownStateStoreException
> >>
> >> But StreamsNotStartedException is strange... It means that
> >> one code path got a handle on a specific KafkaStreams object
> >> instance and sent it a query before another code path
> >> invoked the start() method on the exact same object instance.
> >> It seems like the most likely scenario is that whoever wrote
> >> the program just forgot to call start() before querying, in
> >> which case, retrying isn't going to help, and a fatal exception
> >> is more appropriate. I.e., it sounds like a "first 15 minutes
> >> experience" problem, and making 

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi John,

About `StreamsNotStartedException is strange` --
The original idea came from Matthias, two years ago. :)
You can reference here:
https://mail-archives.apache.org/mod_mbox/kafka-dev/201806.mbox/%3c6c32083e-b63c-435b-521d-032d45cc5...@confluent.io%3e

About omitting the categorization --
It looks reasonable. I'm fine with omitting the categorization but not very
sure it is a good choice.
Does any other folks provide opinion?


Hi, folks,

Just update the KIP-216, please take a look.

---
Vito


On Thu, Jan 16, 2020 at 6:35 AM Vito Jeng  wrote:

>
> Hi, folks,
>
> Thank you suggestion, really appreciate it. :)
> I understand your concern. I'll merge StreamsNotRunningException and
> StateStoreNotAvailableException.
>
>
> ---
> Vito
>
>
> On Thu, Jan 16, 2020 at 6:22 AM John Roesler  wrote:
>
>> Hey Vito,
>>
>> Yes, thanks for the KIP. Sorry the discussion has been so long.
>> Hopefully, we can close it out soon.
>>
>> I agree we can drop StreamsNotRunningException in favor of
>> just StateStoreNotAvailableException.
>>
>> Unfortunately, I have some higher-level concerns. The value
>> of these exceptions is that they tell you how to handle the
>> various situations that can arise while querying a distributed
>> data store.
>>
>> Ideally, as a caller, I should be able to just catch "retriable" or
>> "fatal" and handle them appropriately. Otherwise, there's no
>> point in having categories, and we should just have all the
>> exceptions extend InvalidStateStoreException.
>>
>> Presently, it's not possible to tell from just the
>> "retriable"/"fatal" distinction what to do. You  can tell
>> from the descriptions of the various exceptions. E.g.:
>>
>> Retriable:
>>  * StreamsRebalancingException: the exact same call
>> should just be retried until the rebalance is complete
>>  * StateStoreMigratedException: the store handle is
>> now invalid, so you need to re-discover the instance
>> and get a new handle on that instance. In other words,
>> the query itself may be valid, but the particular method
>> invocation on this particular instance has encountered
>> a fatal exception.
>>
>> Fatal:
>>  * UnknownStateStoreException: this is truly fatal. No amount
>> of retrying or re-discovering is going to get you a handle on a
>> store that doesn't exist in the cluster.
>>  * StateStoreNotAvailableException: this is actually recoverable,
>> since the store might exist in the cluster, but isn't available on
>> this particular instance (which is shut down or whatever).
>>
>> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
>> with omitting the categorization and just having 5 subclasses
>> of InvalidStateStoreException. Each of them would tell you
>> how to handle them, and it's not too many to really
>> understand and handle each one.
>>
>> If you really want to have a middle tier, I'd recommend:
>> * RetryableStateStoreException: the exact same call
>> should be repeated.
>> * RecoverableStateStoreException: the store handle
>> should be discarded and the caller should re-discover
>> the location of the store and repeat the query on the
>> correct instance.
>> * FatalStateStoreException: the query/request is totally
>> invalid and will never succeed.
>>
>> However, attempting to categorize the proposed exceptions
>> reveals even problems with this categorization:
>> Retriable:
>> * StreamsRebalancingException
>> Recoverable:
>> * StateStoreMigratedException
>> * StreamsNotRunningException
>> Fatal:
>> * UnknownStateStoreException
>>
>> But StreamsNotStartedException is strange... It means that
>> one code path got a handle on a specific KafkaStreams object
>> instance and sent it a query before another code path
>> invoked the start() method on the exact same object instance.
>> It seems like the most likely scenario is that whoever wrote
>> the program just forgot to call start() before querying, in
>> which case, retrying isn't going to help, and a fatal exception
>> is more appropriate. I.e., it sounds like a "first 15 minutes
>> experience" problem, and making it fatal would be more
>> helpful. Even in a production context, there's no reason not
>> to sequence your application startup such that you don't
>> accept queries until after Streams is started. Thus, I guess
>> I'd categorize it under "fatal".
>>
>> Regardless of whether you make it fatal or retriable, you'd
>> still have a whole category with only one exception in it,
>> and the other two categories only have two exceptions.
>> Plus, as you pointed out in the KIP, you can't get all
>> exceptions in all cases anyway:
>> * store() can only throw NotStarted, NotRunning,
>> and Unknown
>> * actual store queries can only throw Rebalancing,
>> Migrated, and NotRunning
>>
>> Thus, in practice also, there are exactly three categories
>> and also exactly three exception types. It doesn't seem
>> like there's a great advantage to the categories here. To
>> avoid the 

Re: Streams, Kafka windows

[John Roesler]

Hi Viktor,

I’m not sure why you get two identical outputs in response to a single record. 
Regardless, since you say that you want to get a single, final result for the 
window and you expect multiple inputs to the windows, you need Suppression.

My guess is that you just sent one record to try it out and didn’t see any 
output? This is expected. Just as the window boundaries are defined by the time 
stamps of the records, not by the current system time, suppression is governed 
by the timestamp of the records. I.e., a thirty-second window is not actually 
closed until you see a new record with a timestamp thirty seconds later.

 Maybe try just sending a sequence of updates with incrementing timestamps. If 
the first record has timestamp T, then you should see an output when you pass 
in a record with timestamp T+30. 

Important note: there is a built-in grace period that delays the output of 
final results after the window ends. For complicated reasons, the default is 24 
hours! So you would actually not see an output until you send a record with 
timestamp T+30+(24 hours) ! I strongly recommend you set the grace period on 
TimeWindows to zero for your testing. You can increase it later if you want to 
tolerate some late-arriving records. 

Thanks,
-John

On Tue, Jan 14, 2020, at 10:41, Viktor Markvardt wrote:
> Hi,
> 
> My name is Viktor. I'm currently working with Kafka streams and have
> several questions about Kafka and I can not find answers in the official
> docs.
> 
> 1. Why suppress functionality does not work with Hopping windows? How to
> make it work?
> 
> Example of the code:
> 
> KStream finalStream = source
> .groupByKey()
> 
> .windowedBy(TimeWindows.of(Duration.ofSeconds(30)).advanceBy(Duration.ofSeconds(10)))
> .reduce((aggValue, newValue) -> newValue,
> Materialized.with(Serdes.String(), Serdes.String()))
> 
> .suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded()))
> .toStream();
> 
> finalStream.print(Printed.toSysOut());
> finalStream.to(outputTopic);
> 
> After I run the code above - output stream is empty. There were no
> errors/exceptions.
> NOTE: With Tumbling Window the code working as expected.
> Maybe I simply use it incorrectly?
> 
> 2. Why with Hopping windows (without suppress) there are duplicates in the
> output stream?
> E.g., I send one record in the input kstream with Hopping window
> (duration=30s, advanceBy=2s) but get two same records (duplicate) in the
> output kstream.
> Is that an expected behavior? If so, how can I filter/switch off these
> duplicates?
> 
> 3. Mainly I'm trying to solve this problem:
> I have kstream with events inside and events can be repeated (duplicates).
> In the output kstream I would like to receive only unique events for the
> last 24 hours (window duration) with 1 hour window overlay (window
> advanceBy).
> Could you recommend me any examples of code or docs please?
> I have already read official docs and examples but it was not enough to get
> full understanding of how I can achieve this.
> 
> Best regards,
> Viktor Markvardt
>

Re: [VOTE] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Anna McDonald]

Done. Thanks for pointing that out.

anna

On Wed, Jan 15, 2020 at 8:52 PM Guozhang Wang  wrote:
>
> Hi Anna,
>
> Just a minor comment on the wiki page itself:
>
> ```
>
> The new method, handleSerializationException, in ProductionExceptionHandler
> will be invoked when
>
>1. ClassCastException is thrown while serializing record key / value. We
>will continue to throw this exception and not invoke the new method.
>
> ```
>
> I think you meant to say that when ClassCastException is thrown, we would
> NOT trigger the handler method. But at the beginning it mentioned "will be
> invoked when.." which sounds a bit conflicting with itself. Could you
> update the wiki page?
>
> Otherwise, I'm +1 on this.
>
>
> Guozhang.
>
>
> On Wed, Jan 15, 2020 at 1:42 PM Matthias J. Sax 
> wrote:
>
> > Thanks for pushing this KIP over the finish line!
> >
> > +1 (binding)
> >
> >
> > -Matthias
> >
> > On 1/15/20 12:57 PM, Bill Bejeck wrote:
> > > Thanks for the KIP.
> > >
> > > +1 (binding)
> > >
> > > -Bill
> > >
> > > On Wed, Jan 15, 2020 at 3:45 PM M. Manna  wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> Thanks for this KIP
> > >>
> > >> Regards,
> > >>
> > >> On Wed, 15 Jan 2020 at 20:35, Mitchell  wrote:
> > >>
> > >>> +1(non-binding)
> > >>>
> > >>> Very useful
> > >>> -mitch
> > >>>
> > >>> On Wed, Jan 15, 2020, 3:29 PM Anna McDonald 
> > >>> wrote:
> > >>>
> >  Greetings,
> >  I would like to propose a vote on KIP-399, extending the
> >  ProductionExceptionHandler to cover serialization exceptions. This KIP
> >  is aimed at improving the error-handling semantics in Kafka Streams
> >  when Kafka Streams fails to serialize a message to the downstream
> >  sink.
> > 
> >  KIP details located here:
> > 
> > 
> > >>>
> > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
> > 
> >  Discussion Thread:
> > 
> > 
> > >>>
> > >>
> > https://lists.apache.org/thread.html/rbbc887ca31d46f6e73ffc6e08df7e4bda69c89ff820986c30274e272%40%3Cdev.kafka.apache.org%3E
> > 
> >  Thanks,
> >  anna
> > 
> > >>>
> > >>
> > >
> >
> >
>
> --
> -- Guozhang

Re: [VOTE] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Guozhang Wang]

Hi Anna,

Just a minor comment on the wiki page itself:

```

The new method, handleSerializationException, in ProductionExceptionHandler
will be invoked when

   1. ClassCastException is thrown while serializing record key / value. We
   will continue to throw this exception and not invoke the new method.

```

I think you meant to say that when ClassCastException is thrown, we would
NOT trigger the handler method. But at the beginning it mentioned "will be
invoked when.." which sounds a bit conflicting with itself. Could you
update the wiki page?

Otherwise, I'm +1 on this.


Guozhang.


On Wed, Jan 15, 2020 at 1:42 PM Matthias J. Sax 
wrote:

> Thanks for pushing this KIP over the finish line!
>
> +1 (binding)
>
>
> -Matthias
>
> On 1/15/20 12:57 PM, Bill Bejeck wrote:
> > Thanks for the KIP.
> >
> > +1 (binding)
> >
> > -Bill
> >
> > On Wed, Jan 15, 2020 at 3:45 PM M. Manna  wrote:
> >
> >> +1 (non-binding)
> >>
> >> Thanks for this KIP
> >>
> >> Regards,
> >>
> >> On Wed, 15 Jan 2020 at 20:35, Mitchell  wrote:
> >>
> >>> +1(non-binding)
> >>>
> >>> Very useful
> >>> -mitch
> >>>
> >>> On Wed, Jan 15, 2020, 3:29 PM Anna McDonald 
> >>> wrote:
> >>>
>  Greetings,
>  I would like to propose a vote on KIP-399, extending the
>  ProductionExceptionHandler to cover serialization exceptions. This KIP
>  is aimed at improving the error-handling semantics in Kafka Streams
>  when Kafka Streams fails to serialize a message to the downstream
>  sink.
> 
>  KIP details located here:
> 
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
> 
>  Discussion Thread:
> 
> 
> >>>
> >>
> https://lists.apache.org/thread.html/rbbc887ca31d46f6e73ffc6e08df7e4bda69c89ff820986c30274e272%40%3Cdev.kafka.apache.org%3E
> 
>  Thanks,
>  anna
> 
> >>>
> >>
> >
>
>

-- 
-- Guozhang

Build failed in Jenkins: kafka-trunk-jdk8 #4162

[Apache Jenkins Server]

See 


Changes:

[vvcephei] KAFKA-6144: IQ option to query standbys (#7962)

[vahid.hashemian] MINOR: Removed accidental double negation in error message. 
(#7834)


--
[...truncated 5.71 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Konstantine Karantasis]

Hey Almog, thanks for the comments!
Here's my take:

1) I think that an approximate grouping of topics to
highly-active/active/inactive (because a precise one would be too
expensive) seems like something we could leave out of this first version of
topic tracking. Interestingly, as you point out, given a topic of
continuous flow, you may get a similar view by resetting and subsequently
querying the topics endpoint. The way I understand Randall's comment, is
that persisting timestamps would have to do more with knowing approximately
the first time this connector has been using its topics. But given the
implications that topic retention might have, I suggest that we leave
timestamp recording out at this point.

2) As I described in my previous answer above (#6 on my first reply to
Randall's comments), this asymmetry is inherited from the sink connector's
configuration. I'm ok not resetting the topics for both upon
reconfiguration, since this will result in simpler code. But I'd like to
know that we are ok with a sink connector showing topics in its active set
that are not in its current configuration (unless a reset request is
issued).

3) A similar idea crossed my mind while I was thinking how the
"/connectors" endpoint evolved with KIP-465 to show a roll up of the status
of the tasks of all the connectors. However, here what you describe would
probably require an additional top-level "/topics" endpoint and a more
complex filtering based on permissions. I'd suggest punting this feature,
unless people think that is a really nice to have. In the meantime, as you
mention, it is something that can be constructed with consecutive queries
after an applications gets the list of connectors running in the Connect
cluster.

Cheers,
Konstantine

On Wed, Jan 15, 2020 at 2:41 PM Randall Hauch  wrote:

> On Wed, Jan 15, 2020 at 4:36 PM Randall Hauch  wrote:
>
> > On Wed, Jan 15, 2020 at 2:05 PM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> >>
> >> 9. I assumed that partitioning is implied by default, because there's no
> >> requirement for complete ordering of topic status records. But I'll add
> >> this fact as a separate bullet. The status.storage.topic is already a
> >> partitioned topic.
> >>
> >
> > Agreed. I think it'd be sufficient to simply mention that partition will
> > be chosen based upon the active topic records' keys, ensuring that all
> > active topic records for the same connector will be written to the same
> > partition and will be totally ordered.
> >
>
> Well, my previous statement is not quite right. All topic records for the
> same *connector and topic* will be written to the same partition and will
> be totally ordered. But as you pointed out, it doesn't really matter, other
> than that this feature will work with any # of partitions. The new bullet
> you described would be sufficient. :-D
>
>
> >
> >>
> >> I'm following up with the rest of the comments, shortly.
> >> Thanks,
> >> Konstantine
> >>
> >>
> >> On Wed, Jan 15, 2020 at 9:19 AM Almog Gavra  wrote:
> >>
> >> > Hi Konstantine,
> >> >
> >> > Thanks for the KIP! This is going to make automatic integration with
> >> > Connect much more powerful.
> >> >
> >> > My thoughts are mostly around freshness of the data and being able to
> >> > expose that to users. Riffing on Randall's timestamp question - have
> we
> >> > considered adding some interval at which point a connector will
> >> republish
> >> > any topics that it encounters and update the timestamp? That way we
> have
> >> > some refreshing mechanism that isn't as powerful as the complete reset
> >> > (which may not be practical in many scenarios).
> >> >
> >> > I also agree with Randall's other point (Would it be better to not
> >> > automatically reset connector's active topics when a sink connector is
> >> > restarted?). I think keeping the behavior as symmetrical between sink
> >> and
> >> > source connectors is a good idea.
> >> >
> >> > Lastly, with regards to the API, I can imagine it is also pretty
> useful
> >> to
> >> > answer the inverse question: "which connectors write to topic X".
> >> Perhaps
> >> > we can achieve this by letting the users compute it and just expose an
> >> API
> >> > that returns the entire mapping at once (instead of needing to call
> the
> >> > /connectors/{name}/topics endpoint for each connector).
> >> >
> >> > Otherwise, looks good to me! Hits the requirements that I had in mind
> on
> >> > the nose.
> >> > - Almog
> >> >
> >> > On Wed, Jan 15, 2020 at 1:14 AM Tom Bentley 
> >> wrote:
> >> >
> >> > > Hi Konstantine,
> >> > >
> >> > > Thanks for the KIP, I can see how it could be useful.
> >> > >
> >> > > a) Did you consider using a metric for this? I don't think it would
> >> > satisfy
> >> > > all the use cases you have in mind, but you could mention it in the
> >> > > rejected alternatives.
> >> > >
> >> > > b) If the topic name contains the string "-connector" then the key
> >> format
> >> > > is ambiguous. This isn't 

Build failed in Jenkins: kafka-trunk-jdk11 #1084

[Apache Jenkins Server]

See 


Changes:

[vvcephei] KAFKA-6144: IQ option to query standbys (#7962)

[vahid.hashemian] MINOR: Removed accidental double negation in error message. 
(#7834)


--
[...truncated 2.85 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task 

[jira] [Created] (KAFKA-9442) Kafka connect REST API times out when trying to create a connector

[Kedar Shenoy (Jira)]

Kedar Shenoy created KAFKA-9442:
---

 Summary: Kafka connect REST API times out when trying to create a 
connector
 Key: KAFKA-9442
 URL: https://issues.apache.org/jira/browse/KAFKA-9442
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.3.1
 Environment: Kafka - Amazon MSK running version 2.3.1 with only 
Plaintext configured
Connect - Docker image version : latest (5.4.0) running on Amazon ECS cluster 
with tasks configured. I have attached a snapshot of the environment variables 
passed to the image. 
Reporter: Kedar Shenoy
 Attachments: 404.txt, Startup-log.txt, environment-variables.png, get 
call1.jpg, get-wrong-connector-name.png, post-s3.png

REST API for some resources results in a timeout and there is no exception in 
the logs. 

Summary of what was done so far:
 * Kafka connect docker image running as a ECS service with 2 tasks 
 ** Service starts up and creates the 3 topics as expected.
 ** There are a few warnings during start up but no errors - logs attached
 * REST API 
 ** Root resource / returns the connect version.
 ** GET call to /connectors returns a empty list as expected.
 ** wrong resource e.g. /connectors1 returns a 404 
 ** GET call with a wrong id results in a timeout e,g, /connectors/abc where 
abc does not exist
 ** POST call to create a connector config results in timeout
 ** POST call to create a connector with some wrong data returns a 400 e.g. 
providing different values for attribute "name" in top level vs config entry

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9441) Refactor commit logic

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-9441:
--

 Summary: Refactor commit logic
 Key: KAFKA-9441
 URL: https://issues.apache.org/jira/browse/KAFKA-9441
 Project: Kafka
  Issue Type: Sub-task
  Components: streams
Reporter: Matthias J. Sax


Using producer per thread in combination with EOS, it's not possible any longer 
to commit individual task independently (as done currently).

We need to refactor StreamsThread, to commit all tasks at the same time for the 
new model.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Randall Hauch]

On Wed, Jan 15, 2020 at 4:36 PM Randall Hauch  wrote:

> On Wed, Jan 15, 2020 at 2:05 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
>>
>> 9. I assumed that partitioning is implied by default, because there's no
>> requirement for complete ordering of topic status records. But I'll add
>> this fact as a separate bullet. The status.storage.topic is already a
>> partitioned topic.
>>
>
> Agreed. I think it'd be sufficient to simply mention that partition will
> be chosen based upon the active topic records' keys, ensuring that all
> active topic records for the same connector will be written to the same
> partition and will be totally ordered.
>

Well, my previous statement is not quite right. All topic records for the
same *connector and topic* will be written to the same partition and will
be totally ordered. But as you pointed out, it doesn't really matter, other
than that this feature will work with any # of partitions. The new bullet
you described would be sufficient. :-D


>
>>
>> I'm following up with the rest of the comments, shortly.
>> Thanks,
>> Konstantine
>>
>>
>> On Wed, Jan 15, 2020 at 9:19 AM Almog Gavra  wrote:
>>
>> > Hi Konstantine,
>> >
>> > Thanks for the KIP! This is going to make automatic integration with
>> > Connect much more powerful.
>> >
>> > My thoughts are mostly around freshness of the data and being able to
>> > expose that to users. Riffing on Randall's timestamp question - have we
>> > considered adding some interval at which point a connector will
>> republish
>> > any topics that it encounters and update the timestamp? That way we have
>> > some refreshing mechanism that isn't as powerful as the complete reset
>> > (which may not be practical in many scenarios).
>> >
>> > I also agree with Randall's other point (Would it be better to not
>> > automatically reset connector's active topics when a sink connector is
>> > restarted?). I think keeping the behavior as symmetrical between sink
>> and
>> > source connectors is a good idea.
>> >
>> > Lastly, with regards to the API, I can imagine it is also pretty useful
>> to
>> > answer the inverse question: "which connectors write to topic X".
>> Perhaps
>> > we can achieve this by letting the users compute it and just expose an
>> API
>> > that returns the entire mapping at once (instead of needing to call the
>> > /connectors/{name}/topics endpoint for each connector).
>> >
>> > Otherwise, looks good to me! Hits the requirements that I had in mind on
>> > the nose.
>> > - Almog
>> >
>> > On Wed, Jan 15, 2020 at 1:14 AM Tom Bentley 
>> wrote:
>> >
>> > > Hi Konstantine,
>> > >
>> > > Thanks for the KIP, I can see how it could be useful.
>> > >
>> > > a) Did you consider using a metric for this? I don't think it would
>> > satisfy
>> > > all the use cases you have in mind, but you could mention it in the
>> > > rejected alternatives.
>> > >
>> > > b) If the topic name contains the string "-connector" then the key
>> format
>> > > is ambiguous. This isn't necessarily fatal because the value will
>> > > disambiguate, but it could be misleading. Any reason not to just use a
>> > JSON
>> > > key, and simplify the value?
>> > >
>> > > c) I didn't understand this part: "As soon as a worker detects the
>> > addition
>> > > of a topic to a connector's set of active topics, the worker will
>> cease
>> > to
>> > > post update messages to the status.storage.topic for that connector.
>> ".
>> > I'm
>> > > sure I've overlooking something but why is this necessary? Is this
>> were
>> > the
>> > > task id in the value is used?
>> > >
>> > > Thanks again,
>> > >
>> > > Tom
>> > >
>> > > On Wed, Jan 15, 2020 at 12:15 AM Randall Hauch 
>> wrote:
>> > >
>> > > > Oh, one more thing:
>> > > >
>> > > > 9. There's no mention of how the status topic is partitioned, or how
>> > > > partitioning will be used by the new topic records. The KIP should
>> > > probably
>> > > > outline this for clarity and completeness.
>> > > >
>> > > > Best regards,
>> > > >
>> > > > Randall
>> > > >
>> > > > On Tue, Jan 14, 2020 at 5:25 PM Randall Hauch 
>> > wrote:
>> > > >
>> > > > > Thanks, Konstantine. Overall, this KIP looks interesting and
>> really
>> > > > > useful, and for the most part is spot on. I do have a number of
>> > > > > questions/comments about specifics:
>> > > > >
>> > > > >1. The topic records have a value that includes the connector
>> > name,
>> > > > >task number that last reported the topic is used, and the topic
>> > > name.
>> > > > >There's no mention of record timestamps, but I wonder if it'd
>> be
>> > > > useful to
>> > > > >record this. One challenge might be that a connector does not
>> > write
>> > > > to a
>> > > > >topic for a while or the task remains running for long periods
>> of
>> > > > time and
>> > > > >therefore the worker doesn't record that this topic has been
>> newly
>> > > > written
>> > > > >to since it the task was restarted. IOW, the semantics of the
>> > > > timestamp may

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Randall Hauch]

My responses are inline:

On Wed, Jan 15, 2020 at 2:05 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:

> Hi Randall, Tom and Almog. I'm excited to read your comments. I'll reply in
> separate emails, in order.
>
> First, to Randall's comments, I'm replying below with a reference to the
> comment number:
>
> 1. Although I can imagine we'd be interested in adding additional metadata
> in the record value, I didn't see the need for a timestamp in this first
> draft.
> Now that you mention, the way I'd interpret a timestamp in the connector
> status record value would be as an approximation of since when this
> connector has been using this topic.
> Happy to add this if we think this info is useful. Of course, accuracy of
> this information depends on message retention in Kafka and on how long the
> workers have been running without a restart, so this might make this
> approximation less useful if it gets recomputed from time to time.
> To your reference in "Recording active topics" I'll reply below, because
> that's Tom's question too.
>

Makes sense that the timestamp in the connector is the (approximate) time
that the connector has been using the topic. I do think it's worth adding
in the record value (not relying upon Kafka record timestamp).

Regarding "message retention", by default Connect creates the status topic
with compaction but no deletion policy, which means infinite retention.
Don't several things become problematic if finite retention is used on the
status topic, or do we need to worry about this for the active topic
records. Do we need to periodically rewrite all of the active topic
records? If so, we could just write new records using the original
timestamp as originally read by the worker. If the worker does periodically
(maybe just on task startup) rewrite the active topic records, then we'd
have to be sure about the semantics of and interplay with concurrent
explicit "reset" calls.


>
> 2. I'll explain with an example, that maybe is worth adding to the KIP
> because what's expected to happen might not be as obvious as I thought when
> a new topic is recorded.
> Let's say we have two workers, W1 and W2, each running two worker tasks T11
> T12 and T21 T22 respectively associated with a connector C1. All tasks will
> run producers that will produce records to the same topic, "test-topic".
> When the connector starts, both workers track this connector's set of
> active topics as empty. Given the absence of synchronization (that's good)
> in how this information is recorded and persisted in the status topic, all
> four tasks might race to record status messages:
>
> For example:
>
> T11, running at worker W1, will send Kafka records with:
> key: topic-test-topic-connector-C1
> value: "topic": {  "connector": "some-source",  "task": "some-source-TT11",
>  "name": "test-topic" }
>
> and T22, running at worker W2, will send Kafka records with:
> key: topic-test-topic-connector-C1
> value: "topic": {  "connector": "some-source",  "task": "some-source-TT22",
>  "name": "test-topic" }
>
> (similarly tasks T12 and T21 might send topic status records).
>
> These four records (they might not even be four but there's going to be at
> least one) may be written in any order. Because the topic is compacted and
> these records have the same key, eventually only one message will be
> retained.
> The task ID of that message will be the ID of the task that wrote last. I
> can see this being used mostly for troubleshooting.
>

Thanks for the clarification. Might be good to clarify the language a bit
more, though I'm not convinced an example is really needed.


>
> 3. I believe across the whole KIP, when I'm referring to the task entity, I
> imply the worker task. Not the user code that is running as implementation
> of the SourceTask or SinkTask abstract classes. Didn't want to increase
> complexity by referring to a task as worker task.
> But I see your point and I'm going to prefer the terms "worker" and "worker
> task" to highlight that it's the framework that is aware of this feature
> and not the user code.
>

Thank you.


>
> 4. I assumed that absence of changes to the public API would indicate that
> these interfaces/abstract classes remain unchanged. But definitely it's
> worth to explicitly mention that.
>

Thanks!


>
> 5. That is correct. My intention is to make reset work well with the
> streaming programming model. Resetting (which btw is not mandatory) means
> that you are cleaning the slate for a connector that is currently running,
> and its currently active topics will soon be populated from scratch because
> new records will be produced or consumed.
> But resetting is not required. I see it more like a useful operation, in
> case users want to clean the active topics history, without having to
> delete a connector, since delete has further implications in the
> connector's progress tracking.
>

I do think it's worth trying to clarify in the document what happens when
active 

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Vito Jeng]

Hi, folks,

Thank you suggestion, really appreciate it. :)
I understand your concern. I'll merge StreamsNotRunningException and
StateStoreNotAvailableException.


---
Vito


On Thu, Jan 16, 2020 at 6:22 AM John Roesler  wrote:

> Hey Vito,
>
> Yes, thanks for the KIP. Sorry the discussion has been so long.
> Hopefully, we can close it out soon.
>
> I agree we can drop StreamsNotRunningException in favor of
> just StateStoreNotAvailableException.
>
> Unfortunately, I have some higher-level concerns. The value
> of these exceptions is that they tell you how to handle the
> various situations that can arise while querying a distributed
> data store.
>
> Ideally, as a caller, I should be able to just catch "retriable" or
> "fatal" and handle them appropriately. Otherwise, there's no
> point in having categories, and we should just have all the
> exceptions extend InvalidStateStoreException.
>
> Presently, it's not possible to tell from just the
> "retriable"/"fatal" distinction what to do. You  can tell
> from the descriptions of the various exceptions. E.g.:
>
> Retriable:
>  * StreamsRebalancingException: the exact same call
> should just be retried until the rebalance is complete
>  * StateStoreMigratedException: the store handle is
> now invalid, so you need to re-discover the instance
> and get a new handle on that instance. In other words,
> the query itself may be valid, but the particular method
> invocation on this particular instance has encountered
> a fatal exception.
>
> Fatal:
>  * UnknownStateStoreException: this is truly fatal. No amount
> of retrying or re-discovering is going to get you a handle on a
> store that doesn't exist in the cluster.
>  * StateStoreNotAvailableException: this is actually recoverable,
> since the store might exist in the cluster, but isn't available on
> this particular instance (which is shut down or whatever).
>
> Personally, I'm not a fan of code bureaucracy, so I'm 100% fine
> with omitting the categorization and just having 5 subclasses
> of InvalidStateStoreException. Each of them would tell you
> how to handle them, and it's not too many to really
> understand and handle each one.
>
> If you really want to have a middle tier, I'd recommend:
> * RetryableStateStoreException: the exact same call
> should be repeated.
> * RecoverableStateStoreException: the store handle
> should be discarded and the caller should re-discover
> the location of the store and repeat the query on the
> correct instance.
> * FatalStateStoreException: the query/request is totally
> invalid and will never succeed.
>
> However, attempting to categorize the proposed exceptions
> reveals even problems with this categorization:
> Retriable:
> * StreamsRebalancingException
> Recoverable:
> * StateStoreMigratedException
> * StreamsNotRunningException
> Fatal:
> * UnknownStateStoreException
>
> But StreamsNotStartedException is strange... It means that
> one code path got a handle on a specific KafkaStreams object
> instance and sent it a query before another code path
> invoked the start() method on the exact same object instance.
> It seems like the most likely scenario is that whoever wrote
> the program just forgot to call start() before querying, in
> which case, retrying isn't going to help, and a fatal exception
> is more appropriate. I.e., it sounds like a "first 15 minutes
> experience" problem, and making it fatal would be more
> helpful. Even in a production context, there's no reason not
> to sequence your application startup such that you don't
> accept queries until after Streams is started. Thus, I guess
> I'd categorize it under "fatal".
>
> Regardless of whether you make it fatal or retriable, you'd
> still have a whole category with only one exception in it,
> and the other two categories only have two exceptions.
> Plus, as you pointed out in the KIP, you can't get all
> exceptions in all cases anyway:
> * store() can only throw NotStarted, NotRunning,
> and Unknown
> * actual store queries can only throw Rebalancing,
> Migrated, and NotRunning
>
> Thus, in practice also, there are exactly three categories
> and also exactly three exception types. It doesn't seem
> like there's a great advantage to the categories here. To
> avoid the categorization problem and also to clarify what
> exceptions can actually be thrown in different circumstances,
> it seems like we should just:
> * get rid of the middle tier and make all the exceptions
> extend InvalidStateStoreException
> * drop StateStoreNotAvailableException in favor of
> StreamsNotRunningException
> * clearly document on all public methods which exceptions
> need to be handled
>
> How do you feel about this?
> Thanks,
> -John
>
> On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> > Thanks for KIP Vito.
> >
> > Overall the KIP LGTM, but I'd have to agree with others on merging the
> > `StreamsNotRunningException` and 

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[John Roesler]

Hey Vito,

Yes, thanks for the KIP. Sorry the discussion has been so long. 
Hopefully, we can close it out soon.

I agree we can drop StreamsNotRunningException in favor of 
just StateStoreNotAvailableException.

Unfortunately, I have some higher-level concerns. The value 
of these exceptions is that they tell you how to handle the 
various situations that can arise while querying a distributed 
data store.

Ideally, as a caller, I should be able to just catch "retriable" or 
"fatal" and handle them appropriately. Otherwise, there's no 
point in having categories, and we should just have all the 
exceptions extend InvalidStateStoreException.

Presently, it's not possible to tell from just the 
"retriable"/"fatal" distinction what to do. You  can tell
from the descriptions of the various exceptions. E.g.:

Retriable:
 * StreamsRebalancingException: the exact same call 
should just be retried until the rebalance is complete
 * StateStoreMigratedException: the store handle is
now invalid, so you need to re-discover the instance 
and get a new handle on that instance. In other words,
the query itself may be valid, but the particular method 
invocation on this particular instance has encountered 
a fatal exception.

Fatal:
 * UnknownStateStoreException: this is truly fatal. No amount 
of retrying or re-discovering is going to get you a handle on a 
store that doesn't exist in the cluster.
 * StateStoreNotAvailableException: this is actually recoverable, 
since the store might exist in the cluster, but isn't available on 
this particular instance (which is shut down or whatever).

Personally, I'm not a fan of code bureaucracy, so I'm 100% fine 
with omitting the categorization and just having 5 subclasses 
of InvalidStateStoreException. Each of them would tell you 
how to handle them, and it's not too many to really 
understand and handle each one.

If you really want to have a middle tier, I'd recommend:
* RetryableStateStoreException: the exact same call 
should be repeated.
* RecoverableStateStoreException: the store handle 
should be discarded and the caller should re-discover 
the location of the store and repeat the query on the 
correct instance.
* FatalStateStoreException: the query/request is totally 
invalid and will never succeed.

However, attempting to categorize the proposed exceptions 
reveals even problems with this categorization:
Retriable:
* StreamsRebalancingException
Recoverable:
* StateStoreMigratedException
* StreamsNotRunningException
Fatal:
* UnknownStateStoreException

But StreamsNotStartedException is strange... It means that 
one code path got a handle on a specific KafkaStreams object 
instance and sent it a query before another code path 
invoked the start() method on the exact same object instance. 
It seems like the most likely scenario is that whoever wrote 
the program just forgot to call start() before querying, in 
which case, retrying isn't going to help, and a fatal exception 
is more appropriate. I.e., it sounds like a "first 15 minutes 
experience" problem, and making it fatal would be more 
helpful. Even in a production context, there's no reason not 
to sequence your application startup such that you don't 
accept queries until after Streams is started. Thus, I guess 
I'd categorize it under "fatal".

Regardless of whether you make it fatal or retriable, you'd 
still have a whole category with only one exception in it, 
and the other two categories only have two exceptions. 
Plus, as you pointed out in the KIP, you can't get all 
exceptions in all cases anyway:
* store() can only throw NotStarted, NotRunning, 
and Unknown
* actual store queries can only throw Rebalancing, 
Migrated, and NotRunning

Thus, in practice also, there are exactly three categories 
and also exactly three exception types. It doesn't seem 
like there's a great advantage to the categories here. To 
avoid the categorization problem and also to clarify what 
exceptions can actually be thrown in different circumstances, 
it seems like we should just:
* get rid of the middle tier and make all the exceptions 
extend InvalidStateStoreException
* drop StateStoreNotAvailableException in favor of 
StreamsNotRunningException
* clearly document on all public methods which exceptions 
need to be handled

How do you feel about this?
Thanks,
-John

On Wed, Jan 15, 2020, at 15:13, Bill Bejeck wrote:
> Thanks for KIP Vito.
> 
> Overall the KIP LGTM, but I'd have to agree with others on merging the
> `StreamsNotRunningException` and `StateStoreNotAvailableException` classes.
> 
> Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
> NOT_RUNNING || ERROR` I'm not even sure how we could distinguish when to
> use the different
> exceptions.  Maybe a good middle ground would be to have a detailed
> exception message.
> 
> The KIP freeze is close, so I think if we can agree on this, we can wrap up
> the voting soon.
> 
> 

[jira] [Resolved] (KAFKA-3596) Kafka Streams: Window expiration needs to consider more than event time

[Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sophie Blee-Goldman resolved KAFKA-3596.

Resolution: Not A Problem

> Kafka Streams: Window expiration needs to consider more than event time
> ---
>
> Key: KAFKA-3596
> URL: https://issues.apache.org/jira/browse/KAFKA-3596
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Henry Cai
>Priority: Minor
>  Labels: architecture
>
> Currently in Kafka Streams, the way the windows are expired in RocksDB is 
> triggered by new event insertion.  When a window is created at T0 with 10 
> minutes retention, when we saw a new record coming with event timestamp T0 + 
> 10 +1, we will expire that window (remove it) out of RocksDB.
> In the real world, it's very easy to see event coming with future timestamp 
> (or out-of-order events coming with big time gaps between events), this way 
> of retiring a window based on one event's event timestamp is dangerous.  I 
> think at least we need to consider both the event's event time and 
> server/stream time elapse.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-550: Mechanism to Delete Stray Partitions on Broker

[Colin McCabe]

On Wed, Jan 15, 2020, at 03:54, Dhruvil Shah wrote:
> Hi Colin,
> 
> We could add a configuration to disable stray partition deletion if needed,
> but I wasn't sure if an operator would really want to disable it. Perhaps
> if the implementation were buggy, the configuration could be used to
> disable the feature until a bug fix is made. Is that the kind of use case
> you were thinking of?
> 
> I was thinking that there would not be any delay between detection and
> deletion of stray logs. We would schedule an async task to do the actual
> deletion though.

Based on my experience in HDFS, immediately deleting data that looks out of 
place can cause severe issues when a bug occurs.  See 
https://issues.apache.org/jira/browse/HDFS-6186 for details.  So I really do 
think there should be a delay, and a metric + log message in the meantime to 
alert the operators to what is about to happen.

best,
Colin

> 
> Thanks,
> Dhruvil
> 
> On Tue, Jan 14, 2020 at 11:04 PM Colin McCabe  wrote:
> 
> > Hi Dhruvil,
> >
> > Thanks for the KIP.  I think there should be some way to turn this off, in
> > case that becomes necessary.  I'm also curious how long we intend to wait
> > between detecting the duplication and  deleting the extra logs.  The KIP
> > says "scheduled for deletion" but doesn't give a time frame -- is it
> > assumed to be immediate?
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jan 14, 2020, at 05:56, Dhruvil Shah wrote:
> > > If there are no more questions or concerns, I will start a vote thread
> > > tomorrow.
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Mon, Jan 13, 2020 at 6:59 PM Dhruvil Shah 
> > wrote:
> > >
> > > > Hi Nikhil,
> > > >
> > > > Thanks for looking at the KIP. The kind of race condition you mention
> > is
> > > > not possible as stray partition detection is done synchronously while
> > > > handling the LeaderAndIsrRequest. In other words, we atomically
> > evaluate
> > > > the partitions the broker must host and the extra partitions it is
> > hosting
> > > > and schedule deletions based on that.
> > > >
> > > > One possible shortcoming of the KIP is that we do not have the ability
> > to
> > > > detect a stray partition if the topic has been recreated since. We will
> > > > have the ability to disambiguate between different generations of a
> > > > partition with KIP-516.
> > > >
> > > > Thanks,
> > > > Dhruvil
> > > >
> > > > On Sat, Jan 11, 2020 at 11:40 AM Nikhil Bhatia 
> > > > wrote:
> > > >
> > > >> Thanks Dhruvil, the proposal looks reasonable to me.
> > > >>
> > > >> is there a potential of a race between a new topic being assigned to
> > the
> > > >> same node that is still performing a cleanup of the stray partition ?
> > > >> Topic
> > > >> ID will definitely solve this issue.
> > > >>
> > > >> Thanks
> > > >> Nikhil
> > > >>
> > > >> On 2020/01/06 04:30:20, Dhruvil Shah  wrote:
> > > >> > Here is the link to the KIP:>
> > > >> >
> > > >>
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-550%3A+Mechanism+to+Delete+Stray+Partitions+on+Broker
> > > >> >
> > > >>
> > > >> >
> > > >> > On Mon, Jan 6, 2020 at 9:59 AM Dhruvil Shah 
> > > >> wrote:>
> > > >> >
> > > >> > > Hi all, I would like to kick off discussion for KIP-550 which
> > proposes
> > > >> a>
> > > >> > > mechanism to detect and delete stray partitions on a broker.
> > > >> Suggestions>
> > > >> > > and feedback are welcome.>
> > > >> > >>
> > > >> > > - Dhruvil>
> > > >> > >>
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Colin McCabe]

On Wed, Jan 15, 2020, at 13:41, Ron Dagostino wrote:
> Hi Colin.  Two things come to mind with respect to ZooKeeper camelCase
> style vs Kafka-style config names for ZooKeeper.  First, I think it
> would be desirable for the client configs and broker configs to be
> interoperable.  For example, it feels like it would be convenient to
> be able to pass the broker's config file to the ZK Security Migrator
> tool.  I think whatever we use (ZooKeeper camelCase style or Kafka
> style), I think we should use the same for both.
> 
> The second thing to think about is the fact that ZooKeeper
> configuration inherently uses Java system properties as a first pass.
> So while we might switch to Kafka-style configs in the config file
> (e.g. zookeeper.ssl.trust.store.location), the
> org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
> configs are set) is going to look for the camelCase
> zookeeper.ssl.trustStore.location system property and use any value
> there.  I think this could be very confusing for people.  Granted, we
> are doing this so that people don't have to use system properties, but
> there is no way to turn off the use of system properties, so I think
> there would be pretty substantial potential for confusion.

I don't think we document or recommend that anyone use system properties to 
configure Zookeeper usage within  Kafka.  And I think the reason is exactly the 
issue you pointed out -- it wouldn't be consistent with our other 
configurations.

best,
Colin

> 
> One idea is to migrate the system properties -- i.e. look to see if
> zookeeper.ssl.trustStore.location is set and if it is clear out the
> value there and move it under the zookeeper.ssl.trust.store.location
> system property.  (I said it was an idea -- not necessarily a good one
> :-)
> 
> Ron
> 
> On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe  wrote:
> >
> > On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> > > Thanks Colin and Rajini.
> > >
> > > I've updated the KIP to say that KIP-421 functionality is available to
> > > encrypt sensitive configs like the ZK key store and trust store
> > > passwords.  (I've also made it clear that the configs are not
> > > dynamically reconfigurable since dynamic values are stored in ZK and
> > > the data is needed to get to ZK in the first place).  Colin, let me
> > > know if you think anything else needs to be said/done about it -- I
> > > wasn't sure if your comment above implies that there are additional
> > > direct actions that we need to take aside from this.
> > >
> > > I agree that making the brokers' ZooKeeper clients dynamic with
> > > respect to key and trust stores (e.g. via
> > > zookeeper.ssl.context.supplier.class) may increase the risk that this
> > > KIP may not land in time for 2.5.
> > >
> > > Regarding the config names for ZK being different than the ZooKeeper
> > > configs (e.g. camel-case keyStore/trustStore instead of
> > > keystore/truststore, ciphersuites instead of cipher.suites,
> > > enabledProtocols instead of enabled.protocols), I agree it is an
> > > issue.  I tried to mitigate it by putting warning text in the config
> > > docs for these.  Regarding configuration inheritance, I think what you
> > > are saying is that for any configs where the broker has a clear
> > > semantic equivalent, the broker could fall back to the broker value if
> > > the ZK value is not given.  The potential list is:
> > >
> > > zookeeper.ssl.keyStore.location => ssl.keystore.location
> > > zookeeper.ssl.keyStore.password => ssl.keystore.password
> > > zookeeper.ssl.keyStore.type => ssl.keystore.type
> > > zookeeper.ssl.trustStore.location => ssl.truststore.location
> > > zookeeper.ssl.trustStore.password => ssl.truststore.password
> > > zookeeper.ssl.trustStore.type => ssl.truststore.type
> > > zookeeper.ssl.ciphersuites => ssl.cipher.suites
> > >
> > > Note that there are two configs that look the same based on their key
> > > names but whose allowable values differ:
> > >
> > > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
> > > zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> > > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> > >
> > > Not sure what the right answer is, but hopefully this detail will help
> > > get us there.
> > >
> > > Ron
> >
> > I think, on balance, I agree with Rajini that it would be nice to make the 
> > configs look more like Kafka configs.  We don't have camel-case in 
> > configuration keys, so we should avoid it here.
> >
> > best,
> > Colin
> >
> > >
> > > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe  wrote:
> > > >
> > > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > > > Hi Ron,
> > > > >
> > > > > Thanks for the detailed explanation, sounds good to me.
> > > > >
> > > > > A few more questions:
> > > > >
> > > > > 1) At the moment, all sensitive broker configs including
> > > > > keystore/truststore passwords can be stored encrypted in ZooKeeper 
> > > > > prior to
> > 

Re: [VOTE] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Matthias J. Sax]

Thanks for pushing this KIP over the finish line!

+1 (binding)


-Matthias

On 1/15/20 12:57 PM, Bill Bejeck wrote:
> Thanks for the KIP.
> 
> +1 (binding)
> 
> -Bill
> 
> On Wed, Jan 15, 2020 at 3:45 PM M. Manna  wrote:
> 
>> +1 (non-binding)
>>
>> Thanks for this KIP
>>
>> Regards,
>>
>> On Wed, 15 Jan 2020 at 20:35, Mitchell  wrote:
>>
>>> +1(non-binding)
>>>
>>> Very useful
>>> -mitch
>>>
>>> On Wed, Jan 15, 2020, 3:29 PM Anna McDonald 
>>> wrote:
>>>
 Greetings,
 I would like to propose a vote on KIP-399, extending the
 ProductionExceptionHandler to cover serialization exceptions. This KIP
 is aimed at improving the error-handling semantics in Kafka Streams
 when Kafka Streams fails to serialize a message to the downstream
 sink.

 KIP details located here:


>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions

 Discussion Thread:


>>>
>> https://lists.apache.org/thread.html/rbbc887ca31d46f6e73ffc6e08df7e4bda69c89ff820986c30274e272%40%3Cdev.kafka.apache.org%3E

 Thanks,
 anna

>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Ron Dagostino]

Hi Colin.  Two things come to mind with respect to ZooKeeper camelCase
style vs Kafka-style config names for ZooKeeper.  First, I think it
would be desirable for the client configs and broker configs to be
interoperable.  For example, it feels like it would be convenient to
be able to pass the broker's config file to the ZK Security Migrator
tool.  I think whatever we use (ZooKeeper camelCase style or Kafka
style), I think we should use the same for both.

The second thing to think about is the fact that ZooKeeper
configuration inherently uses Java system properties as a first pass.
So while we might switch to Kafka-style configs in the config file
(e.g. zookeeper.ssl.trust.store.location), the
org.apache.zookeeper.client.ZKClientConfig class (which is how TLS
configs are set) is going to look for the camelCase
zookeeper.ssl.trustStore.location system property and use any value
there.  I think this could be very confusing for people.  Granted, we
are doing this so that people don't have to use system properties, but
there is no way to turn off the use of system properties, so I think
there would be pretty substantial potential for confusion.

One idea is to migrate the system properties -- i.e. look to see if
zookeeper.ssl.trustStore.location is set and if it is clear out the
value there and move it under the zookeeper.ssl.trust.store.location
system property.  (I said it was an idea -- not necessarily a good one
:-)

Ron

On Wed, Jan 15, 2020 at 3:54 PM Colin McCabe  wrote:
>
> On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> > Thanks Colin and Rajini.
> >
> > I've updated the KIP to say that KIP-421 functionality is available to
> > encrypt sensitive configs like the ZK key store and trust store
> > passwords.  (I've also made it clear that the configs are not
> > dynamically reconfigurable since dynamic values are stored in ZK and
> > the data is needed to get to ZK in the first place).  Colin, let me
> > know if you think anything else needs to be said/done about it -- I
> > wasn't sure if your comment above implies that there are additional
> > direct actions that we need to take aside from this.
> >
> > I agree that making the brokers' ZooKeeper clients dynamic with
> > respect to key and trust stores (e.g. via
> > zookeeper.ssl.context.supplier.class) may increase the risk that this
> > KIP may not land in time for 2.5.
> >
> > Regarding the config names for ZK being different than the ZooKeeper
> > configs (e.g. camel-case keyStore/trustStore instead of
> > keystore/truststore, ciphersuites instead of cipher.suites,
> > enabledProtocols instead of enabled.protocols), I agree it is an
> > issue.  I tried to mitigate it by putting warning text in the config
> > docs for these.  Regarding configuration inheritance, I think what you
> > are saying is that for any configs where the broker has a clear
> > semantic equivalent, the broker could fall back to the broker value if
> > the ZK value is not given.  The potential list is:
> >
> > zookeeper.ssl.keyStore.location => ssl.keystore.location
> > zookeeper.ssl.keyStore.password => ssl.keystore.password
> > zookeeper.ssl.keyStore.type => ssl.keystore.type
> > zookeeper.ssl.trustStore.location => ssl.truststore.location
> > zookeeper.ssl.trustStore.password => ssl.truststore.password
> > zookeeper.ssl.trustStore.type => ssl.truststore.type
> > zookeeper.ssl.ciphersuites => ssl.cipher.suites
> >
> > Note that there are two configs that look the same based on their key
> > names but whose allowable values differ:
> >
> > zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
> > zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> > ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> >
> > Not sure what the right answer is, but hopefully this detail will help
> > get us there.
> >
> > Ron
>
> I think, on balance, I agree with Rajini that it would be nice to make the 
> configs look more like Kafka configs.  We don't have camel-case in 
> configuration keys, so we should avoid it here.
>
> best,
> Colin
>
> >
> > On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe  wrote:
> > >
> > > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > > Hi Ron,
> > > >
> > > > Thanks for the detailed explanation, sounds good to me.
> > > >
> > > > A few more questions:
> > > >
> > > > 1) At the moment, all sensitive broker configs including
> > > > keystore/truststore passwords can be stored encrypted in ZooKeeper 
> > > > prior to
> > > > starting up brokers. We are now adding SSL keystore/truststore passwords
> > > > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > > > these to connect to ZK. We should perhaps document that these configs 
> > > > can
> > > > be encrypted using KIP-421.
> > >
> > > That's a good point.  Previously, we didn't have ZK on-the-wire security 
> > > support.  However, now that we do, sending sensitive keystore and 
> > > truststore passwords to ZK in cleartext should be 

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Konstantine Karantasis]

Hi Tom,

here are my replies to your comments:

a) Interesting point. It's worth indeed adding it in the "Rejected
Alternatives" sections. I did not consider it as an implementation option.
Connect has already a REST API that, as you note, seems a more natural
choice, especially since its already in place and this feature actually
fits well under the connector endpoint namespace. See KIP-495 for an
interesting discussion on doubling down to the initial design decision to
use the REST API instead of jmx for similar things.

b) I thought hard but briefly about this. Although there's potential for
ambiguity if someone inspects the key on its own, I believe that if we
combine this information with what we save in the value, the ambiguity is
removed. With respect to whether a connector can override another
connector's key, I don't think that's possible for any combination of
unique topic name and connector name. I hope I'm not missing anything.

c) That means that worker tasks will produce topic status records once they
detect that the worker they are running on does not include a topic that is
used by the tasks in the list of active topics for this connector. But once
the worker adds this topic to the set of active topics for this connector,
then the worker tasks stop producing those messages. They'll produce one
again if the worker stops including that topic in the active topics set for
some reason. I could improve the wording on the KIP, but I'd also like to
know we are on the same page re: the design here.

Let me know if the above address your questions.
Thanks,
Konstantine



On Wed, Jan 15, 2020 at 12:05 PM Randall Hauch  wrote:

> Almog,
>
> You raise some interesting questions. Comments inline below.
>
> On Wed, Jan 15, 2020 at 11:19 AM Almog Gavra  wrote:
>
> > Hi Konstantine,
> >
> > Thanks for the KIP! This is going to make automatic integration with
> > Connect much more powerful.
> >
> > My thoughts are mostly around freshness of the data and being able to
> > expose that to users. Riffing on Randall's timestamp question - have we
> > considered adding some interval at which point a connector will republish
> > any topics that it encounters and update the timestamp? That way we have
> > some refreshing mechanism that isn't as powerful as the complete reset
> > (which may not be practical in many scenarios).
> >
>
> My question about recording the timestamp at which each active topic record
> were (infrequently) written was more about making a bit more information
> available given the current design, and whether recording a bit more
> information (for very little additional storage cost and no extra runtime
> cost) may be worth it if in the future we figure out how to use this
> information.
>
> I think it's more complicated to try to record the history of when topics
> were most recently used, since that requires recording a lot more active
> topic records than the current proposal. Besides, it's not unexpected that
> source and sink connectors sometimes don't use topics for periods of time.
> A sink connector only consumes from a topic when there are additional
> records to consume, and a source connector only needs to write to a topic
> when there is information in the upstream system targeted to that topic. An
> example of the latter is that most database connectors will write to a
> topic for a particular table only when rows in that table have changed.
>
>
> > I also agree with Randall's other point (Would it be better to not
> > automatically reset connector's active topics when a sink connector is
> > restarted?). I think keeping the behavior as symmetrical between sink and
> > source connectors is a good idea.
> >
> > Lastly, with regards to the API, I can imagine it is also pretty useful
> to
> > answer the inverse question: "which connectors write to topic X". Perhaps
> > we can achieve this by letting the users compute it and just expose an
> API
> > that returns the entire mapping at once (instead of needing to call the
> > /connectors/{name}/topics endpoint for each connector).
>
>
> It may be worth considering a method such as /topics that would produce a
> result that is an aggregate of the potentially many
> /connectors/{name}/topic responses, especially if the result of the /topics
> is an array of the individual responses from /connectors/{name}/topic. The
> benefit of a single request is that the answer to "which connectors use
> topic X" can be computed using simple tools such as 'jq'. And, if tooling
> frequently fetches the active topic information for all connectors,
> providing the aggregate method would reduce the load on the tooling and the
> server. It may also be relatively easy to implement.
>
>
> > Otherwise, looks good to me! Hits the requirements that I had in mind on
> > the nose.
> > - Almog
> >
> > On Wed, Jan 15, 2020 at 1:14 AM Tom Bentley  wrote:
> >
> > > Hi Konstantine,
> > >
> > > Thanks for the KIP, I can see how it could be useful.
> > >
> > > a) Did 

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

[Bill Bejeck]

Thanks for KIP Vito.

Overall the KIP LGTM, but I'd have to agree with others on merging the
`StreamsNotRunningException` and `StateStoreNotAvailableException` classes.

Since in both cases, the thread state is in `PENDING_SHUTDOWN ||
NOT_RUNNING || ERROR` I'm not even sure how we could distinguish when to
use the different
exceptions.  Maybe a good middle ground would be to have a detailed
exception message.

The KIP freeze is close, so I think if we can agree on this, we can wrap up
the voting soon.

Thanks,
Bill

On Tue, Jan 14, 2020 at 2:12 PM Matthias J. Sax 
wrote:

> Vito,
>
> It's still unclear to me what the advantage is, to have both
> `StreamsNotRunningException` and `StateStoreNotAvailableException`?
>
> For both cased, the state is `PENDING_SHUTDOWN / NOT_RUNNING / ERROR`
> and thus, for a user point of view, why does it matter if the store is
> closed on not? I don't understand why/how this information would be
> useful? Do you have a concrete example in mind how a user would react
> differently to both exceptions?
>
>
> @Vinoth: about `StreamsRebalancingException` -- to me, it seems best to
> actually do this on a per-query basis, ie, have an overload
> `KafkaStreams#store(...)` that takes a boolean flag that allow to
> _disable_ the exception and opt-in to query a active store during
> recovery. However, as KIP-535 actually introduces this change in
> behavior, I think KIP-216 should not cover this, but KIP-535 should be
> updated. I'll follow up on the other KIP thread to raise this point.
>
>
> -Matthias
>
> On 1/11/20 12:26 AM, Vito Jeng wrote:
> > Hi, Matthias & Vinoth,
> >
> > Thanks for the feedback.
> >
> >> What is still unclear to me is, what we gain by having both
> >> `StreamsNotRunningException` and `StateStoreNotAvailableException`. Both
> >> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> >> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> >> state store is closed on not -- I can't query it anyway? Maybe I miss
> >> something thought?
> >
> > Yes, both `StreamsNotRunningException` and
> > `StateStoreNotAvailableException` are fatal exception.
> > But `StateStoreNotAvailableException` is fatal exception about state
> store
> > related.
> > I think it would be helpful that if user need to distinguish these two
> > different case to handle it.
> >
> > I'm not very sure, does that make sense?
> >
> >
> > ---
> > Vito
> >
> >
> > On Fri, Jan 10, 2020 at 3:35 AM Vinoth Chandar 
> wrote:
> >
> >> +1 on merging `StreamsNotRunningException` and
> >> `StateStoreNotAvailableException`, both exceptions are fatal anyway. IMO
> >> its best to have these exceptions be about the state store (and not
> streams
> >> state), to easier understanding.
> >>
> >> Additionally, KIP-535 allows for querying of state stores in rebalancing
> >> state. So do we need the StreamsRebalancingException?
> >>
> >>
> >> On 2020/01/09 03:38:11, "Matthias J. Sax" 
> wrote:
> >>> Sorry that I dropped the ball on this...
> >>>
> >>> Thanks for updating the KIP. Overall LGTM now. Feel free to start a
> VOTE
> >>> thread.
> >>>
> >>> What is still unclear to me is, what we gain by having both
> >>> `StreamsNotRunningException` and `StateStoreNotAvailableException`.
> Both
> >>> exception are thrown when KafkaStreams is in state PENDING_SHUTDOWN /
> >>> NOT_RUNNING / ERROR. Hence, as a user what do I gain to know if the
> >>> state store is closed on not -- I can't query it anyway? Maybe I miss
> >>> something thought?
> >>>
> >>>
> >>> -Matthias
> >>>
> >>>
> >>> On 11/3/19 6:07 PM, Vito Jeng wrote:
>  Sorry for the late reply, thanks for the review.
> 
> 
> > About `StateStoreMigratedException`:
> >
> > Why is it only thrown if the state is REBALANCING? A store might be
> > migrated during a rebalance, and Kafka Streams might resume back to
> > RUNNING state and afterward somebody tries to use an old store
> handle.
> > Also, if state is REBALANCING, should we throw
> > `StreamThreadRebalancingException`? Hence, I think
> > `StateStoreMigratedException` does only make sense during `RUNNING`
> >> state.
> >
> 
>  Thank you point this, already updated.
> 
> 
>  Why do we need to distinguish between
> `KafkaStreamsNotRunningException`
> > and `StateStoreNotAvailableException`?
> >
> 
>  `KafkaStreamsNotRunningException` may be caused by various reasons, I
> >> think
>  it would be helpful that the
>  user can distinguish whether it is caused by the state store closed.
>  (Maybe I am wrong...)
> 
> 
>  Last, why do we distinguish between `KafkaStreams` instance and
> > `StreamsThread`? To me, it seems we should always refer to the
> >> instance,
> > because that is the level of granularity in which we enable/disable
> >> IQ atm.
> >
> 
>  Totally agree. Do you mean the naming of state store exceptions?
>  I don't have special reason to 

Re: [VOTE] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Bill Bejeck]

Thanks for the KIP.

+1 (binding)

-Bill

On Wed, Jan 15, 2020 at 3:45 PM M. Manna  wrote:

> +1 (non-binding)
>
> Thanks for this KIP
>
> Regards,
>
> On Wed, 15 Jan 2020 at 20:35, Mitchell  wrote:
>
> > +1(non-binding)
> >
> > Very useful
> > -mitch
> >
> > On Wed, Jan 15, 2020, 3:29 PM Anna McDonald 
> > wrote:
> >
> > > Greetings,
> > > I would like to propose a vote on KIP-399, extending the
> > > ProductionExceptionHandler to cover serialization exceptions. This KIP
> > > is aimed at improving the error-handling semantics in Kafka Streams
> > > when Kafka Streams fails to serialize a message to the downstream
> > > sink.
> > >
> > > KIP details located here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
> > >
> > > Discussion Thread:
> > >
> > >
> >
> https://lists.apache.org/thread.html/rbbc887ca31d46f6e73ffc6e08df7e4bda69c89ff820986c30274e272%40%3Cdev.kafka.apache.org%3E
> > >
> > > Thanks,
> > > anna
> > >
> >
>

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Colin McCabe]

On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> Thanks Colin and Rajini.
> 
> I've updated the KIP to say that KIP-421 functionality is available to
> encrypt sensitive configs like the ZK key store and trust store
> passwords.  (I've also made it clear that the configs are not
> dynamically reconfigurable since dynamic values are stored in ZK and
> the data is needed to get to ZK in the first place).  Colin, let me
> know if you think anything else needs to be said/done about it -- I
> wasn't sure if your comment above implies that there are additional
> direct actions that we need to take aside from this.
> 
> I agree that making the brokers' ZooKeeper clients dynamic with
> respect to key and trust stores (e.g. via
> zookeeper.ssl.context.supplier.class) may increase the risk that this
> KIP may not land in time for 2.5.
> 
> Regarding the config names for ZK being different than the ZooKeeper
> configs (e.g. camel-case keyStore/trustStore instead of
> keystore/truststore, ciphersuites instead of cipher.suites,
> enabledProtocols instead of enabled.protocols), I agree it is an
> issue.  I tried to mitigate it by putting warning text in the config
> docs for these.  Regarding configuration inheritance, I think what you
> are saying is that for any configs where the broker has a clear
> semantic equivalent, the broker could fall back to the broker value if
> the ZK value is not given.  The potential list is:
> 
> zookeeper.ssl.keyStore.location => ssl.keystore.location
> zookeeper.ssl.keyStore.password => ssl.keystore.password
> zookeeper.ssl.keyStore.type => ssl.keystore.type
> zookeeper.ssl.trustStore.location => ssl.truststore.location
> zookeeper.ssl.trustStore.password => ssl.truststore.password
> zookeeper.ssl.trustStore.type => ssl.truststore.type
> zookeeper.ssl.ciphersuites => ssl.cipher.suites
> 
> Note that there are two configs that look the same based on their key
> names but whose allowable values differ:
> 
> zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
> zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> 
> Not sure what the right answer is, but hopefully this detail will help
> get us there.
> 
> Ron

I think, on balance, I agree with Rajini that it would be nice to make the 
configs look more like Kafka configs.  We don't have camel-case in 
configuration keys, so we should avoid it here.

best,
Colin

> 
> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe  wrote:
> >
> > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > Hi Ron,
> > >
> > > Thanks for the detailed explanation, sounds good to me.
> > >
> > > A few more questions:
> > >
> > > 1) At the moment, all sensitive broker configs including
> > > keystore/truststore passwords can be stored encrypted in ZooKeeper prior 
> > > to
> > > starting up brokers. We are now adding SSL keystore/truststore passwords
> > > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > > these to connect to ZK. We should perhaps document that these configs can
> > > be encrypted using KIP-421.
> >
> > That's a good point.  Previously, we didn't have ZK on-the-wire security 
> > support.  However, now that we do, sending sensitive keystore and 
> > truststore passwords to ZK in cleartext should be deprecated in favor of 
> > using KIP-421.
> >
> > >
> > > 2) We can dynamically update keystores and trust stores used by brokers
> > > without restarting the broker. Can we support this easily for ZK clients
> > > created by the broker, for example by adding our own
> > > `zookeeper.ssl.context.supplier.class`?
> > >
> >
> > Hmm.  That might be better to consider in a follow-up, since the deadline 
> > for 2.5 KIPs is coming up?
> >
> > best,
> > Colin
> >
> > > 3) Looks like we are using config names that directly map to ZK configs.
> > > Have we considered using equivalent Kafka config names with prefixes,
> > > perhaps with inheritance from the non-prefixed names? Not sure if this is 
> > > a
> > > good idea, but perhaps worth documenting in Rejected Alternatives at 
> > > least.
> > >
> > >
> > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe  wrote:
> > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the explanation.  I guess thinking about it a little bit 
> > > > more,
> > > > we should just add --zk-tls-config-file to all of these commands.
> > > >
> > > > We will be removing this option (plus ZK in general) from these commands
> > > > in the next major release, but ZK is still supported in 2.5, so we 
> > > > should
> > > > just do the logical thing.  (The exception is ZkSecurityMigrator, which
> > > > will stay).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > Hi Colin.
> > > > >
> > > > > <<< It seems like this [--zk-tls-config-file information] could just
> > > > appear
> > > > > in a configuration file, which all of these tools already accept 

Re: [VOTE] KIP-441: Smooth Scaling Out for Kafka Streams

[John Roesler]

Hello all,

After a long hiatus, I've just realized that I'm now able to upgrade my 
non-binding support to a binding +1 for KIP-441.

This brings the vote tally to:
3 binding +1s: Guozhang, Bill, and myself
3 non-binding +1s: Bruno, Vinoth, and Sophie

Since the vote has been open for at least 72 hours, the KIP is accepted.

Thanks all,
-John



On Mon, Oct 28, 2019 at 21:02 PM John Roesler  wrote:
> Hey all,
> 
> Now that the 2.4 release storm is over, I'd like to bump this vote thread.
> 
> Currently, we have two binding +1s (Guozhang and Bill), and four
> non-binding ones (Bruno, Vinoth, Sophie, and myself), and no vetoes.
> 
> Thanks,
> -John
> 
> On Thu, Sep 12, 2019 at 12:54 PM Bill Bejeck  wrote:
> >
> > +1 (binding)
> >
> > On Thu, Sep 12, 2019 at 1:53 PM Sophie Blee-Goldman  
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Wed, Sep 11, 2019 at 11:38 AM Vinoth Chandar  
> > > wrote:
> > >
> > > > +1 (non-binding).
> > > >
> > > > On Fri, Sep 6, 2019 at 12:46 AM Bruno Cadonna  
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Fri, Sep 6, 2019 at 12:32 AM Guozhang Wang  
> > > > > wrote:
> > > > > >
> > > > > > +1 (binding).
> > > > > >
> > > > > > On Thu, Sep 5, 2019 at 2:47 PM John Roesler  
> > > > > > wrote:
> > > > > >
> > > > > > > Hello, all,
> > > > > > >
> > > > > > > After a great discussion, I'd like to open voting on KIP-441,
> > > > > > > to avoid long restore times in Streams after rebalancing.
> > > > > > > Please cast your votes!
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-441:+Smooth+Scaling+Out+for+Kafka+Streams
> > > > > > >
> > > > > > > Thanks,
> > > > > > > -John
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -- Guozhang
> > > > >
> > > >
> > >
> 

Re: [VOTE] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[M. Manna]

+1 (non-binding)

Thanks for this KIP

Regards,

On Wed, 15 Jan 2020 at 20:35, Mitchell  wrote:

> +1(non-binding)
>
> Very useful
> -mitch
>
> On Wed, Jan 15, 2020, 3:29 PM Anna McDonald 
> wrote:
>
> > Greetings,
> > I would like to propose a vote on KIP-399, extending the
> > ProductionExceptionHandler to cover serialization exceptions. This KIP
> > is aimed at improving the error-handling semantics in Kafka Streams
> > when Kafka Streams fails to serialize a message to the downstream
> > sink.
> >
> > KIP details located here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
> >
> > Discussion Thread:
> >
> >
> https://lists.apache.org/thread.html/rbbc887ca31d46f6e73ffc6e08df7e4bda69c89ff820986c30274e272%40%3Cdev.kafka.apache.org%3E
> >
> > Thanks,
> > anna
> >
>

Re: [VOTE] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Mitchell]

+1(non-binding)

Very useful
-mitch

On Wed, Jan 15, 2020, 3:29 PM Anna McDonald  wrote:

> Greetings,
> I would like to propose a vote on KIP-399, extending the
> ProductionExceptionHandler to cover serialization exceptions. This KIP
> is aimed at improving the error-handling semantics in Kafka Streams
> when Kafka Streams fails to serialize a message to the downstream
> sink.
>
> KIP details located here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
>
> Discussion Thread:
>
> https://lists.apache.org/thread.html/rbbc887ca31d46f6e73ffc6e08df7e4bda69c89ff820986c30274e272%40%3Cdev.kafka.apache.org%3E
>
> Thanks,
> anna
>

[VOTE] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Anna McDonald]

Greetings,
I would like to propose a vote on KIP-399, extending the
ProductionExceptionHandler to cover serialization exceptions. This KIP
is aimed at improving the error-handling semantics in Kafka Streams
when Kafka Streams fails to serialize a message to the downstream
sink.

KIP details located here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions

Discussion Thread:
https://lists.apache.org/thread.html/rbbc887ca31d46f6e73ffc6e08df7e4bda69c89ff820986c30274e272%40%3Cdev.kafka.apache.org%3E

Thanks,
anna

Build failed in Jenkins: kafka-trunk-jdk11 #1083

[Apache Jenkins Server]

See 


Changes:

[vvcephei] KAFKA-6144: Add KeyQueryMetadata APIs to KafkaStreams (#7960)


--
[...truncated 5.75 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnAllStoresNames[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessConsumerRecordList[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSinkSpecificSerializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldFlushStoreForFirstInput[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourceThatMatchPattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUpdateStoreForNewKey[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldSendRecordViaCorrectSourceTopicDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldSetRecordMetadata[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForLargerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectInMemoryStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldThrowForMissingTime[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnWallClockTimeDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessRecordForTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldForwardRecordsFromSubtopologyToSubtopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotUpdateStoreForSmallerValue[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Konstantine Karantasis]

Hi Randall, Tom and Almog. I'm excited to read your comments. I'll reply in
separate emails, in order.

First, to Randall's comments, I'm replying below with a reference to the
comment number:

1. Although I can imagine we'd be interested in adding additional metadata
in the record value, I didn't see the need for a timestamp in this first
draft.
Now that you mention, the way I'd interpret a timestamp in the connector
status record value would be as an approximation of since when this
connector has been using this topic.
Happy to add this if we think this info is useful. Of course, accuracy of
this information depends on message retention in Kafka and on how long the
workers have been running without a restart, so this might make this
approximation less useful if it gets recomputed from time to time.
To your reference in "Recording active topics" I'll reply below, because
that's Tom's question too.

2. I'll explain with an example, that maybe is worth adding to the KIP
because what's expected to happen might not be as obvious as I thought when
a new topic is recorded.
Let's say we have two workers, W1 and W2, each running two worker tasks T11
T12 and T21 T22 respectively associated with a connector C1. All tasks will
run producers that will produce records to the same topic, "test-topic".
When the connector starts, both workers track this connector's set of
active topics as empty. Given the absence of synchronization (that's good)
in how this information is recorded and persisted in the status topic, all
four tasks might race to record status messages:

For example:

T11, running at worker W1, will send Kafka records with:
key: topic-test-topic-connector-C1
value: "topic": {  "connector": "some-source",  "task": "some-source-TT11",
 "name": "test-topic" }

and T22, running at worker W2, will send Kafka records with:
key: topic-test-topic-connector-C1
value: "topic": {  "connector": "some-source",  "task": "some-source-TT22",
 "name": "test-topic" }

(similarly tasks T12 and T21 might send topic status records).

These four records (they might not even be four but there's going to be at
least one) may be written in any order. Because the topic is compacted and
these records have the same key, eventually only one message will be
retained.
The task ID of that message will be the ID of the task that wrote last. I
can see this being used mostly for troubleshooting.

3. I believe across the whole KIP, when I'm referring to the task entity, I
imply the worker task. Not the user code that is running as implementation
of the SourceTask or SinkTask abstract classes. Didn't want to increase
complexity by referring to a task as worker task.
But I see your point and I'm going to prefer the terms "worker" and "worker
task" to highlight that it's the framework that is aware of this feature
and not the user code.

4. I assumed that absence of changes to the public API would indicate that
these interfaces/abstract classes remain unchanged. But definitely it's
worth to explicitly mention that.

5. That is correct. My intention is to make reset work well with the
streaming programming model. Resetting (which btw is not mandatory) means
that you are cleaning the slate for a connector that is currently running,
and its currently active topics will soon be populated from scratch because
new records will be produced or consumed.
But resetting is not required. I see it more like a useful operation, in
case users want to clean the active topics history, without having to
delete a connector, since delete has further implications in the
connector's progress tracking.

6. I fixed the typo - thanks! I'm very much in favor of preserving symmetry
between the two connector types. This has definitely more long term
benefits and may help to avoid confusion. However, the asymmetry is
inherited here by the asymmetry that exists today between source and sink
connectors.
Source connector don't list topics in their configurations but sink
connectors do. So, if a user reconfigures a sink connector with a different
set of topics, if we don't reset the topics based on the new configs (and
my thought here was to match the new configuration with the set of active
topics), the old topics, currently not listed in the connectors
configuration, will keep showing up as active topics. The user will have to
explicitly reset the active topics after reconfiguring to avoid this. If
there's consensus that preserving this asymmetry is worse than having to
reset the active topics, I'm happy to change this in the KIP.

7. What I try to avoid here is the following situation: For some reason (a
sequence of failures to write tombstones to the status topic), stale topic
status records remain in that topic even after a connector has been
deleted. Requiring to restart a connector with the same name just to apply
a follow up reset of active topics doesn't seem necessary. I like the idea
of decoupling connector existence from the maintenance of the status topic.
Of 

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Randall Hauch]

Almog,

You raise some interesting questions. Comments inline below.

On Wed, Jan 15, 2020 at 11:19 AM Almog Gavra  wrote:

> Hi Konstantine,
>
> Thanks for the KIP! This is going to make automatic integration with
> Connect much more powerful.
>
> My thoughts are mostly around freshness of the data and being able to
> expose that to users. Riffing on Randall's timestamp question - have we
> considered adding some interval at which point a connector will republish
> any topics that it encounters and update the timestamp? That way we have
> some refreshing mechanism that isn't as powerful as the complete reset
> (which may not be practical in many scenarios).
>

My question about recording the timestamp at which each active topic record
were (infrequently) written was more about making a bit more information
available given the current design, and whether recording a bit more
information (for very little additional storage cost and no extra runtime
cost) may be worth it if in the future we figure out how to use this
information.

I think it's more complicated to try to record the history of when topics
were most recently used, since that requires recording a lot more active
topic records than the current proposal. Besides, it's not unexpected that
source and sink connectors sometimes don't use topics for periods of time.
A sink connector only consumes from a topic when there are additional
records to consume, and a source connector only needs to write to a topic
when there is information in the upstream system targeted to that topic. An
example of the latter is that most database connectors will write to a
topic for a particular table only when rows in that table have changed.


> I also agree with Randall's other point (Would it be better to not
> automatically reset connector's active topics when a sink connector is
> restarted?). I think keeping the behavior as symmetrical between sink and
> source connectors is a good idea.
>
> Lastly, with regards to the API, I can imagine it is also pretty useful to
> answer the inverse question: "which connectors write to topic X". Perhaps
> we can achieve this by letting the users compute it and just expose an API
> that returns the entire mapping at once (instead of needing to call the
> /connectors/{name}/topics endpoint for each connector).


It may be worth considering a method such as /topics that would produce a
result that is an aggregate of the potentially many
/connectors/{name}/topic responses, especially if the result of the /topics
is an array of the individual responses from /connectors/{name}/topic. The
benefit of a single request is that the answer to "which connectors use
topic X" can be computed using simple tools such as 'jq'. And, if tooling
frequently fetches the active topic information for all connectors,
providing the aggregate method would reduce the load on the tooling and the
server. It may also be relatively easy to implement.


> Otherwise, looks good to me! Hits the requirements that I had in mind on
> the nose.
> - Almog
>
> On Wed, Jan 15, 2020 at 1:14 AM Tom Bentley  wrote:
>
> > Hi Konstantine,
> >
> > Thanks for the KIP, I can see how it could be useful.
> >
> > a) Did you consider using a metric for this? I don't think it would
> satisfy
> > all the use cases you have in mind, but you could mention it in the
> > rejected alternatives.
> >
> > b) If the topic name contains the string "-connector" then the key format
> > is ambiguous. This isn't necessarily fatal because the value will
> > disambiguate, but it could be misleading. Any reason not to just use a
> JSON
> > key, and simplify the value?
> >
> > c) I didn't understand this part: "As soon as a worker detects the
> addition
> > of a topic to a connector's set of active topics, the worker will cease
> to
> > post update messages to the status.storage.topic for that connector. ".
> I'm
> > sure I've overlooking something but why is this necessary? Is this were
> the
> > task id in the value is used?
> >
> > Thanks again,
> >
> > Tom
> >
> > On Wed, Jan 15, 2020 at 12:15 AM Randall Hauch  wrote:
> >
> > > Oh, one more thing:
> > >
> > > 9. There's no mention of how the status topic is partitioned, or how
> > > partitioning will be used by the new topic records. The KIP should
> > probably
> > > outline this for clarity and completeness.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Tue, Jan 14, 2020 at 5:25 PM Randall Hauch 
> wrote:
> > >
> > > > Thanks, Konstantine. Overall, this KIP looks interesting and really
> > > > useful, and for the most part is spot on. I do have a number of
> > > > questions/comments about specifics:
> > > >
> > > >1. The topic records have a value that includes the connector
> name,
> > > >task number that last reported the topic is used, and the topic
> > name.
> > > >There's no mention of record timestamps, but I wonder if it'd be
> > > useful to
> > > >record this. One challenge might be that a connector does not
> 

Re: [DISCUSS] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Bill Bejeck]

Great all of that sounds good to me.

Since we are all in agreement, we can move to a vote.

-Bill

On Wed, Jan 15, 2020 at 11:51 AM am  wrote:

> I agree WARN seems like it would be fine given we already log at that
> level for the handler. It also seems reasonable to exclude the
> ClassCastException as that can indicate something other then a simple
> serialization exception and would keep the current behavior.
>
> anna
>
> On Tue, Jan 14, 2020 at 11:41 PM Matthias J. Sax 
> wrote:
> >
> > I was just checking the existing code, and we currently log at WARN
> > level if the handler returns CONTINUE -- we did not have any complaints
> > about it, hence, I don't see an issue with WARN -- and we should keep it
> > consistent.
> >
> > I think the explicit mentioning of `ClassCastException` is based on the
> > current code that catches this exception to rethrow it -- this was a
> > minor improvement to help people to more easily detect miss-configure
> > serdes.
> >
> > In think, we can just catch all exception and the handler can decide
> > what to do. Thinking about this once more, it might actually be better
> > if we could _exclude_ `ClassCastException` as it may indicate a miss
> > configured Serde?
> >
> >
> > -Matthias
> >
> > On 1/14/20 4:15 PM, Bill Bejeck wrote:
> > > Hi Anna,
> > >
> > > Thanks for getting this KIP going again.
> > >
> > > I agree with pushing forward on option 0 as well.  I a couple of
> questions
> > > about the KIP as written.
> > >
> > > The KIP states that any {{ClassCastException}} thrown plus any other
> > > unchecked exceptions will result in a log statement and not stop
> processing
> > > if the handler returns CONTINUE.
> > >
> > >1. I'm wondering if DEBUG is the correct level vs. a WARN,
> although, at
> > >WARN, we could end up spamming the log file.
> > >2. Are allowing all unchecked exceptions to proceed to permissive?
> I
> > >could be overly cautious here, but I'm afraid of masking a serious
> > >problem.
> > >
> > > Overall I'm in favor of this KIP and if you feel it's good as is, I
> > > wouldn't block on these questions  I just wanted to throw in my 2
> cents.
> > >
> > > Thanks,
> > > Bill
> > >
> > > On Sat, Jan 11, 2020 at 7:02 PM Mitchell  wrote:
> > >
> > >> I'm happy to have the serialization handler now.  I've hit this issue
> > >> a number of times in the past.
> > >>
> > >> I think the other options are also large enough they probably deserve
> > >> their own KIPs to properly document the changes.
> > >> -mitch
> > >>
> > >> On Fri, Jan 10, 2020 at 7:33 PM am  wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> I would like to re-restart the discussion of KIP-399
> > >>>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
> > >>>
> > >>> The last conversation centered on if this KIP should address the
> issues
> > >>> around state store/change log divergence with Matthias presenting
> three
> > >>> options:
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> *To move this KIP forward, maybe we can just (0) add the handler
> > >>> forserialization exceptions when writing into any topic and consider
> it
> > >>> anincremental improvement. Ie, (1) we keep the door open to let state
> > >>> andchangelog topic diverge (current status) (2) we allow people to
> > >>> violateEOS (current state) (3) and we don't improve the handling of
> DSL
> > >>> statestore serialization exceptions.We could address (1), (2),
> and/or (3)
> > >>> in follow up KIPs.Thoughts? Let us know if you only want to address
> (0),
> > >> or
> > >>> extend thecurrent KIP to include any of (1-3).*
> > >>>
> > >>>
> > >>> I would like to propose we go with option 0 and treat this as an
> > >>> incremental improvement that applies to any topic and address the
> issue
> > >> of
> > >>> divergence in future KIP(s).
> > >>>
> > >>> Feedback, thoughts and musings appreciated,
> > >>>
> > >>> anna
> > >>
> > >
> >
>

[jira] [Created] (KAFKA-9440) Add ConsumerGroupCommand to delete static members

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9440:
--

 Summary: Add ConsumerGroupCommand to delete static members
 Key: KAFKA-9440
 URL: https://issues.apache.org/jira/browse/KAFKA-9440
 Project: Kafka
  Issue Type: Improvement
Reporter: Boyang Chen


We introduced a new AdminClient API removeMembersFromConsumerGroup in 2.4. It 
would be good to instantiate the API as part of the ConsumerGroupCommand for 
easy command line usage. 
This change requires a new KIP, and just posting out here in case anyone who 
uses static membership to pick it up, if they would like to use.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9439) Add more public API tests for KafkaProducer

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9439:
--

 Summary: Add more public API tests for KafkaProducer
 Key: KAFKA-9439
 URL: https://issues.apache.org/jira/browse/KAFKA-9439
 Project: Kafka
  Issue Type: Improvement
Reporter: Boyang Chen


While working on KIP-447, we realized a lack of test coverage on the 
KafkaProducer public APIs. For example, `commitTransaction` and 
`sendOffsetsToTransaction` are not even called in the `KafkaProducerTest.java` 
and the code coverage is only 75%. 

Adding more unit tests here will be pretty valuable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[Guozhang Wang]

Thanks for the update of the PR John! I have taken a look at 7962 and it
looks good to me overall.


Guozhang

On Wed, Jan 15, 2020 at 9:35 AM John Roesler  wrote:

> Hello again all,
>
> I had a bit of inspiration last night and realized that it's not necessary
> (and maybe even inappropriate) for StreamThreadStateStoreProvider
> and WrappingStoreProvider to implement the public StateStoreProvider
> interface.
>
> By breaking this dependency, I was able to implement the flag without
> touching
> any public interfaces except adding the new overload to KafkaStreams as
> originally
> discussed.
>
> You can take a look at https://github.com/apache/kafka/pull/7962 for the
> details.
>
> Since there was no objection to that new overload, I'll go ahead and
> update the KIP
> and we can proceed with a final round of code reviews on
> https://github.com/apache/kafka/pull/7962
>
> Thanks, all,
> -John
>
> On Tue, Jan 14, 2020, at 22:52, Matthias J. Sax wrote:
> > Thanks. SGTM.
> >
> > -Matthias
> >
> > On 1/14/20 4:52 PM, John Roesler wrote:
> > > Hey Matthias,
> > >
> > > Thanks for taking a look! I felt a little uneasy about it, but didn’t
> think about the case you pointed out. Throwing an exception would indeed be
> safer.
> > >
> > > Given a choice between throwing in the default method or defining a
> new interface and throwing if the wrong interface is implemented, it seems
> nicer for everyone to go the default method route. Since we’re not
> referencing the other method anymore, I should probably deprecate it, too.
> > >
> > > Thanks again for your help. I really appreciate it.
> > >
> > > -John
> > >
> > > On Tue, Jan 14, 2020, at 18:15, Matthias J. Sax wrote:
> > >> Thanks for the PR. That helps a lot.
> > >>
> > >> I actually do have a concern: the proposed default method, would
> disable
> > >> the new feature to allow querying an active task during restore
> > >> automatically. Hence, if a user has an existing custom store type, and
> > >> would use the new
> > >>
> > >> KafkaStreams.store(..., true)
> > >>
> > >> method to enable querying during restore it would not work, and it
> would
> > >> be unclear why. It would even be worth if there are two developers and
> > >> one provide the store type while the other one just uses it.
> > >>
> > >> Hence, the default implementation should maybe throw an exception by
> > >> default? Or maybe, we would introduce a new interface that extends
> > >> `QueryableStoreType` and add this new method. For this case, we could
> > >> check within
> > >>
> > >> KafkaStreams.store(..., true)
> > >>
> > >> if the StoreType implements the new interface and if not, throw an
> > >> exception there.
> > >>
> > >> Those exceptions would be more descriptive (ie, state store does not
> > >> support querying during restore) and give the user a chance to figure
> > >> out what's wrong.
> > >>
> > >> Not sure if overwriting a default method or a new interface is the
> > >> better choice to let people opt-into the feature.
> > >>
> > >>
> > >>
> > >> -Matthias
> > >>
> > >> On 1/14/20 3:22 PM, John Roesler wrote:
> > >>> Hi again all,
> > >>>
> > >>> I've sent a PR including this new option, and while implementing it,
> I
> > >>> realized we also have to make a (source-compatible) addition to the
> > >>> QueryableStoreType interface, so that the IQ store wrapper can pass
> the
> > >>> flag down to the composite store provider.
> > >>>
> > >>> See https://github.com/apache/kafka/pull/7962
> > >>> In particular:
> https://github.com/apache/kafka/pull/7962/files#diff-d0242b7289f4e0886490351a5a803d41
> > >>>
> > >>> If there are no objections to these additions, I'll update the KIP
> tomorrow.
> > >>>
> > >>> Thanks,
> > >>> -John
> > >>>
> > >>> On Tue, Jan 14, 2020, at 14:11, John Roesler wrote:
> >  Thanks for calling this out, Matthias. You're correct that this
> looks like a
> >  harmful behavioral change. I'm fine with adding the new overload you
> >  mentioned, just a simple boolean flag to enable the new behavior.
> > 
> >  I'd actually propose that we call this flag "queryStaleData", with
> a default
> >  of "false". The meaning of this would be to preserve exactly the
> existing
> >  semantics. I.e., that the store must be both active and running to
> be
> >  included.
> > 
> >  It seems less severe to just suddenly start returning queries from
> standbys,
> >  but in the interest of safety, the easiest thing is just flag the
> whole feature.
> > 
> >  If you, Navinder, and Vinoth agree, we can just update the KIP. It
> seems like
> >  a pretty uncontroversial amendment to avoid breaking query
> consistency
> >  semantics.
> > 
> >  Thanks,
> >  -John
> > 
> > 
> >  On Tue, Jan 14, 2020, at 13:21, Matthias J. Sax wrote:
> > > During the discussion of KIP-216
> > > (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Ron Dagostino]

Thanks Colin and Rajini.

I've updated the KIP to say that KIP-421 functionality is available to
encrypt sensitive configs like the ZK key store and trust store
passwords.  (I've also made it clear that the configs are not
dynamically reconfigurable since dynamic values are stored in ZK and
the data is needed to get to ZK in the first place).  Colin, let me
know if you think anything else needs to be said/done about it -- I
wasn't sure if your comment above implies that there are additional
direct actions that we need to take aside from this.

I agree that making the brokers' ZooKeeper clients dynamic with
respect to key and trust stores (e.g. via
zookeeper.ssl.context.supplier.class) may increase the risk that this
KIP may not land in time for 2.5.

Regarding the config names for ZK being different than the ZooKeeper
configs (e.g. camel-case keyStore/trustStore instead of
keystore/truststore, ciphersuites instead of cipher.suites,
enabledProtocols instead of enabled.protocols), I agree it is an
issue.  I tried to mitigate it by putting warning text in the config
docs for these.  Regarding configuration inheritance, I think what you
are saying is that for any configs where the broker has a clear
semantic equivalent, the broker could fall back to the broker value if
the ZK value is not given.  The potential list is:

zookeeper.ssl.keyStore.location => ssl.keystore.location
zookeeper.ssl.keyStore.password => ssl.keystore.password
zookeeper.ssl.keyStore.type => ssl.keystore.type
zookeeper.ssl.trustStore.location => ssl.truststore.location
zookeeper.ssl.trustStore.password => ssl.truststore.password
zookeeper.ssl.trustStore.type => ssl.truststore.type
zookeeper.ssl.ciphersuites => ssl.cipher.suites

Note that there are two configs that look the same based on their key
names but whose allowable values differ:

zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)

Not sure what the right answer is, but hopefully this detail will help
get us there.

Ron

On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe  wrote:
>
> On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > Hi Ron,
> >
> > Thanks for the detailed explanation, sounds good to me.
> >
> > A few more questions:
> >
> > 1) At the moment, all sensitive broker configs including
> > keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> > starting up brokers. We are now adding SSL keystore/truststore passwords
> > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > these to connect to ZK. We should perhaps document that these configs can
> > be encrypted using KIP-421.
>
> That's a good point.  Previously, we didn't have ZK on-the-wire security 
> support.  However, now that we do, sending sensitive keystore and truststore 
> passwords to ZK in cleartext should be deprecated in favor of using KIP-421.
>
> >
> > 2) We can dynamically update keystores and trust stores used by brokers
> > without restarting the broker. Can we support this easily for ZK clients
> > created by the broker, for example by adding our own
> > `zookeeper.ssl.context.supplier.class`?
> >
>
> Hmm.  That might be better to consider in a follow-up, since the deadline for 
> 2.5 KIPs is coming up?
>
> best,
> Colin
>
> > 3) Looks like we are using config names that directly map to ZK configs.
> > Have we considered using equivalent Kafka config names with prefixes,
> > perhaps with inheritance from the non-prefixed names? Not sure if this is a
> > good idea, but perhaps worth documenting in Rejected Alternatives at least.
> >
> >
> > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe  wrote:
> >
> > > Hi Ron,
> > >
> > > Thanks for the explanation.  I guess thinking about it a little bit more,
> > > we should just add --zk-tls-config-file to all of these commands.
> > >
> > > We will be removing this option (plus ZK in general) from these commands
> > > in the next major release, but ZK is still supported in 2.5, so we should
> > > just do the logical thing.  (The exception is ZkSecurityMigrator, which
> > > will stay).
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > Hi Colin.
> > > >
> > > > <<< It seems like this [--zk-tls-config-file information] could just
> > > appear
> > > > in a configuration file, which all of these tools already accept (I
> > > think)
> > > >
> > > > ZkSecurityMigrator has no such property file facility; adding a
> > > > "--zk-tls-config-file" parameter is exactly for this purpose.  If we add
> > > > that to ZkSecurityMigrator then it is trivial to add it to other 
> > > > commands
> > > > (the same code is simply reused; it ends up being just a few extra
> > > lines).
> > > > I do not see any parameter in the other two commands to adjust the ZK
> > > > connection; ConfigCommand accepts a "--command-config" flag, but
> > > 

Build failed in Jenkins: kafka-trunk-jdk8 #4161

[Apache Jenkins Server]

See 


Changes:

[vvcephei] KAFKA-6144: Add KeyQueryMetadata APIs to KafkaStreams (#7960)


--
[...truncated 2.82 MB...]

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCreateStateDirectoryForStatefulTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfWallClockTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis STARTED

org.apache.kafka.streams.MockTimeTest > shouldGetNanosAsMillis PASSED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime STARTED

org.apache.kafka.streams.MockTimeTest > shouldSetStartTime PASSED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldNotAllowNegativeSleep PASSED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep STARTED

org.apache.kafka.streams.MockTimeTest > shouldAdvanceTimeOnSleep PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldReturnName 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldPutWindowStartTimestampWithUnknownTimestamp PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > 
shouldReturnIsPersistent PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testConsumerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testToString STARTED

org.apache.kafka.streams.test.TestRecordTest > testToString PASSED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords STARTED

org.apache.kafka.streams.test.TestRecordTest > testInvalidRecords PASSED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:testClasses
> Task :streams:upgrade-system-tests-0100:checkstyleTest
> Task :streams:upgrade-system-tests-0100:spotbugsMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:test
> Task :streams:upgrade-system-tests-0101:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0101:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0101:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0101:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0101:compileTestJava
> Task 

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

[John Roesler]

Hello again all,

I had a bit of inspiration last night and realized that it's not necessary
(and maybe even inappropriate) for StreamThreadStateStoreProvider 
and WrappingStoreProvider to implement the public StateStoreProvider interface.

By breaking this dependency, I was able to implement the flag without touching
any public interfaces except adding the new overload to KafkaStreams as 
originally
discussed.

You can take a look at https://github.com/apache/kafka/pull/7962 for the 
details.

Since there was no objection to that new overload, I'll go ahead and update the 
KIP
and we can proceed with a final round of code reviews on 
https://github.com/apache/kafka/pull/7962

Thanks, all,
-John

On Tue, Jan 14, 2020, at 22:52, Matthias J. Sax wrote:
> Thanks. SGTM.
> 
> -Matthias
> 
> On 1/14/20 4:52 PM, John Roesler wrote:
> > Hey Matthias,
> > 
> > Thanks for taking a look! I felt a little uneasy about it, but didn’t think 
> > about the case you pointed out. Throwing an exception would indeed be safer.
> > 
> > Given a choice between throwing in the default method or defining a new 
> > interface and throwing if the wrong interface is implemented, it seems 
> > nicer for everyone to go the default method route. Since we’re not 
> > referencing the other method anymore, I should probably deprecate it, too. 
> > 
> > Thanks again for your help. I really appreciate it.
> > 
> > -John
> > 
> > On Tue, Jan 14, 2020, at 18:15, Matthias J. Sax wrote:
> >> Thanks for the PR. That helps a lot.
> >>
> >> I actually do have a concern: the proposed default method, would disable
> >> the new feature to allow querying an active task during restore
> >> automatically. Hence, if a user has an existing custom store type, and
> >> would use the new
> >>
> >> KafkaStreams.store(..., true)
> >>
> >> method to enable querying during restore it would not work, and it would
> >> be unclear why. It would even be worth if there are two developers and
> >> one provide the store type while the other one just uses it.
> >>
> >> Hence, the default implementation should maybe throw an exception by
> >> default? Or maybe, we would introduce a new interface that extends
> >> `QueryableStoreType` and add this new method. For this case, we could
> >> check within
> >>
> >> KafkaStreams.store(..., true)
> >>
> >> if the StoreType implements the new interface and if not, throw an
> >> exception there.
> >>
> >> Those exceptions would be more descriptive (ie, state store does not
> >> support querying during restore) and give the user a chance to figure
> >> out what's wrong.
> >>
> >> Not sure if overwriting a default method or a new interface is the
> >> better choice to let people opt-into the feature.
> >>
> >>
> >>
> >> -Matthias
> >>
> >> On 1/14/20 3:22 PM, John Roesler wrote:
> >>> Hi again all,
> >>>
> >>> I've sent a PR including this new option, and while implementing it, I 
> >>> realized we also have to make a (source-compatible) addition to the 
> >>> QueryableStoreType interface, so that the IQ store wrapper can pass the
> >>> flag down to the composite store provider.
> >>>
> >>> See https://github.com/apache/kafka/pull/7962
> >>> In particular: 
> >>> https://github.com/apache/kafka/pull/7962/files#diff-d0242b7289f4e0886490351a5a803d41
> >>>
> >>> If there are no objections to these additions, I'll update the KIP 
> >>> tomorrow.
> >>>
> >>> Thanks,
> >>> -John
> >>>
> >>> On Tue, Jan 14, 2020, at 14:11, John Roesler wrote:
>  Thanks for calling this out, Matthias. You're correct that this looks 
>  like a
>  harmful behavioral change. I'm fine with adding the new overload you
>  mentioned, just a simple boolean flag to enable the new behavior.
> 
>  I'd actually propose that we call this flag "queryStaleData", with a 
>  default
>  of "false". The meaning of this would be to preserve exactly the existing
>  semantics. I.e., that the store must be both active and running to be
>  included.
> 
>  It seems less severe to just suddenly start returning queries from 
>  standbys,
>  but in the interest of safety, the easiest thing is just flag the whole 
>  feature.
> 
>  If you, Navinder, and Vinoth agree, we can just update the KIP. It seems 
>  like
>  a pretty uncontroversial amendment to avoid breaking query consistency
>  semantics.
> 
>  Thanks,
>  -John
> 
> 
>  On Tue, Jan 14, 2020, at 13:21, Matthias J. Sax wrote:
> > During the discussion of KIP-216
> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-216%3A+IQ+should+throw+different+exceptions+for+different+errors)
> > we encountered that KIP-535 introduces a behavior change that is not
> > backward compatible, hence, I would like to request a small change.
> >
> > KIP-535 suggests, that active tasks can be queried during recovery and
> > no exception would be thrown and longer. This is a change in behavior
> > and in fact 

Re: [VOTE] KIP-551: Expose disk read and write metrics

[Colin McCabe]

Thanks, all.  I will close the vote later today.

best,
Colin


On Wed, Jan 15, 2020, at 01:48, Mickael Maison wrote:
> +1 (binding)
> Thanks for the KIP
> 
> On Tue, Jan 14, 2020 at 6:50 PM David Arthur  wrote:
> >
> > +1 binding
> >
> > This will be very nice to have. Thanks for the KIP, Colin.
> >
> > -David
> >
> > On Tue, Jan 14, 2020 at 11:39 AM Sönke Liebau
> >  wrote:
> >
> > > +1 (non-binding)
> > >
> > > Thanks for creating this!
> > >
> > > On Tue, 14 Jan 2020 at 17:36, Mitchell  wrote:
> > >
> > > > +1 (non-binding)!
> > > > Very useful kip.
> > > > -mitch
> > > >
> > > > On Tue, Jan 14, 2020 at 10:26 AM Manikumar 
> > > > wrote:
> > > > >
> > > > > +1 (binding).
> > > > >
> > > > > Thanks for the KIP.
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Jan 12, 2020 at 1:23 AM Lucas Bradstreet 
> > > > wrote:
> > > > >
> > > > > > +1 (non binding)
> > > > > >
> > > > > > On Sat, 11 Jan 2020 at 02:32, M. Manna  wrote:
> > > > > >
> > > > > > > Hey Colin,
> > > > > > >
> > > > > > > +1 - Really useful for folks managing a cluster by themselves.
> > > > > > >
> > > > > > > M. MAnna
> > > > > > >
> > > > > > > On Fri, 10 Jan 2020 at 22:35, Jose Garcia Sancio <
> > > > jsan...@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1, LGTM.
> > > > > > > >
> > > > > > > > On Fri, Jan 10, 2020 at 2:19 PM Gwen Shapira 
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1, thanks for driving this
> > > > > > > > >
> > > > > > > > > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe <
> > > cmcc...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I'd like to start the vote on KIP-551: Expose disk read and
> > > > write
> > > > > > > > > metrics.
> > > > > > > > > >
> > > > > > > > > > KIP:  https://cwiki.apache.org/confluence/x/sotSC
> > > > > > > > > >
> > > > > > > > > > Discussion thread:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > https://lists.apache.org/thread.html/cfaac4426455406abe890464a7f4ae23a5c69a39afde66fe6eb3d696%40%3Cdev.kafka.apache.org%3E
> > > > > > > > > >
> > > > > > > > > > cheers,
> > > > > > > > > > Colin
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > -Jose
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> > >
> > > --
> > > Sönke Liebau
> > > Partner
> > > Tel. +49 179 7940878
> > > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> > >
> >
> >
> > --
> > David Arthur
>

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Colin McCabe]

On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> Hi Ron,
> 
> Thanks for the detailed explanation, sounds good to me.
> 
> A few more questions:
> 
> 1) At the moment, all sensitive broker configs including
> keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> starting up brokers. We are now adding SSL keystore/truststore passwords
> for ZooKeeper client that cannot be stored in ZooKeeper since you need
> these to connect to ZK. We should perhaps document that these configs can
> be encrypted using KIP-421.

That's a good point.  Previously, we didn't have ZK on-the-wire security 
support.  However, now that we do, sending sensitive keystore and truststore 
passwords to ZK in cleartext should be deprecated in favor of using KIP-421.

> 
> 2) We can dynamically update keystores and trust stores used by brokers
> without restarting the broker. Can we support this easily for ZK clients
> created by the broker, for example by adding our own
> `zookeeper.ssl.context.supplier.class`?
> 

Hmm.  That might be better to consider in a follow-up, since the deadline for 
2.5 KIPs is coming up?

best,
Colin

> 3) Looks like we are using config names that directly map to ZK configs.
> Have we considered using equivalent Kafka config names with prefixes,
> perhaps with inheritance from the non-prefixed names? Not sure if this is a
> good idea, but perhaps worth documenting in Rejected Alternatives at least.
> 
> 
> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe  wrote:
> 
> > Hi Ron,
> >
> > Thanks for the explanation.  I guess thinking about it a little bit more,
> > we should just add --zk-tls-config-file to all of these commands.
> >
> > We will be removing this option (plus ZK in general) from these commands
> > in the next major release, but ZK is still supported in 2.5, so we should
> > just do the logical thing.  (The exception is ZkSecurityMigrator, which
> > will stay).
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > Hi Colin.
> > >
> > > <<< It seems like this [--zk-tls-config-file information] could just
> > appear
> > > in a configuration file, which all of these tools already accept (I
> > think)
> > >
> > > ZkSecurityMigrator has no such property file facility; adding a
> > > "--zk-tls-config-file" parameter is exactly for this purpose.  If we add
> > > that to ZkSecurityMigrator then it is trivial to add it to other commands
> > > (the same code is simply reused; it ends up being just a few extra
> > lines).
> > > I do not see any parameter in the other two commands to adjust the ZK
> > > connection; ConfigCommand accepts a "--command-config" flag, but
> > according
> > > to the code "This is used only with --bootstrap-server option for
> > > describing and altering broker configs."
> > >
> > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > time
> > > for the next release.
> > >
> > > ConfigCommand supports the "--bootstrap-server" option and will have its
> > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > special
> > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > > prior to starting Kafka will still exist, so it feels like while
> > > "--zk-tls-config-file" would never be used except for this use case it
> > > could probably still be added for this particular situation.
> > >
> > > Ron
> > >
> > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > sent) did not go through.  I am trying to get emails through now to move
> > > this discussion forward.
> > >
> > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe  wrote:
> > >
> > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > Hi everyone.  I would like to make the following changes to the KIP.
> > > > >
> > > > > MOTIVATION:
> > > > > Include a statement that it will be difficult in the short term to
> > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > (which
> > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > cluster
> > > > > with encrypted passwords in Zookeeper is an explicitly-supported use
> > > > case;
> > > > > therefore it is in scope to be able to securely configure the CLI
> > tools
> > > > > that still leverage non-deprecated direct Zookeeper communication
> > for TLS
> > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > zookeeper-security-migration.sh).
> > > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > About deprecations:
> > > >
> > > > * zookeeper-security-migration: clearly, deprecating ZK access in this
> > one
> > > > would not make sense, since it would defeat the whole point of the
> > tool :)
> > > >
> > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > KIP-455
> > > > implementation has dragged a bit, but this should get 

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Almog Gavra]

Hi Konstantine,

Thanks for the KIP! This is going to make automatic integration with
Connect much more powerful.

My thoughts are mostly around freshness of the data and being able to
expose that to users. Riffing on Randall's timestamp question - have we
considered adding some interval at which point a connector will republish
any topics that it encounters and update the timestamp? That way we have
some refreshing mechanism that isn't as powerful as the complete reset
(which may not be practical in many scenarios).

I also agree with Randall's other point (Would it be better to not
automatically reset connector's active topics when a sink connector is
restarted?). I think keeping the behavior as symmetrical between sink and
source connectors is a good idea.

Lastly, with regards to the API, I can imagine it is also pretty useful to
answer the inverse question: "which connectors write to topic X". Perhaps
we can achieve this by letting the users compute it and just expose an API
that returns the entire mapping at once (instead of needing to call the
/connectors/{name}/topics endpoint for each connector).

Otherwise, looks good to me! Hits the requirements that I had in mind on
the nose.
- Almog

On Wed, Jan 15, 2020 at 1:14 AM Tom Bentley  wrote:

> Hi Konstantine,
>
> Thanks for the KIP, I can see how it could be useful.
>
> a) Did you consider using a metric for this? I don't think it would satisfy
> all the use cases you have in mind, but you could mention it in the
> rejected alternatives.
>
> b) If the topic name contains the string "-connector" then the key format
> is ambiguous. This isn't necessarily fatal because the value will
> disambiguate, but it could be misleading. Any reason not to just use a JSON
> key, and simplify the value?
>
> c) I didn't understand this part: "As soon as a worker detects the addition
> of a topic to a connector's set of active topics, the worker will cease to
> post update messages to the status.storage.topic for that connector. ". I'm
> sure I've overlooking something but why is this necessary? Is this were the
> task id in the value is used?
>
> Thanks again,
>
> Tom
>
> On Wed, Jan 15, 2020 at 12:15 AM Randall Hauch  wrote:
>
> > Oh, one more thing:
> >
> > 9. There's no mention of how the status topic is partitioned, or how
> > partitioning will be used by the new topic records. The KIP should
> probably
> > outline this for clarity and completeness.
> >
> > Best regards,
> >
> > Randall
> >
> > On Tue, Jan 14, 2020 at 5:25 PM Randall Hauch  wrote:
> >
> > > Thanks, Konstantine. Overall, this KIP looks interesting and really
> > > useful, and for the most part is spot on. I do have a number of
> > > questions/comments about specifics:
> > >
> > >1. The topic records have a value that includes the connector name,
> > >task number that last reported the topic is used, and the topic
> name.
> > >There's no mention of record timestamps, but I wonder if it'd be
> > useful to
> > >record this. One challenge might be that a connector does not write
> > to a
> > >topic for a while or the task remains running for long periods of
> > time and
> > >therefore the worker doesn't record that this topic has been newly
> > written
> > >to since it the task was restarted. IOW, the semantics of the
> > timestamp may
> > >be a bit murky. Have you thought about recording the timestamp, and
> > if so
> > >what are the pros and cons?
> > >- The "Recording active topics" section says the following:
> > >   "As soon as a worker detects the addition of a topic to a
> > >   connector's set of active topics, all the connector's tasks that
> > inspect
> > >   source or sink records will cease to post update messages to the
> > >   status.storage.topic."
> > >   This probably means the timestamp won't be very useful.
> > >2. The KIP says "the Kafka record value stores the ID of the task
> that
> > >succeeded to store a topic status record last." However, this is a
> bit
> > >unclear: is it really storing the last task that successfully wrote
> > to that
> > >topic (as this would require very frequent writes to this topic), or
> > is it
> > >more that this is the task that was last *recorded* as having
> written
> > >to the topic? (Here, "recorded" could be a bit of a gray area, since
> > this
> > >would depend on the how the worker periodically records this
> > information.)
> > >Any kind of clarity here might be helpful.
> > >3. In the "Recording active topics" section (and the surrounding
> > >sections), the "task" is used ambiguously. For example, "when its
> > tasks
> > >start processing their first records ... these tasks will start
> > inspecting
> > >which is the Kafka topic of each of these records". IIUC, the first
> > "task"
> > >mentioned is the connector's task, and the second is the worker's
> > task. Do
> > >we need to distinguish this more clearly?
> > >4. 

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Colin McCabe]

Thanks, everyone!

best,
Colin

On Wed, Jan 15, 2020, at 07:50, Sean Glover wrote:
> Congratulations Colin, Vahid and Manikumar and thank you for all your
> excellent work on Apache Kafka!
> 
> On Wed, Jan 15, 2020 at 8:42 AM Ron Dagostino  wrote:
> 
> > Congratulations!
> >
> > > On Jan 15, 2020, at 5:04 AM, Viktor Somogyi-Vass <
> > viktorsomo...@gmail.com> wrote:
> > >
> > > Congrats to you guys, it's a great accomplishment! :)
> > >
> > >> On Wed, Jan 15, 2020 at 10:20 AM David Jacot 
> > wrote:
> > >>
> > >> Congrats!
> > >>
> > >>> On Wed, Jan 15, 2020 at 12:00 AM James Cheng 
> > wrote:
> > >>>
> > >>> Congrats Colin, Vahid, and Manikumar!
> > >>>
> > >>> -James
> > >>>
> >  On Jan 14, 2020, at 10:59 AM, Tom Bentley 
> > wrote:
> > 
> >  Congratulations!
> > 
> >  On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram <
> > >> rajinisiva...@gmail.com>
> >  wrote:
> > 
> > > Congratulations Colin, Vahid and Manikumar!
> > >
> > > Regards,
> > > Rajini
> > >
> > > On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison <
> > >>> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > >> Congrats Colin, Vahid and Manikumar!
> > >>
> > >> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma 
> > >> wrote:
> > >>>
> > >>> Congratulations Colin, Vahid and Manikumar!
> > >>>
> > >>> Ismael
> > >>>
> > >>> On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira 
> > > wrote:
> > >>>
> >  Hi everyone,
> > 
> >  I'm happy to announce that Colin McCabe, Vahid Hashemian and
> > > Manikumar
> >  Reddy are now members of Apache Kafka PMC.
> > 
> >  Colin and Manikumar became committers on Sept 2018 and Vahid on
> > Jan
> >  2019. They all contributed many patches, code reviews and
> > > participated
> >  in many KIP discussions. We appreciate their contributions and are
> >  looking forward to many more to come.
> > 
> >  Congrats Colin, Vahid and Manikumar!
> > 
> >  Gwen, on behalf of Apache Kafka PMC
> > 
> > >>
> > >
> > >>>
> > >>>
> > >>
> >
>

Re: [DISCUSS] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[am]

I agree WARN seems like it would be fine given we already log at that
level for the handler. It also seems reasonable to exclude the
ClassCastException as that can indicate something other then a simple
serialization exception and would keep the current behavior.

anna

On Tue, Jan 14, 2020 at 11:41 PM Matthias J. Sax  wrote:
>
> I was just checking the existing code, and we currently log at WARN
> level if the handler returns CONTINUE -- we did not have any complaints
> about it, hence, I don't see an issue with WARN -- and we should keep it
> consistent.
>
> I think the explicit mentioning of `ClassCastException` is based on the
> current code that catches this exception to rethrow it -- this was a
> minor improvement to help people to more easily detect miss-configure
> serdes.
>
> In think, we can just catch all exception and the handler can decide
> what to do. Thinking about this once more, it might actually be better
> if we could _exclude_ `ClassCastException` as it may indicate a miss
> configured Serde?
>
>
> -Matthias
>
> On 1/14/20 4:15 PM, Bill Bejeck wrote:
> > Hi Anna,
> >
> > Thanks for getting this KIP going again.
> >
> > I agree with pushing forward on option 0 as well.  I a couple of questions
> > about the KIP as written.
> >
> > The KIP states that any {{ClassCastException}} thrown plus any other
> > unchecked exceptions will result in a log statement and not stop processing
> > if the handler returns CONTINUE.
> >
> >1. I'm wondering if DEBUG is the correct level vs. a WARN, although, at
> >WARN, we could end up spamming the log file.
> >2. Are allowing all unchecked exceptions to proceed to permissive?  I
> >could be overly cautious here, but I'm afraid of masking a serious
> >problem.
> >
> > Overall I'm in favor of this KIP and if you feel it's good as is, I
> > wouldn't block on these questions  I just wanted to throw in my 2 cents.
> >
> > Thanks,
> > Bill
> >
> > On Sat, Jan 11, 2020 at 7:02 PM Mitchell  wrote:
> >
> >> I'm happy to have the serialization handler now.  I've hit this issue
> >> a number of times in the past.
> >>
> >> I think the other options are also large enough they probably deserve
> >> their own KIPs to properly document the changes.
> >> -mitch
> >>
> >> On Fri, Jan 10, 2020 at 7:33 PM am  wrote:
> >>>
> >>> Hello,
> >>>
> >>> I would like to re-restart the discussion of KIP-399
> >>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-399%3A+Extend+ProductionExceptionHandler+to+cover+serialization+exceptions
> >>>
> >>> The last conversation centered on if this KIP should address the issues
> >>> around state store/change log divergence with Matthias presenting three
> >>> options:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> *To move this KIP forward, maybe we can just (0) add the handler
> >>> forserialization exceptions when writing into any topic and consider it
> >>> anincremental improvement. Ie, (1) we keep the door open to let state
> >>> andchangelog topic diverge (current status) (2) we allow people to
> >>> violateEOS (current state) (3) and we don't improve the handling of DSL
> >>> statestore serialization exceptions.We could address (1), (2), and/or (3)
> >>> in follow up KIPs.Thoughts? Let us know if you only want to address (0),
> >> or
> >>> extend thecurrent KIP to include any of (1-3).*
> >>>
> >>>
> >>> I would like to propose we go with option 0 and treat this as an
> >>> incremental improvement that applies to any topic and address the issue
> >> of
> >>> divergence in future KIP(s).
> >>>
> >>> Feedback, thoughts and musings appreciated,
> >>>
> >>> anna
> >>
> >
>

[jira] [Created] (KAFKA-9438) Issue with mm2 active/active replication

[Roman (Jira)]

Roman created KAFKA-9438:


 Summary: Issue with mm2 active/active replication
 Key: KAFKA-9438
 URL: https://issues.apache.org/jira/browse/KAFKA-9438
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Affects Versions: 2.4.0
Reporter: Roman


Hi,

 

i am trying to configure the the active/active with new kafka 2.4.0 and MM2.

I have 2 kafka clusters of 3 kafkas, one in lets say BO and on in IN.

In each cluster there are 3 kafkas.

Topics are replicated properly so in BO i see
{quote}topics

in.topics
{quote}
 

in IN i see
{quote}topics

bo.topic
{quote}
 

That should be according to documentation.

 

But when I stop the replication process on one data center and start it up, the 
replication replicate the topics with the same prefix twice bo.bo.topics or 
in.in.topics depending on what connector i restart.

I have also blacklisted the topics but they are still replicating.

 

bo.properties file
{quote}name = in-bo
#topics = .*
topics.blacklist = "bo.*"
#groups = .*
connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
tasks.max = 10

source.cluster.alias = in
target.cluster.alias = bo
source.cluster.bootstrap.servers = IN-kafka1:9092,IN-kafka2:9092,IN-kafka3:9092
target.cluster.bootstrap.servers = BO-kafka1:9092,BO-kafka2:9092,bo-kafka39092


# use ByteArrayConverter to ensure that records are not re-encoded
key.converter = org.apache.kafka.connect.converters.ByteArrayConverter
value.converter = org.apache.kafka.connect.converters.ByteArrayConverter

 
{quote}
in.properties
{quote}name = bo-in
#topics = .*
topics.blacklist = "in.*"
#groups = .*
connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
tasks.max = 10

source.cluster.alias = bo
target.cluster.alias = in
target.cluster.bootstrap.servers = IN-kafka1:9092,IN-kafka2:9092,IN-kafka3:9092
source.cluster.bootstrap.servers = BO-kafka1:9092,BO-kafka2:9092,BO-kafka3:9092


# use ByteArrayConverter to ensure that records are not re-encoded
key.converter = org.apache.kafka.connect.converters.ByteArrayConverter
value.converter = org.apache.kafka.connect.converters.ByteArrayConverter
{quote}
 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

[Kamal Chandraprakash]

+1 (non-binding). Thanks for the KIP!

On Mon, Jan 13, 2020 at 1:58 PM M. Manna  wrote:

> Hi Mikael,
>
> Apologies for last minute question, as I just caught up with it. Thanks for
> your work on the KIP.
>
> Just trying to get your thoughts on one thing (I might have misunderstood
> it) - currently it's possible (even though I am strongly against it) to
> create Kafka topics which are under-replicated; despite all brokers being
> online. This the the output of an intentionally under-replicated topic
> "dummy" with p=6 and RF=1 (with a 3 node cluster)
>
>
> virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> --create --topic dummy --partitions 6 --replication-factor 1
> --bootstrap-server localhost:9092
> virtualadmin@kafka-broker-machine-1:/opt/kafka/bin$ ./kafka-topics.sh
> --describe --topic dummy  --bootstrap-server localhost:9092
> Topic:dummy PartitionCount:6ReplicationFactor:1
>
> Configs:compression.type=gzip,min.insync.replicas=2,cleanup.policy=delete,segment.bytes=10485760,max.message.bytes=10642642,retention.bytes=20971520
> Topic: dummyPartition: 0Leader: 3   Replicas: 3
> Isr: 3
> Topic: dummyPartition: 1Leader: 1   Replicas: 1
> Isr: 1
> Topic: dummyPartition: 2Leader: 2   Replicas: 2
> Isr: 2
> Topic: dummyPartition: 3Leader: 3   Replicas: 3
> Isr: 3
> Topic: dummyPartition: 4Leader: 1   Replicas: 1
> Isr: 1
> Topic: dummyPartition: 5Leader: 2   Replicas: 2
> Isr: 2
>
>  This is with respect to the following statement on your KIP (i.e.
> under-replicated topic creation is also permitted when none is offline):
>
> *but note that this may already happen (without this KIP) when
> > topics/partitions are created while all brokers in a rack are offline
> (ie:
> > an availability zone is offline). Tracking topics/partitions not
> optimally
> > spread across all racks can be tackled in a follow up KIP.  *
>
>
>
>
> Did you mean to say that such under-replicated topics (including
> human-created ones) will be handled in a separete KIP ?
>
> Regards,
>
>
> On Mon, 13 Jan 2020 at 10:15, Mickael Maison 
> wrote:
>
> > Hi all.
> >
> > With 2.5.0 approaching, bumping this thread once more as feedback or
> > votes would be nice.
> >
> > Thanks
> >
> > On Wed, Dec 18, 2019 at 1:59 PM Tom Bentley  wrote:
> > >
> > > +1 non-binding. Thanks!
> > >
> > > On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau
> > >  wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > thanks for your response! That all makes perfect sense and I cannot
> > > > give any actual use cases for where what I asked about would be
> useful
> > > > :)
> > > > It was more the idle thought if this might be low hanging fruit while
> > > > changing this anyway to avoid having to circle back later on and
> > > > wanted to at least mention it.
> > > >
> > > > I am totally happy either way!
> > > >
> > > > Best regards,
> > > > Sönke
> > > >
> > > > On Wed, 18 Dec 2019 at 11:20, Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > Thanks Sönke for the feedback.
> > > > >
> > > > > I debated this point quite a bit before deciding to base creation
> > > > > around "min.insync.replicas".
> > > > >
> > > > > For me, the goal of this KIP is to enable administrators to provide
> > > > > higher availability. In a 3 node cluster configured for high
> > > > > availability (3 replicas, 2 min ISR), by enabling this feature,
> > > > > clusters should be fully usable even when 1 broker is down. This
> > > > > should cover all "normal" maintenance operations like a rolling
> > > > > restart or just the recovery of a broker.
> > > > >
> > > > > At the moment, when creating a topic/partition, the assumption is
> > that
> > > > > the resource will be fully functioning. This KIP does not change
> this
> > > > > assumption. If this is something someone wants, I think it should
> be
> > > > > handled in a different KIP that targets that use case. By relying
> on
> > > > > "min.insync.replicas", we don't break any assumptions the user has
> > and
> > > > > this should be fully transparent from the user point of view.
> > > > >
> > > > > About "min.insync.replicas", one caveat that is not explicit in the
> > > > > KIP is that it's currently possible to create topics with less
> > > > > replicas than this settings. For that reason, I think the
> > > > > implementation will actually rely on min(replicas, min-isr) instead
> > of
> > > > > simply min.insync.replicas. I have updated the KIP to explicitly
> > > > > mention this point.
> > > > >
> > > > > I hope that answers your question, let me know.
> > > > > Thanks
> > > > >
> > > > >
> > > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau
> > > > >  wrote:
> > > > > >
> > > > > > Hi Michael,
> > > > > >
> > > > > > that sounds like a useful addition! I can't help but wonder
> > whether by
> > > > > > leaving in the restriction that 

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Sean Glover]

Congratulations Colin, Vahid and Manikumar and thank you for all your
excellent work on Apache Kafka!

On Wed, Jan 15, 2020 at 8:42 AM Ron Dagostino  wrote:

> Congratulations!
>
> > On Jan 15, 2020, at 5:04 AM, Viktor Somogyi-Vass <
> viktorsomo...@gmail.com> wrote:
> >
> > Congrats to you guys, it's a great accomplishment! :)
> >
> >> On Wed, Jan 15, 2020 at 10:20 AM David Jacot 
> wrote:
> >>
> >> Congrats!
> >>
> >>> On Wed, Jan 15, 2020 at 12:00 AM James Cheng 
> wrote:
> >>>
> >>> Congrats Colin, Vahid, and Manikumar!
> >>>
> >>> -James
> >>>
>  On Jan 14, 2020, at 10:59 AM, Tom Bentley 
> wrote:
> 
>  Congratulations!
> 
>  On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram <
> >> rajinisiva...@gmail.com>
>  wrote:
> 
> > Congratulations Colin, Vahid and Manikumar!
> >
> > Regards,
> > Rajini
> >
> > On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison <
> >>> mickael.mai...@gmail.com>
> > wrote:
> >
> >> Congrats Colin, Vahid and Manikumar!
> >>
> >> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma 
> >> wrote:
> >>>
> >>> Congratulations Colin, Vahid and Manikumar!
> >>>
> >>> Ismael
> >>>
> >>> On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira 
> > wrote:
> >>>
>  Hi everyone,
> 
>  I'm happy to announce that Colin McCabe, Vahid Hashemian and
> > Manikumar
>  Reddy are now members of Apache Kafka PMC.
> 
>  Colin and Manikumar became committers on Sept 2018 and Vahid on
> Jan
>  2019. They all contributed many patches, code reviews and
> > participated
>  in many KIP discussions. We appreciate their contributions and are
>  looking forward to many more to come.
> 
>  Congrats Colin, Vahid and Manikumar!
> 
>  Gwen, on behalf of Apache Kafka PMC
> 
> >>
> >
> >>>
> >>>
> >>
>

Re: CompletableFuture?

[Vamsi Subahsh]

Hi,

I'm interested in picking this up as I have already worked on internal code
bases to make a wrapper on current Api to make it expose CompletableFuture
(using callbacks in the current api).

Could you give me comment/edit access to the confluence doc, I can write up
the new api and the logic proposal?

Regards,
Vamsi Subhash



On Wed, 15 Jan 2020 at 19:59, Ismael Juma  wrote:

> Good question. I have a draft KIP for the producer change:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-XXX%3A+Return+CompletableFuture+from+KafkaProducer.send
>
> I was still debating which was the best path forward (i.e. what should be
> in rejected alternatives versus the actual proposal). Feedback is welcome.
> You're also welcome to take over the KIP if you have the cycles and
> interest.
>
> Ismael
>
> On Tue, Jan 14, 2020 at 8:27 PM radai  wrote:
>
> > Hi
> >
> > With kip-118 (Drop Support for Java 7) officially done, is there a
> > timeline replacing usage of "plain" Futures with java 8
> > CompletableFutures?
> >
> > kafka 2.0 was mentioned at some point as a possible target for this ...
> >
>

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

[M. Manna]

+1 (Binding) - a long awaited KIP to have a simpler partition reassignment
script (without ZK)>

Kudos to you Colin :)

On Wed, 15 Jan 2020 at 10:10, Viktor Somogyi-Vass 
wrote:

> +1 (non-binding)
>
> Viktor
>
> On Wed, Jan 15, 2020 at 10:27 AM Manikumar 
> wrote:
>
> > +1 (binding)
> >
> > Thanks for the KIP.
> >
> > On Wed, Jan 15, 2020 at 2:48 AM Gwen Shapira  wrote:
> >
> > > +1 (binding, re-vote)
> > >
> > > On Tue, Jan 14, 2020 at 11:23 AM Colin McCabe 
> > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'm reposting this since I've been informed that gmail mashed the
> > > original VOTE thread into a different email thread.  Hopefully the
> > > different thread title will prevent it doing that in this case.
> > > >
> > > > I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper
> > access
> > > in Kafka Administrative Tools.
> > > >
> > > > KIP:  https://cwiki.apache.org/confluence/x/Wg6dC
> > > >
> > > > Discussion thread:
> > >
> >
> https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E
> > > >
> > > > cheers,
> > > > Colin
> > >
> >
>

Re: CompletableFuture?

[Ismael Juma]

Good question. I have a draft KIP for the producer change:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-XXX%3A+Return+CompletableFuture+from+KafkaProducer.send

I was still debating which was the best path forward (i.e. what should be
in rejected alternatives versus the actual proposal). Feedback is welcome.
You're also welcome to take over the KIP if you have the cycles and
interest.

Ismael

On Tue, Jan 14, 2020 at 8:27 PM radai  wrote:

> Hi
>
> With kip-118 (Drop Support for Java 7) officially done, is there a
> timeline replacing usage of "plain" Futures with java 8
> CompletableFutures?
>
> kafka 2.0 was mentioned at some point as a possible target for this ...
>

[jira] [Resolved] (KAFKA-9414) sink-task-metrics.sink-record-lag-max metric is not exposed

[Aidar Makhmutov (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aidar Makhmutov resolved KAFKA-9414.

Resolution: Not A Bug

It's not been implemented yet.

> sink-task-metrics.sink-record-lag-max metric is not exposed
> ---
>
> Key: KAFKA-9414
> URL: https://issues.apache.org/jira/browse/KAFKA-9414
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 2.2.0
>Reporter: Aidar Makhmutov
>Priority: Minor
>
> In group of metrics
> {{kafka.connect:type=sink-task-metrics,connector="\{connector}",task="\{task}"}}
> The following metric is not exposed by JMX (but present in documentation): 
> {{sink-record-lag-max}}
>  
> Details:
>  * Docker image: confluentinc/cp-kafka-connect:5.2.1
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] On the new KIP-158: Kafka Connect allows source connectors to set topic settings when creating new topics

[Gwen Shapira]

+1 (binding)
Looks super useful. Thank you.




On Mon, Jan 13, 2020 at 8:16 AM Konstantine Karantasis
 wrote:
>
> Hi everyone.
>
> I hope y'all had a nice break. The discussion on KIP-158 seems to have
> wrapped up since last year, so I'd like to open the vote on this KIP.
>
> A reminder that this is an updated KIP-158 (that had also been approved in
> its earlier version) and it seems to be a highly anticipated feature for
> many of us. I hope we can get this in for the upcoming release.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics
>
> Best,
> Konstantine

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Ron Dagostino]

Congratulations!

> On Jan 15, 2020, at 5:04 AM, Viktor Somogyi-Vass  
> wrote:
> 
> Congrats to you guys, it's a great accomplishment! :)
> 
>> On Wed, Jan 15, 2020 at 10:20 AM David Jacot  wrote:
>> 
>> Congrats!
>> 
>>> On Wed, Jan 15, 2020 at 12:00 AM James Cheng  wrote:
>>> 
>>> Congrats Colin, Vahid, and Manikumar!
>>> 
>>> -James
>>> 
 On Jan 14, 2020, at 10:59 AM, Tom Bentley  wrote:
 
 Congratulations!
 
 On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram <
>> rajinisiva...@gmail.com>
 wrote:
 
> Congratulations Colin, Vahid and Manikumar!
> 
> Regards,
> Rajini
> 
> On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison <
>>> mickael.mai...@gmail.com>
> wrote:
> 
>> Congrats Colin, Vahid and Manikumar!
>> 
>> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma 
>> wrote:
>>> 
>>> Congratulations Colin, Vahid and Manikumar!
>>> 
>>> Ismael
>>> 
>>> On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira 
> wrote:
>>> 
 Hi everyone,
 
 I'm happy to announce that Colin McCabe, Vahid Hashemian and
> Manikumar
 Reddy are now members of Apache Kafka PMC.
 
 Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
 2019. They all contributed many patches, code reviews and
> participated
 in many KIP discussions. We appreciate their contributions and are
 looking forward to many more to come.
 
 Congrats Colin, Vahid and Manikumar!
 
 Gwen, on behalf of Apache Kafka PMC
 
>> 
> 
>>> 
>>> 
>> 

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Ron Dagostino]

Hi Manikumar.  Yes, you are correct: if the ZK Security Migrator session 
authenticates to ZooKeeper with multiple identities — SASL and certificate — 
then ACLs are applied authorizing both the SASL principal and the certificate 
DN.

Ron

> On Jan 15, 2020, at 6:33 AM, Manikumar  wrote:
> 
> Hi Ron,
> 
> Thanks for the KIP. KIP looks good to me.
> 
> Am I correct in understanding that, when we run ZkSecurityMigrator with
> SASL + SSL, multiple identities will be added to the ACLs?
> 
> Thanks,
> 
> On Wed, Jan 15, 2020 at 1:19 AM Rajini Sivaram 
> wrote:
> 
>> Hi Ron,
>> 
>> Thanks for the detailed explanation, sounds good to me.
>> 
>> A few more questions:
>> 
>> 1) At the moment, all sensitive broker configs including
>> keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
>> starting up brokers. We are now adding SSL keystore/truststore passwords
>> for ZooKeeper client that cannot be stored in ZooKeeper since you need
>> these to connect to ZK. We should perhaps document that these configs can
>> be encrypted using KIP-421.
>> 
>> 2) We can dynamically update keystores and trust stores used by brokers
>> without restarting the broker. Can we support this easily for ZK clients
>> created by the broker, for example by adding our own
>> `zookeeper.ssl.context.supplier.class`?
>> 
>> 3) Looks like we are using config names that directly map to ZK configs.
>> Have we considered using equivalent Kafka config names with prefixes,
>> perhaps with inheritance from the non-prefixed names? Not sure if this is a
>> good idea, but perhaps worth documenting in Rejected Alternatives at least.
>> 
>> 
>>> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe  wrote:
>>> 
>>> Hi Ron,
>>> 
>>> Thanks for the explanation.  I guess thinking about it a little bit more,
>>> we should just add --zk-tls-config-file to all of these commands.
>>> 
>>> We will be removing this option (plus ZK in general) from these commands
>>> in the next major release, but ZK is still supported in 2.5, so we should
>>> just do the logical thing.  (The exception is ZkSecurityMigrator, which
>>> will stay).
>>> 
>>> best,
>>> Colin
>>> 
>>> 
 On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
 Hi Colin.
 
 <<< It seems like this [--zk-tls-config-file information] could just
>>> appear
 in a configuration file, which all of these tools already accept (I
>>> think)
 
 ZkSecurityMigrator has no such property file facility; adding a
 "--zk-tls-config-file" parameter is exactly for this purpose.  If we
>> add
 that to ZkSecurityMigrator then it is trivial to add it to other
>> commands
 (the same code is simply reused; it ends up being just a few extra
>>> lines).
 I do not see any parameter in the other two commands to adjust the ZK
 connection; ConfigCommand accepts a "--command-config" flag, but
>>> according
 to the code "This is used only with --bootstrap-server option for
 describing and altering broker configs."
 
 I do agree there would be no need to add "--zk-tls-config-file" to
 ReassignPartitionsCommand if its direct ZK connectivity is replaced in
>>> time
 for the next release.
 
 ConfigCommand supports the "--bootstrap-server" option and will have
>> its
 direct ZooKeeper access formally deprecated as per KIP-555, but the
>>> special
 use case of bootstrapping a ZooKeeper ensemble with encrypted
>> credentials
 prior to starting Kafka will still exist, so it feels like while
 "--zk-tls-config-file" would never be used except for this use case it
 could probably still be added for this particular situation.
 
 Ron
 
 P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
 sent) did not go through.  I am trying to get emails through now to
>> move
 this discussion forward.
 
 On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe 
>> wrote:
 
>> On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
>> Hi everyone.  I would like to make the following changes to the
>> KIP.
>> 
>> MOTIVATION:
>> Include a statement that it will be difficult in the short term to
>> deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> (which
>> invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
>>> cluster
>> with encrypted passwords in Zookeeper is an explicitly-supported
>> use
> case;
>> therefore it is in scope to be able to securely configure the CLI
>>> tools
>> that still leverage non-deprecated direct Zookeeper communication
>>> for TLS
>> (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
>> zookeeper-security-migration.sh).
> 
> Hi Ron,
> 
> Thanks for the KIP.
> 
> About deprecations:
> 
> * zookeeper-security-migration: clearly, deprecating ZK access in
>> this
>>> one
> would not make sense, since it would defeat the whole point of the
>>> tool :)
> 

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

[Habib Nahas]

Hi Sean,

Thats great, look forward to it.

Thanks,
Habib

On Tue, Jan 14, 2020, at 2:55 PM, Sean Glover wrote:
> Hi Habib,
> 
> Thank you for the reminder. I'll update the KIP this week and address the
> feedback from you and Gokul.
> 
> Regards,
> Sean
> 
> On Tue, Jan 14, 2020 at 9:06 AM Habib Nahas  wrote:
> 
> > Any chance of an update on the KIP? We are interested in seeing this move
> > forward.
> >
> > Thanks,
> > Habib
> > Sr SDE, AWS
> >
> > On Wed, Dec 18, 2019, at 3:27 PM, Habib Nahas wrote:
> > > Thanks Sean. Look forward to the updated KIP.
> > >
> > > Regards,
> > > Habib
> > >
> > > On Fri, Dec 13, 2019, at 6:22 AM, Sean Glover wrote:
> > > > Hi,
> > > >
> > > > After my last reply I had a nagging feeling something wasn't right,
> > and I
> > > > remembered that epoch time is UTC. This makes the discussion about
> > > > timezone irrelevant, since we're always using UTC. This makes the need
> > for
> > > > the LatencyTime interface that I proposed in the design irrelevant as
> > well,
> > > > since I can no longer think about how that might be useful. I'll update
> > > > the KIP. I'll also review KIP-32 to understand message timestamps
> > better
> > > > so I can explain the different types of latency results that could be
> > > > reported with this metric.
> > > >
> > > > Regards,
> > > > Sean
> > > >
> > > > On Thu, Dec 12, 2019 at 6:25 PM Sean Glover  > >
> > > > wrote:
> > > >
> > > > > Hi Habib,
> > > > >
> > > > > Thanks for question! If the consumer is in a different timezone than
> > the
> > > > > timezone used to produce messages to a partition then you can use an
> > > > > implementation of LatencyTime to return the current time of that
> > timezone.
> > > > > The current design assumes that messages produced to a partition
> > must all
> > > > > be produced from the same timezone. If timezone metadata were
> > encoded into
> > > > > each message then it would be possible to automatically determine the
> > > > > source timezone and calculate latency, however the current design
> > will not
> > > > > pass individual messages into LatencyTime to retrieve message
> > metadata.
> > > > > Instead, the LatencyTime.getWallClockTime method is only called once
> > per
> > > > > fetch request response per partition and then the metric is recorded
> > once
> > > > > the latency calculation is complete. This follows the same design as
> > the
> > > > > current consumer lag metric which calculates offset lag based on the
> > last
> > > > > message of the fetch request response for a partition. Since the
> > metric is
> > > > > just an aggregate (max/mean) over some time window we only need to
> > > > > occasionally calculate latency, which will have negligible impact on
> > the
> > > > > performance of consumer polling.
> > > > >
> > > > > A simple implementation of LatencyTime that returns wall clock time
> > for
> > > > > the Asia/Singapore timezone for all partitions:
> > > > >
> > > > > import java.time.*;
> > > > >
> > > > > class SingaporeTime implements LatencyTime {
> > > > > ZoneId zoneSingapore = ZoneId.of("Asia/Singapore");
> > > > > Clock clockSingapore = Clock.system(zoneSingapore);
> > > > >
> > > > > @Override
> > > > > public long getWallClockTime(TopicPartition tp) {
> > > > > return clockSingapore.instant.getEpochSecond();
> > > > > }
> > > > >
> > > > > ...
> > > > > }
> > > > >
> > > > > Regards,
> > > > > Sean
> > > > >
> > > > >
> > > > > On Thu, Dec 12, 2019 at 6:18 AM Habib Nahas  wrote:
> > > > >
> > > > >> Hi Sean,
> > > > >>
> > > > >> Thanks for the KIP.
> > > > >>
> > > > >> As I understand it users are free to set their own timestamp on
> > > > >> ProducerRecord. What is the recommendation for the proposed metric
> > in a
> > > > >> scenario where the user sets this timestamp in timezone A and
> > consumes the
> > > > >> record in timezone B. Its not clear to me if a custom
> > implementation of
> > > > >> LatencyTime will help here.
> > > > >>
> > > > >> Thanks,
> > > > >> Habib
> > > > >>
> > > > >> On Wed, Dec 11, 2019, at 4:52 PM, Sean Glover wrote:
> > > > >> > Hello again,
> > > > >> >
> > > > >> > There has been some interest in this KIP recently. I'm bumping the
> > > > >> thread
> > > > >> > to encourage feedback on the design.
> > > > >> >
> > > > >> > Regards,
> > > > >> > Sean
> > > > >> >
> > > > >> > On Mon, Jul 15, 2019 at 9:01 AM Sean Glover <
> > sean.glo...@lightbend.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > To hopefully spark some discussion I've copied the motivation
> > section
> > > > >> from
> > > > >> > > the KIP:
> > > > >> > >
> > > > >> > > Consumer lag is a useful metric to monitor how many records are
> > > > >> queued to
> > > > >> > > be processed. We can look at individual lag per partition or we
> > may
> > > > >> > > aggregate metrics. For example, we may want to monitor what the
> > > > >> maximum lag
> > > > >> > > of any particular partition in our consumer subscription so we
> > can
> > > > >> identify
> > > > >> > 

[DISCUSS] KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[David Jacot]

Hi all,

I just posted KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-559%3A+Make+the+Kafka+Protocol+Friendlier+with+L7+Proxies

Have a look and let me know what you think.

Best,
David

[jira] [Created] (KAFKA-9437) KIP-559: Make the Kafka Protocol Friendlier with L7 Proxies

[David Jacot (Jira)]

David Jacot created KAFKA-9437:
--

 Summary: KIP-559: Make the Kafka Protocol Friendlier with L7 
Proxies
 Key: KAFKA-9437
 URL: https://issues.apache.org/jira/browse/KAFKA-9437
 Project: Kafka
  Issue Type: Improvement
Reporter: David Jacot
Assignee: David Jacot






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-550: Mechanism to Delete Stray Partitions on Broker

[Dhruvil Shah]

Hi Colin,

We could add a configuration to disable stray partition deletion if needed,
but I wasn't sure if an operator would really want to disable it. Perhaps
if the implementation were buggy, the configuration could be used to
disable the feature until a bug fix is made. Is that the kind of use case
you were thinking of?

I was thinking that there would not be any delay between detection and
deletion of stray logs. We would schedule an async task to do the actual
deletion though.

Thanks,
Dhruvil

On Tue, Jan 14, 2020 at 11:04 PM Colin McCabe  wrote:

> Hi Dhruvil,
>
> Thanks for the KIP.  I think there should be some way to turn this off, in
> case that becomes necessary.  I'm also curious how long we intend to wait
> between detecting the duplication and  deleting the extra logs.  The KIP
> says "scheduled for deletion" but doesn't give a time frame -- is it
> assumed to be immediate?
>
> best,
> Colin
>
>
> On Tue, Jan 14, 2020, at 05:56, Dhruvil Shah wrote:
> > If there are no more questions or concerns, I will start a vote thread
> > tomorrow.
> >
> > Thanks,
> > Dhruvil
> >
> > On Mon, Jan 13, 2020 at 6:59 PM Dhruvil Shah 
> wrote:
> >
> > > Hi Nikhil,
> > >
> > > Thanks for looking at the KIP. The kind of race condition you mention
> is
> > > not possible as stray partition detection is done synchronously while
> > > handling the LeaderAndIsrRequest. In other words, we atomically
> evaluate
> > > the partitions the broker must host and the extra partitions it is
> hosting
> > > and schedule deletions based on that.
> > >
> > > One possible shortcoming of the KIP is that we do not have the ability
> to
> > > detect a stray partition if the topic has been recreated since. We will
> > > have the ability to disambiguate between different generations of a
> > > partition with KIP-516.
> > >
> > > Thanks,
> > > Dhruvil
> > >
> > > On Sat, Jan 11, 2020 at 11:40 AM Nikhil Bhatia 
> > > wrote:
> > >
> > >> Thanks Dhruvil, the proposal looks reasonable to me.
> > >>
> > >> is there a potential of a race between a new topic being assigned to
> the
> > >> same node that is still performing a cleanup of the stray partition ?
> > >> Topic
> > >> ID will definitely solve this issue.
> > >>
> > >> Thanks
> > >> Nikhil
> > >>
> > >> On 2020/01/06 04:30:20, Dhruvil Shah  wrote:
> > >> > Here is the link to the KIP:>
> > >> >
> > >>
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-550%3A+Mechanism+to+Delete+Stray+Partitions+on+Broker
> > >> >
> > >>
> > >> >
> > >> > On Mon, Jan 6, 2020 at 9:59 AM Dhruvil Shah 
> > >> wrote:>
> > >> >
> > >> > > Hi all, I would like to kick off discussion for KIP-550 which
> proposes
> > >> a>
> > >> > > mechanism to detect and delete stray partitions on a broker.
> > >> Suggestions>
> > >> > > and feedback are welcome.>
> > >> > >>
> > >> > > - Dhruvil>
> > >> > >>
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication

[Manikumar]

Hi Ron,

Thanks for the KIP. KIP looks good to me.

Am I correct in understanding that, when we run ZkSecurityMigrator with
SASL + SSL, multiple identities will be added to the ACLs?

Thanks,

On Wed, Jan 15, 2020 at 1:19 AM Rajini Sivaram 
wrote:

> Hi Ron,
>
> Thanks for the detailed explanation, sounds good to me.
>
> A few more questions:
>
> 1) At the moment, all sensitive broker configs including
> keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> starting up brokers. We are now adding SSL keystore/truststore passwords
> for ZooKeeper client that cannot be stored in ZooKeeper since you need
> these to connect to ZK. We should perhaps document that these configs can
> be encrypted using KIP-421.
>
> 2) We can dynamically update keystores and trust stores used by brokers
> without restarting the broker. Can we support this easily for ZK clients
> created by the broker, for example by adding our own
> `zookeeper.ssl.context.supplier.class`?
>
> 3) Looks like we are using config names that directly map to ZK configs.
> Have we considered using equivalent Kafka config names with prefixes,
> perhaps with inheritance from the non-prefixed names? Not sure if this is a
> good idea, but perhaps worth documenting in Rejected Alternatives at least.
>
>
> On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe  wrote:
>
> > Hi Ron,
> >
> > Thanks for the explanation.  I guess thinking about it a little bit more,
> > we should just add --zk-tls-config-file to all of these commands.
> >
> > We will be removing this option (plus ZK in general) from these commands
> > in the next major release, but ZK is still supported in 2.5, so we should
> > just do the logical thing.  (The exception is ZkSecurityMigrator, which
> > will stay).
> >
> > best,
> > Colin
> >
> >
> > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > Hi Colin.
> > >
> > > <<< It seems like this [--zk-tls-config-file information] could just
> > appear
> > > in a configuration file, which all of these tools already accept (I
> > think)
> > >
> > > ZkSecurityMigrator has no such property file facility; adding a
> > > "--zk-tls-config-file" parameter is exactly for this purpose.  If we
> add
> > > that to ZkSecurityMigrator then it is trivial to add it to other
> commands
> > > (the same code is simply reused; it ends up being just a few extra
> > lines).
> > > I do not see any parameter in the other two commands to adjust the ZK
> > > connection; ConfigCommand accepts a "--command-config" flag, but
> > according
> > > to the code "This is used only with --bootstrap-server option for
> > > describing and altering broker configs."
> > >
> > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > ReassignPartitionsCommand if its direct ZK connectivity is replaced in
> > time
> > > for the next release.
> > >
> > > ConfigCommand supports the "--bootstrap-server" option and will have
> its
> > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > special
> > > use case of bootstrapping a ZooKeeper ensemble with encrypted
> credentials
> > > prior to starting Kafka will still exist, so it feels like while
> > > "--zk-tls-config-file" would never be used except for this use case it
> > > could probably still be added for this particular situation.
> > >
> > > Ron
> > >
> > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3 more I
> > > sent) did not go through.  I am trying to get emails through now to
> move
> > > this discussion forward.
> > >
> > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe 
> wrote:
> > >
> > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > Hi everyone.  I would like to make the following changes to the
> KIP.
> > > > >
> > > > > MOTIVATION:
> > > > > Include a statement that it will be difficult in the short term to
> > > > > deprecate direct Zookeeper communication in kafka-configs.{sh, bat}
> > > > (which
> > > > > invoke kafka.admin.ConfigCommand) because bootstrapping a Kafka
> > cluster
> > > > > with encrypted passwords in Zookeeper is an explicitly-supported
> use
> > > > case;
> > > > > therefore it is in scope to be able to securely configure the CLI
> > tools
> > > > > that still leverage non-deprecated direct Zookeeper communication
> > for TLS
> > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat} and
> > > > > zookeeper-security-migration.sh).
> > > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the KIP.
> > > >
> > > > About deprecations:
> > > >
> > > > * zookeeper-security-migration: clearly, deprecating ZK access in
> this
> > one
> > > > would not make sense, since it would defeat the whole point of the
> > tool :)
> > > >
> > > > * kafka-reassign-partitions: ZK access should be deprecated here.
> > KIP-455
> > > > implementation has dragged a bit, but this should get done soon.
> > Certainly
> > > > before the next release.
> > > >
> > > > * kafka-configs: I think ZK access should be deprecated here as well.
> 

[jira] [Created] (KAFKA-9436) New Kafka Connect SMT for plainText => Struct(or Map)

[whsoul (Jira)]

whsoul created KAFKA-9436:
-

 Summary: New Kafka Connect SMT for plainText => Struct(or Map)
 Key: KAFKA-9436
 URL: https://issues.apache.org/jira/browse/KAFKA-9436
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: whsoul


I'd like to parse and convert plain text rows to struct(or map) data, and load 
into documented database such as mongoDB, elasticSearch, etc... with SMT

 

For example

 

plain text apache log
{code:java}
"111.61.73.113 - - [08/Aug/2019:18:15:29 +0900] \"OPTIONS 
/api/v1/service_config HTTP/1.1\" 200 - 101989 \"http://local.test.com/\; 
\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/75.0.3770.142 Safari/537.36\""
{code}
SMT connect config with regular expression below can easily transform a plain 
text to struct (or map) data.

 
{code:java}
"transforms": "TimestampTopic, RegexTransform",
"transforms.RegexTransform.type": 
"org.apache.kafka.connect.transforms.ToStructByRegexTransform$Value",

"transforms.RegexTransform.struct.field": "message",
"transforms.RegexTransform.regex": "^([\\d.]+) (\\S+) (\\S+) 
\\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(GET|POST|OPTIONS|HEAD|PUT|DELETE|PATCH) 
(.+?) (.+?)\" (\\d{3}) ([0-9|-]+) ([0-9|-]+) \"([^\"]+)\" \"([^\"]+)\""

"transforms.RegexTransform.mapping": 
"IP,RemoteUser,AuthedRemoteUser,DateTime,Method,Request,Protocol,Response,BytesSent,Ms:NUMBER,Referrer,UserAgent"
{code}
 

I have PR about this



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-518: Allow listing consumer groups per state

[Manikumar]

Hi Mickael,

Thanks for the KIP.  Can you respond to the comments from David on discuss
thread?

Thanks,

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[Viktor Somogyi-Vass]

Congrats to you guys, it's a great accomplishment! :)

On Wed, Jan 15, 2020 at 10:20 AM David Jacot  wrote:

> Congrats!
>
> On Wed, Jan 15, 2020 at 12:00 AM James Cheng  wrote:
>
> > Congrats Colin, Vahid, and Manikumar!
> >
> > -James
> >
> > > On Jan 14, 2020, at 10:59 AM, Tom Bentley  wrote:
> > >
> > > Congratulations!
> > >
> > > On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram <
> rajinisiva...@gmail.com>
> > > wrote:
> > >
> > >> Congratulations Colin, Vahid and Manikumar!
> > >>
> > >> Regards,
> > >> Rajini
> > >>
> > >> On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > >> wrote:
> > >>
> > >>> Congrats Colin, Vahid and Manikumar!
> > >>>
> > >>> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma 
> wrote:
> > 
> >  Congratulations Colin, Vahid and Manikumar!
> > 
> >  Ismael
> > 
> >  On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira 
> > >> wrote:
> > 
> > > Hi everyone,
> > >
> > > I'm happy to announce that Colin McCabe, Vahid Hashemian and
> > >> Manikumar
> > > Reddy are now members of Apache Kafka PMC.
> > >
> > > Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
> > > 2019. They all contributed many patches, code reviews and
> > >> participated
> > > in many KIP discussions. We appreciate their contributions and are
> > > looking forward to many more to come.
> > >
> > > Congrats Colin, Vahid and Manikumar!
> > >
> > > Gwen, on behalf of Apache Kafka PMC
> > >
> > >>>
> > >>
> >
> >
>

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

[Viktor Somogyi-Vass]

Hey folks, bumping this again as KIP freeze is nearing and I hope to get
this into the next release.
We need only one binding vote.

Thanks,
Viktor

On Thu, Jan 9, 2020 at 1:56 PM Viktor Somogyi-Vass 
wrote:

> Bumping this in the hope of a vote or additional feedback.
>
> Viktor
>
> On Tue, Dec 3, 2019 at 1:07 PM Viktor Somogyi-Vass <
> viktorsomo...@gmail.com> wrote:
>
>> Hi Folks,
>>
>> I'd like to bump this once more in the hope of a binding vote or any
>> additional feedback.
>>
>> Thanks,
>> Viktor
>>
>> On Fri, Oct 25, 2019 at 2:24 PM Viktor Somogyi-Vass <
>> viktorsomo...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Would like to bump this in the hope of one binding vote (or any
>>> additional feedback).
>>>
>>> Thanks,
>>> Viktor
>>>
>>> On Wed, Sep 18, 2019 at 5:25 PM Viktor Somogyi-Vass <
>>> viktorsomo...@gmail.com> wrote:
>>>
 Hi All,

 Harsha, Ryanne: thanks for the vote!

 I'd like to bump this again as today is the KIP freeze date and there
 is still one binding vote needed which I'm hoping to get in order to have
 this included in 2.4.

 Thanks,
 Viktor

 On Tue, Sep 17, 2019 at 1:18 AM Ryanne Dolan 
 wrote:

> +1 non-binding
>
> Ryanne
>
> On Mon, Sep 16, 2019, 5:11 PM Harsha Ch  wrote:
>
> > +1 (binding). Thanks for the KIP Viktor
> >
> > Thanks,
> >
> > Harsha
> >
> > On Mon, Sep 16, 2019 at 3:02 AM, Viktor Somogyi-Vass <
> > viktorsomo...@gmail.com > wrote:
> >
> > >
> > >
> > >
> > > Hi All,
> > >
> > >
> > >
> > > I'd like to bump this again in order to get some more binding votes
> > and/or
> > > feedback in the hope we can push this in for 2.4.
> > >
> > >
> > >
> > > Thank you Manikumar, Gabor and Ryanne so far for the votes! (the
> last two
> > > were on the discussion thread after starting the vote but I think
> it
> > still
> > > counts :) )
> > >
> > >
> > >
> > > Thanks,
> > > Viktor
> > >
> > >
> > >
> > > On Wed, Aug 21, 2019 at 1:44 PM Manikumar < manikumar. reddy@
> gmail.
> > com (
> > > manikumar.re...@gmail.com ) > wrote:
> > >
> > >
> > >>
> > >>
> > >> Hi,
> > >>
> > >>
> > >>
> > >> +1 (binding).
> > >>
> > >>
> > >>
> > >> Thanks for the updated KIP. LGTM.
> > >>
> > >>
> > >>
> > >> Thanks,
> > >> Manikumar
> > >>
> > >>
> > >>
> > >> On Tue, Aug 6, 2019 at 3:14 PM Viktor Somogyi-Vass <
> viktorsomogyi@
> > gmail.
> > >> com ( viktorsomo...@gmail.com ) >
> > >> wrote:
> > >>
> > >>
> > >>>
> > >>>
> > >>> Hi All,
> > >>>
> > >>>
> > >>>
> > >>> Bumping this, I'd be happy to get some additional feedback and/or
> > votes.
> > >>>
> > >>>
> > >>>
> > >>> Thanks,
> > >>> Viktor
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vass <
> viktorsomogyi@
> > gmail.
> > >>> com ( viktorsomo...@gmail.com ) > wrote:
> > >>>
> > >>>
> > 
> > 
> >  Hi All,
> > 
> > 
> > 
> >  I'd like to start a vote on this KIP.
> > 
> > 
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >> https:/ / cwiki. apache. org/ confluence/ display/ KAFKA/
> > KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
> > >> (
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users
> > >> )
> > >>
> > >>
> > >>>
> > 
> > 
> >  To summarize it: the proposed feature would allow users (usually
> >  superusers) to create delegation tokens for other users. This is
> > 
> > 
> > >>>
> > >>>
> > >>>
> > >>> especially
> > >>>
> > >>>
> > 
> > 
> >  helpful in Spark where the delegation token created this way
> can be
> >  distributed to workers.
> > 
> > 
> > 
> >  I'd be happy to receive any votes or additional feedback.
> > 
> > 
> > 
> >  Viktor
> > 
> > 
> > >>>
> > >>>
> > >>
> > >>
> > >
> > >
> > >
>

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

[Mickael Maison]

Thanks Colin
+1 (binding)

On Wed, Jan 15, 2020 at 9:27 AM Manikumar  wrote:
>
> +1 (binding)
>
> Thanks for the KIP.
>
> On Wed, Jan 15, 2020 at 2:48 AM Gwen Shapira  wrote:
>
> > +1 (binding, re-vote)
> >
> > On Tue, Jan 14, 2020 at 11:23 AM Colin McCabe  wrote:
> > >
> > > Hi all,
> > >
> > > I'm reposting this since I've been informed that gmail mashed the
> > original VOTE thread into a different email thread.  Hopefully the
> > different thread title will prevent it doing that in this case.
> > >
> > > I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access
> > in Kafka Administrative Tools.
> > >
> > > KIP:  https://cwiki.apache.org/confluence/x/Wg6dC
> > >
> > > Discussion thread:
> > https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E
> > >
> > > cheers,
> > > Colin
> >

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

[Viktor Somogyi-Vass]

+1 (non-binding)

Viktor

On Wed, Jan 15, 2020 at 10:27 AM Manikumar 
wrote:

> +1 (binding)
>
> Thanks for the KIP.
>
> On Wed, Jan 15, 2020 at 2:48 AM Gwen Shapira  wrote:
>
> > +1 (binding, re-vote)
> >
> > On Tue, Jan 14, 2020 at 11:23 AM Colin McCabe 
> wrote:
> > >
> > > Hi all,
> > >
> > > I'm reposting this since I've been informed that gmail mashed the
> > original VOTE thread into a different email thread.  Hopefully the
> > different thread title will prevent it doing that in this case.
> > >
> > > I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper
> access
> > in Kafka Administrative Tools.
> > >
> > > KIP:  https://cwiki.apache.org/confluence/x/Wg6dC
> > >
> > > Discussion thread:
> >
> https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E
> > >
> > > cheers,
> > > Colin
> >
>

[jira] [Created] (KAFKA-9435) Replace DescribeLogDirs request/response with automated protocol

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-9435:
--

 Summary: Replace DescribeLogDirs request/response with automated 
protocol
 Key: KAFKA-9435
 URL: https://issues.apache.org/jira/browse/KAFKA-9435
 Project: Kafka
  Issue Type: Sub-task
Reporter: Tom Bentley
Assignee: Tom Bentley






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9434) Replace AlterReplicaLogDirs request/response with automated protocol

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-9434:
--

 Summary: Replace AlterReplicaLogDirs request/response with 
automated protocol
 Key: KAFKA-9434
 URL: https://issues.apache.org/jira/browse/KAFKA-9434
 Project: Kafka
  Issue Type: Sub-task
Reporter: Tom Bentley
Assignee: Tom Bentley






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9432) Replace DescribeConfigs request/response with automated protocol

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-9432:
--

 Summary: Replace DescribeConfigs request/response with automated 
protocol
 Key: KAFKA-9432
 URL: https://issues.apache.org/jira/browse/KAFKA-9432
 Project: Kafka
  Issue Type: Sub-task
Reporter: Tom Bentley
Assignee: Tom Bentley






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9433) Replace AlterConfigs request/response with automated protocol

[Tom Bentley (Jira)]

Tom Bentley created KAFKA-9433:
--

 Summary: Replace AlterConfigs request/response with automated 
protocol
 Key: KAFKA-9433
 URL: https://issues.apache.org/jira/browse/KAFKA-9433
 Project: Kafka
  Issue Type: Sub-task
Reporter: Tom Bentley
Assignee: Tom Bentley






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-551: Expose disk read and write metrics

[Mickael Maison]

+1 (binding)
Thanks for the KIP

On Tue, Jan 14, 2020 at 6:50 PM David Arthur  wrote:
>
> +1 binding
>
> This will be very nice to have. Thanks for the KIP, Colin.
>
> -David
>
> On Tue, Jan 14, 2020 at 11:39 AM Sönke Liebau
>  wrote:
>
> > +1 (non-binding)
> >
> > Thanks for creating this!
> >
> > On Tue, 14 Jan 2020 at 17:36, Mitchell  wrote:
> >
> > > +1 (non-binding)!
> > > Very useful kip.
> > > -mitch
> > >
> > > On Tue, Jan 14, 2020 at 10:26 AM Manikumar 
> > > wrote:
> > > >
> > > > +1 (binding).
> > > >
> > > > Thanks for the KIP.
> > > >
> > > >
> > > >
> > > > On Sun, Jan 12, 2020 at 1:23 AM Lucas Bradstreet 
> > > wrote:
> > > >
> > > > > +1 (non binding)
> > > > >
> > > > > On Sat, 11 Jan 2020 at 02:32, M. Manna  wrote:
> > > > >
> > > > > > Hey Colin,
> > > > > >
> > > > > > +1 - Really useful for folks managing a cluster by themselves.
> > > > > >
> > > > > > M. MAnna
> > > > > >
> > > > > > On Fri, 10 Jan 2020 at 22:35, Jose Garcia Sancio <
> > > jsan...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > +1, LGTM.
> > > > > > >
> > > > > > > On Fri, Jan 10, 2020 at 2:19 PM Gwen Shapira 
> > > > > wrote:
> > > > > > >
> > > > > > > > +1, thanks for driving this
> > > > > > > >
> > > > > > > > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe <
> > cmcc...@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I'd like to start the vote on KIP-551: Expose disk read and
> > > write
> > > > > > > > metrics.
> > > > > > > > >
> > > > > > > > > KIP:  https://cwiki.apache.org/confluence/x/sotSC
> > > > > > > > >
> > > > > > > > > Discussion thread:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://lists.apache.org/thread.html/cfaac4426455406abe890464a7f4ae23a5c69a39afde66fe6eb3d696%40%3Cdev.kafka.apache.org%3E
> > > > > > > > >
> > > > > > > > > cheers,
> > > > > > > > > Colin
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > -Jose
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> >
> > --
> > Sönke Liebau
> > Partner
> > Tel. +49 179 7940878
> > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
> >
>
>
> --
> David Arthur

Re: [VOTE] KIP-555: An admin tools proposal to accomplish the deprecation of zookeeper access that is direct

[Manikumar]

+1 (binding)

Thanks for the KIP.

On Wed, Jan 15, 2020 at 2:48 AM Gwen Shapira  wrote:

> +1 (binding, re-vote)
>
> On Tue, Jan 14, 2020 at 11:23 AM Colin McCabe  wrote:
> >
> > Hi all,
> >
> > I'm reposting this since I've been informed that gmail mashed the
> original VOTE thread into a different email thread.  Hopefully the
> different thread title will prevent it doing that in this case.
> >
> > I'd like to start the vote on KIP-555: Deprecate Direct Zookeeper access
> in Kafka Administrative Tools.
> >
> > KIP:  https://cwiki.apache.org/confluence/x/Wg6dC
> >
> > Discussion thread:
> https://lists.apache.org/thread.html/ra0e4338c596d037c406b52a719bf13f775b03797cd5ca8d03d7f71c4%40%3Cdev.kafka.apache.org%3E
> >
> > cheers,
> > Colin
>

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

[David Jacot]

Congrats!

On Wed, Jan 15, 2020 at 12:00 AM James Cheng  wrote:

> Congrats Colin, Vahid, and Manikumar!
>
> -James
>
> > On Jan 14, 2020, at 10:59 AM, Tom Bentley  wrote:
> >
> > Congratulations!
> >
> > On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram 
> > wrote:
> >
> >> Congratulations Colin, Vahid and Manikumar!
> >>
> >> Regards,
> >> Rajini
> >>
> >> On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison <
> mickael.mai...@gmail.com>
> >> wrote:
> >>
> >>> Congrats Colin, Vahid and Manikumar!
> >>>
> >>> On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma  wrote:
> 
>  Congratulations Colin, Vahid and Manikumar!
> 
>  Ismael
> 
>  On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira 
> >> wrote:
> 
> > Hi everyone,
> >
> > I'm happy to announce that Colin McCabe, Vahid Hashemian and
> >> Manikumar
> > Reddy are now members of Apache Kafka PMC.
> >
> > Colin and Manikumar became committers on Sept 2018 and Vahid on Jan
> > 2019. They all contributed many patches, code reviews and
> >> participated
> > in many KIP discussions. We appreciate their contributions and are
> > looking forward to many more to come.
> >
> > Congrats Colin, Vahid and Manikumar!
> >
> > Gwen, on behalf of Apache Kafka PMC
> >
> >>>
> >>
>
>

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

[Tom Bentley]

Hi Konstantine,

Thanks for the KIP, I can see how it could be useful.

a) Did you consider using a metric for this? I don't think it would satisfy
all the use cases you have in mind, but you could mention it in the
rejected alternatives.

b) If the topic name contains the string "-connector" then the key format
is ambiguous. This isn't necessarily fatal because the value will
disambiguate, but it could be misleading. Any reason not to just use a JSON
key, and simplify the value?

c) I didn't understand this part: "As soon as a worker detects the addition
of a topic to a connector's set of active topics, the worker will cease to
post update messages to the status.storage.topic for that connector. ". I'm
sure I've overlooking something but why is this necessary? Is this were the
task id in the value is used?

Thanks again,

Tom

On Wed, Jan 15, 2020 at 12:15 AM Randall Hauch  wrote:

> Oh, one more thing:
>
> 9. There's no mention of how the status topic is partitioned, or how
> partitioning will be used by the new topic records. The KIP should probably
> outline this for clarity and completeness.
>
> Best regards,
>
> Randall
>
> On Tue, Jan 14, 2020 at 5:25 PM Randall Hauch  wrote:
>
> > Thanks, Konstantine. Overall, this KIP looks interesting and really
> > useful, and for the most part is spot on. I do have a number of
> > questions/comments about specifics:
> >
> >1. The topic records have a value that includes the connector name,
> >task number that last reported the topic is used, and the topic name.
> >There's no mention of record timestamps, but I wonder if it'd be
> useful to
> >record this. One challenge might be that a connector does not write
> to a
> >topic for a while or the task remains running for long periods of
> time and
> >therefore the worker doesn't record that this topic has been newly
> written
> >to since it the task was restarted. IOW, the semantics of the
> timestamp may
> >be a bit murky. Have you thought about recording the timestamp, and
> if so
> >what are the pros and cons?
> >- The "Recording active topics" section says the following:
> >   "As soon as a worker detects the addition of a topic to a
> >   connector's set of active topics, all the connector's tasks that
> inspect
> >   source or sink records will cease to post update messages to the
> >   status.storage.topic."
> >   This probably means the timestamp won't be very useful.
> >2. The KIP says "the Kafka record value stores the ID of the task that
> >succeeded to store a topic status record last." However, this is a bit
> >unclear: is it really storing the last task that successfully wrote
> to that
> >topic (as this would require very frequent writes to this topic), or
> is it
> >more that this is the task that was last *recorded* as having written
> >to the topic? (Here, "recorded" could be a bit of a gray area, since
> this
> >would depend on the how the worker periodically records this
> information.)
> >Any kind of clarity here might be helpful.
> >3. In the "Recording active topics" section (and the surrounding
> >sections), the "task" is used ambiguously. For example, "when its
> tasks
> >start processing their first records ... these tasks will start
> inspecting
> >which is the Kafka topic of each of these records". IIUC, the first
> "task"
> >mentioned is the connector's task, and the second is the worker's
> task. Do
> >we need to distinguish this more clearly?
> >4. Maybe I missed it, but does this KIP explicitly say that the
> >Connector API is unchanged? It's probably worth pointing out to help
> >assuage any concerns that connector implementations have to change to
> make
> >use of this feature.
> >5. In the "Resetting a connector's set of active topics" section the
> >behavior is not exactly clear. Consider a user running connector "A",
> the
> >connector has been fully started and is processing records, and the
> worker
> >has recorded topic usage records. Then the user resets the active
> topics
> >for connector A while the connector is still running? If the connector
> >writes to no new topics, before the tasks are rebalanced then is it
> correct
> >that Connect would report no active topics? And after the tasks are
> >rebalance, will the worker record any topics used by connector A?
> >6. In the "Restaring" (misspelled) section: "Reconfiguring a source
> >connector has also no altering effect for a source connector.
> However, when
> >reconfiguring a sink connector if the new configuration no longer
> includes
> >any of the previously tracked topics, these topics will be removed
> from the
> >set of active topics for this sink connector by appending tombstone
> >messages appropriately after the reconfiguration of the connector."
> Would
> >it be better to not automatically reset connector's active topics