Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

[Ismael Juma]

It got 3 binding votes already and the deadline is only tomorrow. :)

Ismael

On Mon, 21 May 2018, 21:50 Colin McCabe,  wrote:

> On Mon, May 21, 2018, at 04:53, Andy Coates wrote:
> > Hey Piyush,
> >
> > Thanks for the updated KIP! Couple of minor points from me:
> >
> > When storing wildcard-suffixed Acls in ZK, drop the asterisk of the end
> for
> > the path, e.g. change "*/kafka-wildcard-acl/Topic/teamA*" * to "*/*
> > *kafka-wildcard-acl**/Topic/**teamA"*. This reduces error conditions,
> i.e.
> > this is a place for storing wildcard-suffixed Acls, so it implicitly ends
> > in an asterisk. If you include the asterisk in the path then you need to
> > validate each entry, when reading, ends with an asterisk, and do
> something
> > if they don't. If you don't include the training asterisk then there is
> > nothing to validate and you can treat the prefix as a literal, (i.e. no
> > escaping needed).  TBH I'd probably drop the asterisk from the in-memory
> > representation as well, for the same reason.
>
> Hi Andy,
>
> I agree.  If everything in ZK under /kafka-wildcard-acl/ is a prefix ACL,
> there is no need to include the star at the end.  And really, it should be
> called something like /kafka-prefix-acl/, since it's only vaguely related
> to the idea of wildcards.
>
> >
> > Rather than creating an enum to indicate the type of a resource, you
> could
> > instead use polymorphism, e.g. make Resource an interface and have two
> > implementations: LiteralResource and WildcardSuffixedResource.  This is
> > also extensible, but may also allow for a cleaner implementation.
>
> Since Resource is a concrete class now, we can't make it an interface
> without breaking API compatibility.
>
> Even if it were possible to do compatibly, I would argue it's a bad idea.
> If we need to add another bit of state like case insensitivity, we don't
> want to have LiteralCaseInsensistiveResource,
> WildcardSuffixedCaseInsensitiveResource, etc. etc.  You need 2^n subclasses
> classes to represent N bits of state.
>
> I would argue that there should be a field in Resource like NameType which
> can be LITERAL or PREFIX.  That leaves us in a good position when someone
> inevitably comes up with a new NameType.
>
> Does this still have a chance to get in, or has the KIP window closed?  I
> would argue with one or two minor changes it's ready to go.  Pretty much
> all of the compatibility problems are solved with the separate ZK hierarchy.
>
> best,
> Colin
>
> >
> > Andy
> >
> > On 21 May 2018 at 01:58, Rajini Sivaram  wrote:
> >
> > > Hi Piyush, Thanks for the KIP!
> > >
> > > +1 (binding)
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Sun, May 20, 2018 at 2:53 PM, Andy Coates 
> wrote:
> > >
> > > > Awesome last minute effort Piyush.
> > > >
> > > > Really appreciate your time and input,
> > > >
> > > > Andy
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 19 May 2018, at 03:43, Piyush Vijay 
> wrote:
> > > > >
> > > > > Updated the KIP.
> > > > >
> > > > > 1. New enum field 'ResourceNameType' in Resource and ResourceFilter
> > > > classes.
> > > > > 2. modify getAcls() and rely on ResourceNameType' field in
> Resource to
> > > > > return either exact matches or all matches based on
> wildcard-suffix.
> > > > > 3. CLI changes to identify if resource name is literal or
> > > wildcard-suffix
> > > > > 4. Escaping doesn't work and isn't required if we're keeping a
> separate
> > > > > path on ZK (kafka-wildcard-acl) to store wildcard-suffix ACLs.
> > > > > 5. New API keys for Create / Delete / Describe Acls request with a
> new
> > > > > field in schemas for 'ResourceNameType'.
> > > > >
> > > > > Looks ready to me for the vote, will start voting thread now.
> Thanks
> > > > > everyone for the valuable feedback.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Piyush Vijay
> > > > >
> > > > >
> > > > > Piyush Vijay
> > > > >
> > > > >> On Fri, May 18, 2018 at 6:07 PM, Andy Coates 
> > > wrote:
> > > > >>
> > > > >> Hi Piyush,
> > > > >>
> > > > >> We're fast approaching the KIP deadline. Are you actively working
> on
> > > > this?
> > > > >> If you're not I can take over.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Andy
> > > > >>
> > > > >>> On 18 May 2018 at 14:25, Andy Coates  wrote:
> > > > >>>
> > > > >>> OK I've read it now.
> > > > >>>
> > > > >>> 1. I see you have an example:
> > > >  For example: If I want to fetch all ACLs that match ’topicA*’,
> it’s
> > > > not
> > > > >>> possible without introducing new API AND maintaining backwards
> > > > >>> compatibility.
> > > > >>> getAcls takes a Resource, right, which would be either a full
> > > resource
> > > > >>> name or 'ALL', i.e. '*', right?  The point of the call is to get
> all
> > > > ACLs
> > > > >>> relating to a specific resource, not a partial resource like
> > > 'topicA*'.
> > > > >>> Currently, I'm guessing 

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

[Colin McCabe]

On Mon, May 21, 2018, at 04:53, Andy Coates wrote:
> Hey Piyush,
> 
> Thanks for the updated KIP! Couple of minor points from me:
> 
> When storing wildcard-suffixed Acls in ZK, drop the asterisk of the end for
> the path, e.g. change "*/kafka-wildcard-acl/Topic/teamA*" * to "*/*
> *kafka-wildcard-acl**/Topic/**teamA"*. This reduces error conditions, i.e.
> this is a place for storing wildcard-suffixed Acls, so it implicitly ends
> in an asterisk. If you include the asterisk in the path then you need to
> validate each entry, when reading, ends with an asterisk, and do something
> if they don't. If you don't include the training asterisk then there is
> nothing to validate and you can treat the prefix as a literal, (i.e. no
> escaping needed).  TBH I'd probably drop the asterisk from the in-memory
> representation as well, for the same reason.

Hi Andy,

I agree.  If everything in ZK under /kafka-wildcard-acl/ is a prefix ACL, there 
is no need to include the star at the end.  And really, it should be called 
something like /kafka-prefix-acl/, since it's only vaguely related to the idea 
of wildcards.

> 
> Rather than creating an enum to indicate the type of a resource, you could
> instead use polymorphism, e.g. make Resource an interface and have two
> implementations: LiteralResource and WildcardSuffixedResource.  This is
> also extensible, but may also allow for a cleaner implementation.

Since Resource is a concrete class now, we can't make it an interface without 
breaking API compatibility.

Even if it were possible to do compatibly, I would argue it's a bad idea.  If 
we need to add another bit of state like case insensitivity, we don't want to 
have LiteralCaseInsensistiveResource, WildcardSuffixedCaseInsensitiveResource, 
etc. etc.  You need 2^n subclasses classes to represent N bits of state.

I would argue that there should be a field in Resource like NameType which can 
be LITERAL or PREFIX.  That leaves us in a good position when someone 
inevitably comes up with a new NameType.

Does this still have a chance to get in, or has the KIP window closed?  I would 
argue with one or two minor changes it's ready to go.  Pretty much all of the 
compatibility problems are solved with the separate ZK hierarchy.

best,
Colin

> 
> Andy
> 
> On 21 May 2018 at 01:58, Rajini Sivaram  wrote:
> 
> > Hi Piyush, Thanks for the KIP!
> >
> > +1 (binding)
> >
> > Regards,
> >
> > Rajini
> >
> > On Sun, May 20, 2018 at 2:53 PM, Andy Coates  wrote:
> >
> > > Awesome last minute effort Piyush.
> > >
> > > Really appreciate your time and input,
> > >
> > > Andy
> > >
> > > Sent from my iPhone
> > >
> > > > On 19 May 2018, at 03:43, Piyush Vijay  wrote:
> > > >
> > > > Updated the KIP.
> > > >
> > > > 1. New enum field 'ResourceNameType' in Resource and ResourceFilter
> > > classes.
> > > > 2. modify getAcls() and rely on ResourceNameType' field in Resource to
> > > > return either exact matches or all matches based on wildcard-suffix.
> > > > 3. CLI changes to identify if resource name is literal or
> > wildcard-suffix
> > > > 4. Escaping doesn't work and isn't required if we're keeping a separate
> > > > path on ZK (kafka-wildcard-acl) to store wildcard-suffix ACLs.
> > > > 5. New API keys for Create / Delete / Describe Acls request with a new
> > > > field in schemas for 'ResourceNameType'.
> > > >
> > > > Looks ready to me for the vote, will start voting thread now. Thanks
> > > > everyone for the valuable feedback.
> > > >
> > > >
> > > >
> > > >
> > > > Piyush Vijay
> > > >
> > > >
> > > > Piyush Vijay
> > > >
> > > >> On Fri, May 18, 2018 at 6:07 PM, Andy Coates 
> > wrote:
> > > >>
> > > >> Hi Piyush,
> > > >>
> > > >> We're fast approaching the KIP deadline. Are you actively working on
> > > this?
> > > >> If you're not I can take over.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Andy
> > > >>
> > > >>> On 18 May 2018 at 14:25, Andy Coates  wrote:
> > > >>>
> > > >>> OK I've read it now.
> > > >>>
> > > >>> 1. I see you have an example:
> > >  For example: If I want to fetch all ACLs that match ’topicA*’, it’s
> > > not
> > > >>> possible without introducing new API AND maintaining backwards
> > > >>> compatibility.
> > > >>> getAcls takes a Resource, right, which would be either a full
> > resource
> > > >>> name or 'ALL', i.e. '*', right?  The point of the call is to get all
> > > ACLs
> > > >>> relating to a specific resource, not a partial resource like
> > 'topicA*'.
> > > >>> Currently, I'm guessing / half-remembering that if you ask it for
> > ACLs
> > > >> for
> > > >>> topic 'foo' it doesn't include global 'ALL' ACLs in the list - that
> > > would
> > > >>> be a different call.  With the introduction of partial wildcards I
> > > think
> > > >>> the _most_ backwards compatible change would be to have
> > > >>> getAcls("topic:foo") to return all the ACLs, including that affect
> > this
> > > 

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

[Colin McCabe]

Oops, will post on the vote thread.

best,
Colin

On Mon, May 21, 2018, at 21:15, Colin McCabe wrote:
> On Mon, May 21, 2018, at 04:53, Andy Coates wrote:
> > Hey Piyush,
> > 
> > Thanks for the updated KIP! Couple of minor points from me:
> > 
> > When storing wildcard-suffixed Acls in ZK, drop the asterisk of the end for
> > the path, e.g. change "*/kafka-wildcard-acl/Topic/teamA*" * to "*/*
> > *kafka-wildcard-acl**/Topic/**teamA"*. This reduces error conditions, i.e.
> > this is a place for storing wildcard-suffixed Acls, so it implicitly ends
> > in an asterisk. If you include the asterisk in the path then you need to
> > validate each entry, when reading, ends with an asterisk, and do something
> > if they don't. If you don't include the training asterisk then there is
> > nothing to validate and you can treat the prefix as a literal, (i.e. no
> > escaping needed).  TBH I'd probably drop the asterisk from the in-memory
> > representation as well, for the same reason.
> 
> Hi Andy,
> 
> I agree.  If everything in ZK under /kafka-wildcard-acl/ is a prefix 
> ACL, there is no need to include the star at the end.  And really, it 
> should be called something like /kafka-prefix-acl/, since it's only 
> vaguely related to the idea of wildcards.
> 
> > 
> > Rather than creating an enum to indicate the type of a resource, you could
> > instead use polymorphism, e.g. make Resource an interface and have two
> > implementations: LiteralResource and WildcardSuffixedResource.  This is
> > also extensible, but may also allow for a cleaner implementation.
> 
> Since Resource is a concrete class now, we can't make it an interface 
> without breaking API compatibility.
> 
> Even if it were possible to do compatibly, I would argue it's a bad 
> idea.  If we need to add another bit of state like case insensitivity, 
> we don't want to have LiteralCaseInsensistiveResource, 
> WildcardSuffixedCaseInsensitiveResource, etc. etc.  You need 2^n 
> subclasses classes to represent N bits of state.
> 
> I would argue that there should be a field in Resource like NameType 
> which can be LITERAL or PREFIX.  That leaves us in a good position when 
> someone inevitably comes up with a new NameType.
> 
> Does this still have a chance to get in, or has the KIP window closed?  
> I would argue with one or two minor changes it's ready to go.  Pretty 
> much all of the compatibility problems are solved with the separate ZK 
> hierarchy.
> 
> best,
> Colin
> 
> > 
> > Andy
> > 
> > On 21 May 2018 at 01:58, Rajini Sivaram  wrote:
> > 
> > > Hi Piyush, Thanks for the KIP!
> > >
> > > +1 (binding)
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Sun, May 20, 2018 at 2:53 PM, Andy Coates  wrote:
> > >
> > > > Awesome last minute effort Piyush.
> > > >
> > > > Really appreciate your time and input,
> > > >
> > > > Andy
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 19 May 2018, at 03:43, Piyush Vijay  wrote:
> > > > >
> > > > > Updated the KIP.
> > > > >
> > > > > 1. New enum field 'ResourceNameType' in Resource and ResourceFilter
> > > > classes.
> > > > > 2. modify getAcls() and rely on ResourceNameType' field in Resource to
> > > > > return either exact matches or all matches based on wildcard-suffix.
> > > > > 3. CLI changes to identify if resource name is literal or
> > > wildcard-suffix
> > > > > 4. Escaping doesn't work and isn't required if we're keeping a 
> > > > > separate
> > > > > path on ZK (kafka-wildcard-acl) to store wildcard-suffix ACLs.
> > > > > 5. New API keys for Create / Delete / Describe Acls request with a new
> > > > > field in schemas for 'ResourceNameType'.
> > > > >
> > > > > Looks ready to me for the vote, will start voting thread now. Thanks
> > > > > everyone for the valuable feedback.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Piyush Vijay
> > > > >
> > > > >
> > > > > Piyush Vijay
> > > > >
> > > > >> On Fri, May 18, 2018 at 6:07 PM, Andy Coates 
> > > wrote:
> > > > >>
> > > > >> Hi Piyush,
> > > > >>
> > > > >> We're fast approaching the KIP deadline. Are you actively working on
> > > > this?
> > > > >> If you're not I can take over.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Andy
> > > > >>
> > > > >>> On 18 May 2018 at 14:25, Andy Coates  wrote:
> > > > >>>
> > > > >>> OK I've read it now.
> > > > >>>
> > > > >>> 1. I see you have an example:
> > > >  For example: If I want to fetch all ACLs that match ’topicA*’, it’s
> > > > not
> > > > >>> possible without introducing new API AND maintaining backwards
> > > > >>> compatibility.
> > > > >>> getAcls takes a Resource, right, which would be either a full
> > > resource
> > > > >>> name or 'ALL', i.e. '*', right?  The point of the call is to get all
> > > > ACLs
> > > > >>> relating to a specific resource, not a partial resource like
> > > 'topicA*'.
> > > > >>> Currently, I'm guessing / half-remembering that if you 

Re: [VOTE] KIP-290: Support for wildcard suffixed ACLs

[Colin McCabe]

Hmm, do we still need one more binding +1, or did I misread the vote thread?

+1 (non-binding) from me.

As I posted in the other thread, I think it would make sense to name the new ZK 
hierarchy /kafka-prefix-acls or similar, to reflect the fact that they are ACLs 
that match by a name prefix.  They're not really "wildcards"

It would also be nice to have a MatchType enum in the Resource class like 
LITERAL (for existing ACLs) and PREFIX (for the new ACL type).  This would put 
us in a good position to implement new ACL types in the future if the desire 
arises, and also clearly explain how each type works.

best,
Colin


On Mon, May 21, 2018, at 13:40, Piyush Vijay wrote:
> Thanks for the +1s. I'll send out a PR shortly.
> 
> Thanks
> 
> 
> Piyush Vijay
> 
> On Mon, May 21, 2018 at 8:59 AM, Guozhang Wang  wrote:
> 
> > Thanks for the KIP, +1 from me (binding).
> >
> >
> > Guozhang
> >
> > On Mon, May 21, 2018 at 5:22 AM, Damian Guy  wrote:
> >
> > > +1 (binding)
> > >
> > > On Sat, 19 May 2018 at 03:51 Piyush Vijay 
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I would like to start a vote for
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 290%3A+Support+for+wildcard+suffixed+ACLs
> > > > .
> > > >
> > > > The KIP proposes a way to support wildcard-suffixed resource names in
> > > Kafka
> > > > ACLs.
> > > >
> > > > The main challenge was to support it in a backward compatible way
> > because
> > > > resources like consumer groups don't have defined naming convention and
> > > can
> > > > have '*' in their names.
> > > >
> > > > Please take a look.
> > > >
> > > > Thanks
> > > >
> > > > Piyush Vijay
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >

Build failed in Jenkins: kafka-trunk-jdk8 #2662

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Ignore test_broker_type_bounce_at_start system test (#5055)

--
[...truncated 423.43 KB...]
kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftPlus STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftPlus PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLatest STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLatest PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNewConsumerExistingTopic STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsNewConsumerExistingTopic PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByLowerThanEarliest PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsByDuration PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToLocalDateTime 
PASSED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions STARTED

kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsToEarliestOnTopicsAndPartitions PASSED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
STARTED

kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliestOnTopics 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldUseDefaultsIfEnabled 
PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 

Re: [DISCUSS] KIP-305: Add Connect primitive number converters

[Ewen Cheslack-Postava]

Sorry, probably not ideal to just be seeing this now given KIP deadline,
but in general straightforward ones where we expect things to be
uncontroversial, I think its fine to kick off a vote thread. Worst case we
have people re-file votes if something substantial changes in the proposal.

-Ewen

On Sat, May 19, 2018 at 6:20 AM Randall Hauch  wrote:

> Considering this KIP is straightforward, what do you think about kicking
> off a vote? Or does it need more discussion time?
>
> Regards,
> Randall
>
> > On May 18, 2018, at 4:30 PM, Ewen Cheslack-Postava 
> wrote:
> >
> > Yeah, the usefulness of short seems questionable, but int is probably a
> > large enough range for some identifiers (e.g. we use an int in schema
> > registry). But yeah, I don't really have a problem with having Converters
> > for each of the existing serdes just for symmetry and since presumably
> > somebody finds them useful for something if they exist.
> >
> > -Ewen
> >
> >> On Fri, May 18, 2018 at 11:55 AM Randall Hauch 
> wrote:
> >>
> >> Thanks, Ewen.
> >>
> >> You make several good points, and I've updated the KIP to hopefully
> address
> >> your comments. I think the symmetry with the Kafka serdes is useful, so
> >> I've kept all 5 converters in the KIP.
> >>
> >> Interestingly, perhaps the short and int converters (with the reduced
> >> ranges) are not necessarily that useful for keys either.
> >>
> >> Regards,
> >>
> >> Randall
> >>
> >> On Thu, May 17, 2018 at 10:08 PM, Ewen Cheslack-Postava <
> e...@confluent.io
> >>>
> >> wrote:
> >>
> >>> Just a couple of minor points that don't really affect the
> >> implementation:
> >>>
> >>> * For nulls, let's just mention the underlying serializers already
> >> support
> >>> this. I'm actually not sure why they should/need to, but given they do,
> >>> let's just defer to that implementation.
> >>> * I'm not sure where Float and Double converters are actually useful.
> The
> >>> use cases I know for integer serdes is for keys, but floats seem like a
> >> bad
> >>> choice for keys. These aren't a lot of overhead to build and maintain,
> >> but
> >>> if we don't know use cases for the specific types, it might be silly to
> >>> spend time and effort building and maintaining them.
> >>>
> >>> Otherwise, this seems simple and straightforward. Generally +1 on the
> >>> proposal.
> >>>
> >>> -Ewen
> >>>
> >>> On Thu, May 17, 2018 at 6:04 PM Magesh Nandakumar <
> mage...@confluent.io>
> >>> wrote:
> >>>
>  Thanks Randall for the KIP. I think it will be super useful and looks
>  pretty straightforward to me.
> 
>  Thanks
>  Magesh
> 
>  On Thu, May 17, 2018 at 4:15 PM, Randall Hauch 
> >> wrote:
> 
> > I'd like to start discussion of a very straightforward proposal for
>  Connect
> > to add converters for the basic primitive number types: integer,
> >> short,
> > long, double, and float. Here is the KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 305%3A+Add+Connect+primitive+number+converters
> >
> > As mentioned in the KIP, I've created a pull request (
> > https://github.com/apache/kafka/pull/5034) for those looking for
> > implementation details.
> >
> > Any feedback is appreciated.
> >
> > Best regards,
> >
> > Randall
> >
> 
> >>>
> >>
>

Build failed in Jenkins: kafka-trunk-jdk10 #122

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Ignore test_broker_type_bounce_at_start system test (#5055)

--
[...truncated 1.06 MB...]
org.apache.kafka.clients.producer.internals.RecordAccumulatorTest > 
testPartialDrain STARTED

org.apache.kafka.clients.producer.internals.RecordAccumulatorTest > 
testPartialDrain PASSED

org.apache.kafka.clients.producer.internals.RecordAccumulatorTest > 
testSplitAndReenqueue STARTED

org.apache.kafka.clients.producer.internals.RecordAccumulatorTest > 
testSplitAndReenqueue PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testSimple STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testSimple PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testBlockTimeout 
STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > testBlockTimeout 
PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnBlockTimeout STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnBlockTimeout PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnInterruption STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityWaiterOnInterruption PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testLargeAvailableMemory STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testLargeAvailableMemory PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
outOfMemoryOnAllocation STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
outOfMemoryOnAllocation PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testStressfulSituation STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testStressfulSituation PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCantAllocateMoreMemoryThanWeHave STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCantAllocateMoreMemoryThanWeHave PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testDelayedAllocation STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testDelayedAllocation PASSED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityOnMetricsException STARTED

org.apache.kafka.clients.producer.internals.BufferPoolTest > 
testCleanupMemoryAvailabilityOnMetricsException PASSED

org.apache.kafka.clients.producer.internals.ProducerInterceptorsTest > 
testOnAcknowledgementWithErrorChain STARTED

org.apache.kafka.clients.producer.internals.ProducerInterceptorsTest > 
testOnAcknowledgementWithErrorChain PASSED

org.apache.kafka.clients.producer.internals.ProducerInterceptorsTest > 
testOnAcknowledgementChain STARTED

org.apache.kafka.clients.producer.internals.ProducerInterceptorsTest > 
testOnAcknowledgementChain PASSED

org.apache.kafka.clients.producer.internals.ProducerInterceptorsTest > 
testOnSendChain STARTED

org.apache.kafka.clients.producer.internals.ProducerInterceptorsTest > 
testOnSendChain PASSED

org.apache.kafka.clients.producer.internals.SenderTest > 
testIdempotenceWithMultipleInflightsWhereFirstFailsFatallyAndSequenceOfFutureBatchesIsAdjusted
 STARTED

org.apache.kafka.clients.producer.internals.SenderTest > 
testIdempotenceWithMultipleInflightsWhereFirstFailsFatallyAndSequenceOfFutureBatchesIsAdjusted
 PASSED

org.apache.kafka.clients.producer.internals.SenderTest > 
testCancelInFlightRequestAfterFatalError STARTED

org.apache.kafka.clients.producer.internals.SenderTest > 
testCancelInFlightRequestAfterFatalError PASSED

org.apache.kafka.clients.producer.internals.SenderTest > 
testIdempotentSplitBatchAndSend STARTED

org.apache.kafka.clients.producer.internals.SenderTest > 
testIdempotentSplitBatchAndSend PASSED

org.apache.kafka.clients.producer.internals.SenderTest > 
testUnknownProducerErrorShouldBeRetriedWhenLogStartOffsetIsUnknown STARTED

org.apache.kafka.clients.producer.internals.SenderTest > 
testUnknownProducerErrorShouldBeRetriedWhenLogStartOffsetIsUnknown PASSED

org.apache.kafka.clients.producer.internals.SenderTest > 
testMetadataTopicExpiry STARTED

org.apache.kafka.clients.producer.internals.SenderTest > 
testMetadataTopicExpiry PASSED

org.apache.kafka.clients.producer.internals.SenderTest > 
testMessageFormatDownConversion STARTED

org.apache.kafka.clients.producer.internals.SenderTest > 
testMessageFormatDownConversion PASSED

org.apache.kafka.clients.producer.internals.SenderTest > 
testUnknownProducerHandlingWhenRetentionLimitReached STARTED

org.apache.kafka.clients.producer.internals.SenderTest > 
testUnknownProducerHandlingWhenRetentionLimitReached PASSED

org.apache.kafka.clients.producer.internals.SenderTest > 

Re: [VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Jason Gustafson]

+1. Just one nit: could we use an INT type for the config? I can't imagine
that not being enough.

-Jason

On Mon, May 21, 2018 at 3:59 PM, Ismael Juma  wrote:

> Thanks for the KIP, +1 (binding).
>
> Ismael
>
> On Mon, May 21, 2018 at 7:52 AM Dhruvil Shah  wrote:
>
> > Hi,
> >
> > I would like to start a vote on KIP-306 which proposes to add a
> > configuration to delay responses to failed authentication.
> >
> > Link to the KIP:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 306%3A+Configuration+for+Delaying+Response+to+Failed+Client+Authentication
> >
> > Thanks,
> > Dhruvil
> >
>

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

[Ewen Cheslack-Postava]

Hey all,

I think think this is a great discussion, and is helping to clarify the
real pain points as well as explore a few more options than just what was
initially proposed.

Stephane, I think why you're ending up in "grand redesign" state is because
you're highlighting (and the KIP's motivation section examples of pain
points) are actually a set of a couple of different high level challenges:


   - Impact of shared resources, especially rebalancing but also just
   shared process space/memory/etc
   - Kafka security credentials / shared client configs
   - How configs are specified in the two modes

I actually think there isn't much in the identified problems that are
unique to containers or k8s/mesos, that's just an environment you'll
commonly run into these challenges.

I want to cover each of these issues to see if we can improve them in a
more incremental way.

*Impact of shared resources, especially rebalancing but also just shared
process space/memory/etc*
There are 2 aspects to this. The first is rebalancing. There's actually
nothing that fundamentally requires every rebalance operation to be
stop-the-world. I've had some thoughts about allowing partial and
incremental rebalancing floating around for awhile. As Connect clusters are
growing larger, this is becoming increasingly important. I don't have a
full write up ready, but it's fixable by making a straightforward change
(and we planned for the ability to make changes like this, so it absolutely
should be possible to do and apply to existing clusters). The basic idea is
to change what starting a rebalance means for ownership of resources --
instead of assuming everything might be rearranged, you assume by default
that you will continue to own everything you currently do and continue
processing through the rebalance. The leader who handles assignment can
then decide what set of rebalances really are required and include that
info in the data it sends back. If any rebalancing is required, do a
subsequent round of rebalancing where you actually give up those resources
if they were assigned to you. This gives you a way to do partial rebalances
and only as needed. You can further extend this in a variety of ways, e.g.
only rebalancing one resource per rebalance, doing a bit of "scheduling" to
spread out the impact, etc. This is definitely not a trivial
change/addition to make and will require very thorough testing, but it's
definitely feasible for 1 person to implement themselves pretty quickly.

The second aspect is shared resources with no control over different
connectors usage (and the fact that someone doing something bad might mean
all connectors crash with the workers). Containers are great for this that
give you more granularity than VMs or physical hosts, but either one works.
If you want to use any of these solutions, you are going to have to have
separate Connect clusters. There are some pain points if you go this route,
but I think some you can just live with (e.g. 3 topics per connector isn't
actually that insane), some are fixable and just not there yet (e.g. the
upgrade example affecting the whole cluster is also fixable by just
supporting dynamically loading in new jars instead of requiring a restart).

*Kafka security credentials / shared client configs*
This is definitely one we hear from time to time. In general, there's no
reason we couldn't just expose security configs in the connector config. I
have pushed back on just exposing all configs because a) I don't think
users *should* have control over them, b) in a shared cluster, some
represent potentially serious stability threats (e.g. I override buffer
sizes and OOM everything), and c) it's actually a compatibility concern.
However, I don't think that means we shouldn't consider exposing some of
them if there are good use cases (such as here, where you might want to
lock down different connectors from each other via ACLs).

For the purposes of this discussion, I don't really think this is the thing
to focus on -- given you're already committing to having separate security
configs for each connector, it seems like whether you configure multiple
clusters vs configure them per connector isn't that much different.

*How configs are specified in the two modes*
This one is, I think, the most interesting because it has been requested a
few times before and I'm not sure anyone has actually worked through
semantics that are consistent and handle all the different requirements. Of
all the stuff mentioned so far in this discussion, this is the piece that
is really the biggest pain point that's not easy to work around today --
even if you split out to separate containers per connector, you still want
distributed mode because a single connector may need to be spread across
multiple containers. However, the default way for apps to be configured is
for them to get config files/parameters directly with each process, and so
it seems nice if you are only running 1 connecter per cluster, to 

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Gwen Shapira]

Recasting my +1

On Mon, May 21, 2018 at 3:38 PM, Guozhang Wang  wrote:

> Thanks Matthias, will do.
>
> On Mon, May 21, 2018 at 3:35 PM, Matthias J. Sax 
> wrote:
>
> > Thanks. I am fine with changing the `StreamPartitioner` interface
> directly.
> >
> > Can you add the idea bout `DynamicStreamPartitioner` to "rejected
> > alternative" section. Thx.
> >
> >
> > recasting +1 (binding)
> >
> >
> > -Matthias
> >
> >
> > On 5/21/18 3:04 PM, Guozhang Wang wrote:
> > > Hello Matthias,
> > >
> > > I've tried it out on the PR, the implementation should be fine but one
> > > concern I had is that, as you may also realize users of
> > > DynamicStreamPartitioner needs to implement two interface functions,
> with
> > > and without the topic name if it is extending from StreamPartitioner;
> we
> > > could also let it to not extend from StreamPartition so it has one
> > function
> > > only but then we'd need Produced to have two functions allowing
> > > StreamPartitioner and DynamicStreamPartitioner. Thinking about the pros
> > and
> > > cons I'm think it may be better to just change the interface of
> > > StreamPartitioner itself, since even without dynamic routing, allowing
> > the
> > > topic name could let users to give one partitioner implementation that
> > > branch on the topic names other than having one partitioner per topic.
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Mon, May 21, 2018 at 11:56 AM, Matthias J. Sax <
> matth...@confluent.io
> > >
> > > wrote:
> > >
> > >> I think that the risk of the change is moderate as I expect most
> people
> > >> to use the DefaultStreamPartitioner.
> > >>
> > >> However, there would still be possibility to define a new interface
> > >> instead of changing the old:
> > >>
> > >>> public interface DynamicStreamPartitioner {
> > >>> Integer partition(String topic, K key, V value, int
> numPartitions);
> > >>> }
> > >>
> > >> The newly added methods `Topology#addSink` and `KStream#to` would take
> > >> this new interface instead of the old.
> > >>
> > >> Maybe `DynamicStreamPartitioner` must extend `StreamPartitioner` to
> make
> > >> runtime code work though...
> > >>
> > >> WDYT?
> > >>
> > >> -Matthias
> > >>
> > >> On 5/21/18 11:47 AM, Guozhang Wang wrote:
> > >>> Hello everyone,
> > >>>
> > >>> While implementing the PR for this KIP I realized there is once place
> > >> which
> > >>> we should consider modifying on public APIs as well:
> > >>> StreamPartitioner#partition, to add the topic name string. Note it
> will
> > >> be
> > >>> a incompatible change that requires users who have customized
> > >>> StreamPartitioner implementations.
> > >>>
> > >>> I've updated the wiki page of KIP-303, please recast your vote on
> this
> > >>> thread. Thanks!!!
> > >>>
> > >>>
> > >>> Guozhang
> > >>>
> > >>>
> > >>> On Thu, May 17, 2018 at 3:15 PM, John Roesler 
> > wrote:
> > >>>
> >  +1 non-binding
> > 
> >  On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax <
> > matth...@confluent.io
> > >>>
> >  wrote:
> > 
> > > +1 (binding)
> > >
> > >
> > > On 5/17/18 12:18 PM, Ted Yu wrote:
> > >> +1
> > >>  Original message From: Gwen Shapira <
> >  g...@confluent.io>
> > > Date: 5/17/18  11:53 AM  (GMT-08:00) To: dev  >
> > > Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka
> >  Streams'
> > > Topology Sink
> > >> Yay, its about time :)
> > >>
> > >> +1
> > >>
> > >> On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang <
> wangg...@gmail.com
> > >
> > > wrote:
> > >>
> > >>> Hello folks,
> > >>>
> > >>> I'd like to start a voting thread on adding dynamic routing
> > > functionality
> > >>> in Streams sink node. Please find a KIP here:
> > >>>
> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>> 303%3A+Add+Dynamic+Routing+in+Streams+Sink
> > >>>
> > >>>
> > >>> And the PR itself ready for review as well under KAFKA-4936:
> > >>>
> > >>> https://github.com/apache/kafka/pull/5018
> > >>>
> > >>>
> > >>>
> > >>> Thanks!
> > >>> -- Guozhang
> > >>>
> > >>
> > >>
> > >>
> > >
> > >
> > 
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >
> > >
> >
> >
>
>
> --
> -- Guozhang
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: KAFKA-6733 feedback

[Ismael Juma]

Hi Mateusz,

This is a good change, but it requires a KIP as mentioned in the PR. You
gave an example of a similar PR without a KIP, but there was a KIP for it:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-265%3A+Make+Windowed+Serde+to+public+APIs

Ismael

On Mon, May 21, 2018 at 4:00 PM Mateusz Zakarczemny 
wrote:

> Hi,
> Could I ask for some feedback regarding
> https://github.com/apache/kafka/pull/4807 ?
> It's waiting 1,5 month. I had to resolve conflicts with trunk couple of
> times.
> I would be grateful if someone could take a look it.
>
> Regards,
> Mateusz Zakarczemny
>

KAFKA-6733 feedback

[Mateusz Zakarczemny]

Hi,
Could I ask for some feedback regarding
https://github.com/apache/kafka/pull/4807 ?
It's waiting 1,5 month. I had to resolve conflicts with trunk couple of
times.
I would be grateful if someone could take a look it.

Regards,
Mateusz Zakarczemny

Re: [VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Ismael Juma]

Thanks for the KIP, +1 (binding).

Ismael

On Mon, May 21, 2018 at 7:52 AM Dhruvil Shah  wrote:

> Hi,
>
> I would like to start a vote on KIP-306 which proposes to add a
> configuration to delay responses to failed authentication.
>
> Link to the KIP:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-306%3A+Configuration+for+Delaying+Response+to+Failed+Client+Authentication
>
> Thanks,
> Dhruvil
>

RE: Kafka consumer to unzip stream of .gz files and read

[Koushik Chitta]

You should read the message value as byte array rather than string .
Other Approach is , while producing you can use the kafka compression = GZIP to 
have similar results.


-Original Message-
From: mayur shah  
Sent: Monday, May 21, 2018 1:50 AM
To: us...@kafka.apache.org; dev@kafka.apache.org
Subject: Kafka consumer to unzip stream of .gz files and read

 HI Team,

Greeting!

I am facing one issue on kafka consumer using python hope you guys help us to 
resolve this issue

Kafka consumer to unzip stream of .gz files and read 


Kafka producer is sending .gz files but not able to decompress and read the 
files at the consumer end. Getting error as "IOError: Not a gzipped file"

Producer -

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Airport < 
~/Downloads/stocks.json.gz

Consumer -

import sys import gzipimport StringIOfrom kafka import KafkaConsumer

consumer = KafkaConsumer(KAFKA_TOPIC, bootstrap_servers=KAFKA_BROKERS)
try:
for message in consumer:
f = StringIO.StringIO(message.value)
gzip_f = gzip.GzipFile(fileobj=f)
unzipped_content = gzip_f.read()
content = unzipped_content.decode('utf8')
print (content)except KeyboardInterrupt:
sys.exit()

Error at consumer -

Traceback (most recent call last):
  File "consumer.py", line 18, in 
unzipped_content = gzip_f.read()
  File "/usr/lib64/python2.6/gzip.py", line 212, in read
self._read(readsize)
  File "/usr/lib64/python2.6/gzip.py", line 255, in _read
self._read_gzip_header()
  File "/usr/lib64/python2.6/gzip.py", line 156, in _read_gzip_header
raise IOError, 'Not a gzipped file'IOError: Not a gzipped file

Regards,
Mayur

Re: [VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Edoardo Comar]

+1 (non-binding)
Thanks !

On 21 May 2018 at 19:18, Bill Bejeck  wrote:
> Thanks for the KIP.
>
> +1
>
> -Bill
>
> On Mon, May 21, 2018 at 1:41 PM, Ted Yu  wrote:
>
>> +1
>>
>> On Mon, May 21, 2018 at 9:02 AM, Guozhang Wang  wrote:
>>
>> > Thanks Dhruvil for the KIP, +1 from me (binding).
>> >
>> >
>> > Guozhang
>> >
>> > On Mon, May 21, 2018 at 7:52 AM, Dhruvil Shah 
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I would like to start a vote on KIP-306 which proposes to add a
>> > > configuration to delay responses to failed authentication.
>> > >
>> > > Link to the KIP:
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > 306%3A+Configuration+for+Delaying+Response+to+Failed+
>> > Client+Authentication
>> > >
>> > > Thanks,
>> > > Dhruvil
>> > >
>> >
>> >
>> >
>> > --
>> > -- Guozhang
>> >
>>



-- 
"When the people fear their government, there is tyranny; when the
government fears the people, there is liberty." [Thomas Jefferson]

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Guozhang Wang]

Thanks Matthias, will do.

On Mon, May 21, 2018 at 3:35 PM, Matthias J. Sax 
wrote:

> Thanks. I am fine with changing the `StreamPartitioner` interface directly.
>
> Can you add the idea bout `DynamicStreamPartitioner` to "rejected
> alternative" section. Thx.
>
>
> recasting +1 (binding)
>
>
> -Matthias
>
>
> On 5/21/18 3:04 PM, Guozhang Wang wrote:
> > Hello Matthias,
> >
> > I've tried it out on the PR, the implementation should be fine but one
> > concern I had is that, as you may also realize users of
> > DynamicStreamPartitioner needs to implement two interface functions, with
> > and without the topic name if it is extending from StreamPartitioner; we
> > could also let it to not extend from StreamPartition so it has one
> function
> > only but then we'd need Produced to have two functions allowing
> > StreamPartitioner and DynamicStreamPartitioner. Thinking about the pros
> and
> > cons I'm think it may be better to just change the interface of
> > StreamPartitioner itself, since even without dynamic routing, allowing
> the
> > topic name could let users to give one partitioner implementation that
> > branch on the topic names other than having one partitioner per topic.
> >
> >
> > Guozhang
> >
> >
> > On Mon, May 21, 2018 at 11:56 AM, Matthias J. Sax  >
> > wrote:
> >
> >> I think that the risk of the change is moderate as I expect most people
> >> to use the DefaultStreamPartitioner.
> >>
> >> However, there would still be possibility to define a new interface
> >> instead of changing the old:
> >>
> >>> public interface DynamicStreamPartitioner {
> >>> Integer partition(String topic, K key, V value, int numPartitions);
> >>> }
> >>
> >> The newly added methods `Topology#addSink` and `KStream#to` would take
> >> this new interface instead of the old.
> >>
> >> Maybe `DynamicStreamPartitioner` must extend `StreamPartitioner` to make
> >> runtime code work though...
> >>
> >> WDYT?
> >>
> >> -Matthias
> >>
> >> On 5/21/18 11:47 AM, Guozhang Wang wrote:
> >>> Hello everyone,
> >>>
> >>> While implementing the PR for this KIP I realized there is once place
> >> which
> >>> we should consider modifying on public APIs as well:
> >>> StreamPartitioner#partition, to add the topic name string. Note it will
> >> be
> >>> a incompatible change that requires users who have customized
> >>> StreamPartitioner implementations.
> >>>
> >>> I've updated the wiki page of KIP-303, please recast your vote on this
> >>> thread. Thanks!!!
> >>>
> >>>
> >>> Guozhang
> >>>
> >>>
> >>> On Thu, May 17, 2018 at 3:15 PM, John Roesler 
> wrote:
> >>>
>  +1 non-binding
> 
>  On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax <
> matth...@confluent.io
> >>>
>  wrote:
> 
> > +1 (binding)
> >
> >
> > On 5/17/18 12:18 PM, Ted Yu wrote:
> >> +1
> >>  Original message From: Gwen Shapira <
>  g...@confluent.io>
> > Date: 5/17/18  11:53 AM  (GMT-08:00) To: dev 
> > Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka
>  Streams'
> > Topology Sink
> >> Yay, its about time :)
> >>
> >> +1
> >>
> >> On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang  >
> > wrote:
> >>
> >>> Hello folks,
> >>>
> >>> I'd like to start a voting thread on adding dynamic routing
> > functionality
> >>> in Streams sink node. Please find a KIP here:
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 303%3A+Add+Dynamic+Routing+in+Streams+Sink
> >>>
> >>>
> >>> And the PR itself ready for review as well under KAFKA-4936:
> >>>
> >>> https://github.com/apache/kafka/pull/5018
> >>>
> >>>
> >>>
> >>> Thanks!
> >>> -- Guozhang
> >>>
> >>
> >>
> >>
> >
> >
> 
> >>>
> >>>
> >>>
> >>
> >>
> >
> >
>
>


-- 
-- Guozhang

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Matthias J. Sax]

Thanks. I am fine with changing the `StreamPartitioner` interface directly.

Can you add the idea bout `DynamicStreamPartitioner` to "rejected
alternative" section. Thx.


recasting +1 (binding)


-Matthias


On 5/21/18 3:04 PM, Guozhang Wang wrote:
> Hello Matthias,
> 
> I've tried it out on the PR, the implementation should be fine but one
> concern I had is that, as you may also realize users of
> DynamicStreamPartitioner needs to implement two interface functions, with
> and without the topic name if it is extending from StreamPartitioner; we
> could also let it to not extend from StreamPartition so it has one function
> only but then we'd need Produced to have two functions allowing
> StreamPartitioner and DynamicStreamPartitioner. Thinking about the pros and
> cons I'm think it may be better to just change the interface of
> StreamPartitioner itself, since even without dynamic routing, allowing the
> topic name could let users to give one partitioner implementation that
> branch on the topic names other than having one partitioner per topic.
> 
> 
> Guozhang
> 
> 
> On Mon, May 21, 2018 at 11:56 AM, Matthias J. Sax 
> wrote:
> 
>> I think that the risk of the change is moderate as I expect most people
>> to use the DefaultStreamPartitioner.
>>
>> However, there would still be possibility to define a new interface
>> instead of changing the old:
>>
>>> public interface DynamicStreamPartitioner {
>>> Integer partition(String topic, K key, V value, int numPartitions);
>>> }
>>
>> The newly added methods `Topology#addSink` and `KStream#to` would take
>> this new interface instead of the old.
>>
>> Maybe `DynamicStreamPartitioner` must extend `StreamPartitioner` to make
>> runtime code work though...
>>
>> WDYT?
>>
>> -Matthias
>>
>> On 5/21/18 11:47 AM, Guozhang Wang wrote:
>>> Hello everyone,
>>>
>>> While implementing the PR for this KIP I realized there is once place
>> which
>>> we should consider modifying on public APIs as well:
>>> StreamPartitioner#partition, to add the topic name string. Note it will
>> be
>>> a incompatible change that requires users who have customized
>>> StreamPartitioner implementations.
>>>
>>> I've updated the wiki page of KIP-303, please recast your vote on this
>>> thread. Thanks!!!
>>>
>>>
>>> Guozhang
>>>
>>>
>>> On Thu, May 17, 2018 at 3:15 PM, John Roesler  wrote:
>>>
 +1 non-binding

 On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax >>
 wrote:

> +1 (binding)
>
>
> On 5/17/18 12:18 PM, Ted Yu wrote:
>> +1
>>  Original message From: Gwen Shapira <
 g...@confluent.io>
> Date: 5/17/18  11:53 AM  (GMT-08:00) To: dev 
> Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka
 Streams'
> Topology Sink
>> Yay, its about time :)
>>
>> +1
>>
>> On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang 
> wrote:
>>
>>> Hello folks,
>>>
>>> I'd like to start a voting thread on adding dynamic routing
> functionality
>>> in Streams sink node. Please find a KIP here:
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 303%3A+Add+Dynamic+Routing+in+Streams+Sink
>>>
>>>
>>> And the PR itself ready for review as well under KAFKA-4936:
>>>
>>> https://github.com/apache/kafka/pull/5018
>>>
>>>
>>>
>>> Thanks!
>>> -- Guozhang
>>>
>>
>>
>>
>
>

>>>
>>>
>>>
>>
>>
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Arjun Satish]

Hey Jason,

This KIP does take serialization errors to be retriable. The typical use
case is that Schema Registry can have a bad/unavailable schema, which can
be corrected over time.

But since the converters throw DataExceptions for all failures, it is hard
to determine what caused these errors. Hence, we are going to retry on any
Exception thrown from a converter.

Hope that works.

Best,


On Mon, May 21, 2018 at 2:15 PM, Jason Gustafson  wrote:

> Thanks Arjun. I had one additional question. High level, I'm wondering if
> it makes sense to treat processing errors such as serialization failures
> the same as IO errors. In the former case, retrying typically doesn't help
> because the processing is deterministic. In the latter case, the failure
> may be downstream (e.g. a kafka partition may be temporarily unavailable).
> As a user, I would probably want the option to skip over processing
> failures, but retry indefinitely if the downstream system is unavailable.
> Is that use case supported?
>
> Thanks,
> Jason
>
>
>
> On Mon, May 21, 2018 at 12:39 PM, Arjun Satish 
> wrote:
>
> > Thanks a lot, Ewen! I'll make sure the documentation is clear on the
> > differences between retries an tolerance.
> >
> > Do you think percentage would have the same problem as the one you
> brought
> > up? Also, if we say 10% tolerance, do we have to wait for the duration to
> > finish before failing the task, or should we fail as soon as we hit 10%
> > error.
> >
> > Alternatively, do you think making tolerance an Enum would be simpler?
> > Where it's values are NONE (any errors kill), ALL (tolerate all errors
> and
> > skip records) and FIRST (tolerate the first error, but fail after that)?
> >
> > Best,
> >
> >
> > On Mon, May 21, 2018 at 11:28 AM, Ewen Cheslack-Postava <
> e...@confluent.io
> > >
> > wrote:
> >
> > > Arjun,
> > >
> > > Understood on retries vs tolerance -- though I suspect this will end up
> > > being a bit confusing to users as well. It's two levels of error
> handling
> > > which is what tripped me up.
> > >
> > > One last comment on KIP (which otherwise looks good): for the tolerance
> > > setting, do we want it to be an absolute value or something like a
> > > percentage? Given the current way of setting things, I'm not sure I'd
> > ever
> > > set it to anything but -1 or 0, with maybe 1 as an easy option for
> > > restarting a connector to get it past one bad message, then reverting
> > back
> > > to -1 or 0.
> > >
> > > -Ewen
> > >
> > > On Mon, May 21, 2018 at 11:01 AM Arjun Satish 
> > > wrote:
> > >
> > > > Hey Jason,
> > > >
> > > > Thanks for your comments. Please find answers inline:
> > > >
> > > > On Mon, May 21, 2018 at 9:52 AM, Jason Gustafson  >
> > > > wrote:
> > > >
> > > > > Hi Arjun,
> > > > >
> > > > > Thanks for the KIP. Just a few comments/questions:
> > > > >
> > > > > 1. The proposal allows users to configure the number of retries. I
> > > > usually
> > > > > find it easier as a user to work with timeouts since it's difficult
> > to
> > > > know
> > > > > how long a retry might take. Have you considered adding a timeout
> > > option
> > > > > which would retry until the timeout expires?
> > > > >
> > > >
> > > > Good point. Updated the KIP.
> > > >
> > > > 2. The configs are named very generically (e.g.
> errors.retries.limit).
> > Do
> > > > > you think it will be clear to users what operations these configs
> > apply
> > > > to?
> > > > >
> > > >
> > > > As of now, these configs are applicable to all operations in the
> > > connector
> > > > pipeline (as mentioned in the proposed changes section). We decided
> not
> > > to
> > > > have per operation limit because of the additional config complexity.
> > > >
> > > >
> > > > > 3. I wasn't entirely clear what messages are stored in the dead
> > letter
> > > > > queue. It sounds like it includes both configs and messages since
> we
> > > have
> > > > > errors.dlq.include.configs? Is there a specific schema you have in
> > > mind?
> > > > >
> > > >
> > > > This has been addressed in the KIP. The DLQ will now contain only raw
> > > > messages (no additional context). We are also supporting DLQs only
> for
> > > sink
> > > > connectors now.
> > > >
> > > >
> > > > > 4. I didn't see it mentioned explicitly in the KIP, but I assume
> the
> > > > > tolerance metrics are reset after every task rebalance?
> > > > >
> > > >
> > > > Great question. Yes, we will reset the tolerance metrics on every
> > > > rebalance.
> > > >
> > > >
> > > > > 5. I wonder if we can do without errors.tolerance.limit. You can
> get
> > a
> > > > > similar effect using errors.tolerance.rate.limit if you allow
> longer
> > > > > durations. I'm not sure how useful an absolute counter is in
> > practice.
> > > > >
> > > >
> > > > Yeah, the rate limit does subsume the features offered by the
> absolute
> > > > counter. Removed it.
> > > >
> > > >
> > > > >
> > > > > Thanks,
> > 

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Guozhang Wang]

Hello Matthias,

I've tried it out on the PR, the implementation should be fine but one
concern I had is that, as you may also realize users of
DynamicStreamPartitioner needs to implement two interface functions, with
and without the topic name if it is extending from StreamPartitioner; we
could also let it to not extend from StreamPartition so it has one function
only but then we'd need Produced to have two functions allowing
StreamPartitioner and DynamicStreamPartitioner. Thinking about the pros and
cons I'm think it may be better to just change the interface of
StreamPartitioner itself, since even without dynamic routing, allowing the
topic name could let users to give one partitioner implementation that
branch on the topic names other than having one partitioner per topic.


Guozhang


On Mon, May 21, 2018 at 11:56 AM, Matthias J. Sax 
wrote:

> I think that the risk of the change is moderate as I expect most people
> to use the DefaultStreamPartitioner.
>
> However, there would still be possibility to define a new interface
> instead of changing the old:
>
> > public interface DynamicStreamPartitioner {
> > Integer partition(String topic, K key, V value, int numPartitions);
> > }
>
> The newly added methods `Topology#addSink` and `KStream#to` would take
> this new interface instead of the old.
>
> Maybe `DynamicStreamPartitioner` must extend `StreamPartitioner` to make
> runtime code work though...
>
> WDYT?
>
> -Matthias
>
> On 5/21/18 11:47 AM, Guozhang Wang wrote:
> > Hello everyone,
> >
> > While implementing the PR for this KIP I realized there is once place
> which
> > we should consider modifying on public APIs as well:
> > StreamPartitioner#partition, to add the topic name string. Note it will
> be
> > a incompatible change that requires users who have customized
> > StreamPartitioner implementations.
> >
> > I've updated the wiki page of KIP-303, please recast your vote on this
> > thread. Thanks!!!
> >
> >
> > Guozhang
> >
> >
> > On Thu, May 17, 2018 at 3:15 PM, John Roesler  wrote:
> >
> >> +1 non-binding
> >>
> >> On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax  >
> >> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>>
> >>> On 5/17/18 12:18 PM, Ted Yu wrote:
>  +1
>   Original message From: Gwen Shapira <
> >> g...@confluent.io>
> >>> Date: 5/17/18  11:53 AM  (GMT-08:00) To: dev 
> >>> Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka
> >> Streams'
> >>> Topology Sink
>  Yay, its about time :)
> 
>  +1
> 
>  On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang 
> >>> wrote:
> 
> > Hello folks,
> >
> > I'd like to start a voting thread on adding dynamic routing
> >>> functionality
> > in Streams sink node. Please find a KIP here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 303%3A+Add+Dynamic+Routing+in+Streams+Sink
> >
> >
> > And the PR itself ready for review as well under KAFKA-4936:
> >
> > https://github.com/apache/kafka/pull/5018
> >
> >
> >
> > Thanks!
> > -- Guozhang
> >
> 
> 
> 
> >>>
> >>>
> >>
> >
> >
> >
>
>


-- 
-- Guozhang

[jira] [Created] (KAFKA-6928) StreamsPartitionAssignor is double retrying within InternalTopicManager

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-6928:


 Summary: StreamsPartitionAssignor is double retrying within 
InternalTopicManager
 Key: KAFKA-6928
 URL: https://issues.apache.org/jira/browse/KAFKA-6928
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Guozhang Wang


Today inside StreamsPartitionAssingor, we are using the admin client to 1) 
check num.partitions for internal topics, 2) try to create internal topics that 
are not exist yet. We are using the StreamsConfig.RETRIES_CONFIG inside the 
InternalTopicManager to capture thrown exceptions from the resulted 
KafkaFuture; however inside the AdminClient, we are again using the same config 
with its own retrying logic.

We should get rid of the outer retrying logic later; as of now (2018.05) there 
are still some issues with the admin client such that its own retrying logic is 
not safe. I'll leave it to [~hachikuji] to decide when admin client's own 
retrying is safe to rely on.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-244: Add Record Header support to Kafka Streams

[Jorge Esteban Quilcate Otoya]

Hi,

I'd like to point out that:

org.apache.kafka.streams.test.ConsumerRecordFactory


Has also been included as part of this KIP to support changes on test
cases, just in case there is additional feedback here.

Cheers,
Jorge.

El lun., 21 may. 2018 a las 16:46, Jorge Esteban Quilcate Otoya (<
quilcate.jo...@gmail.com>) escribió:

> Thanks for your votes and your feedback.
>
> This KIP has been approved with the following results:
>
> Binding +1s: 3 (Matthias, Damina, Guozhang)
> Non-biniding +1s: (Bill, Ted)
>
> Jorge.
>
> El mar., 15 may. 2018 a las 20:01, Bill Bejeck ()
> escribió:
>
>> Thanks for the KIP!
>>
>> +1
>>
>> -Bill
>>
>> On Tue, May 15, 2018 at 1:47 PM, Damian Guy  wrote:
>>
>> > Thanks. +1 (binding)
>> >
>> > On Tue, 15 May 2018 at 01:04 Jorge Esteban Quilcate Otoya <
>> > quilcate.jo...@gmail.com> wrote:
>> >
>> > > @Guozhang added. Thanks!
>> > >
>> > > El mar., 15 may. 2018 a las 5:50, Matthias J. Sax (<
>> > matth...@confluent.io
>> > > >)
>> > > escribió:
>> > >
>> > > > +1 (binding)
>> > > >
>> > > > Thanks a lot for the KIP!
>> > > >
>> > > > -Matthias
>> > > >
>> > > > On 5/14/18 10:17 AM, Guozhang Wang wrote:
>> > > > > +1 from me
>> > > > >
>> > > > > One more comment on the wiki: while reviewing the PR I realized
>> that
>> > > in `
>> > > > > MockProcessorContext.java
>> > > > > <
>> > > >
>> > > https://github.com/apache/kafka/pull/4955/files#diff-
>> > d5440e7338f775230019a86e6bcacccb
>> > > > >`
>> > > > > we are also adding one additional API plus modifying the existing
>> > > > > `setRecordMetadata` API. Since this class is part of the public
>> > > > test-utils
>> > > > > package we should claim it in the wiki as well.
>> > > > >
>> > > > >
>> > > > > Guozhang
>> > > > >
>> > > > > On Mon, May 14, 2018 at 8:43 AM, Ted Yu 
>> wrote:
>> > > > >
>> > > > >> +1
>> > > > >>
>> > > > >> On Mon, May 14, 2018 at 8:31 AM, Jorge Esteban Quilcate Otoya <
>> > > > >> quilcate.jo...@gmail.com> wrote:
>> > > > >>
>> > > > >>> Hi everyone,
>> > > > >>>
>> > > > >>> I would like to start a vote on KIP-244: Add Record Header
>> support
>> > to
>> > > > >> Kafka
>> > > > >>> Streams
>> > > > >>>
>> > > > >>> KIP wiki page:
>> > > > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > > > >>> 244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
>> > > > >>>
>> > > > >>> The discussion thread is here:
>> > > > >>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201805.
>> > > > >>>
>> mbox/%3CCAC3UcJvrgcBfe6%3DiW6%2BuTWsLB%2B4CsHgRmDx9TvCzJQrWvfg7_w%
>> > > > >>> 40mail.gmail.com%3E
>> > > > >>>
>> > > > >>> Cheers,
>> > > > >>> Jorge.
>> > > > >>>
>> > > > >>
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Jason Gustafson]

Thanks Arjun. I had one additional question. High level, I'm wondering if
it makes sense to treat processing errors such as serialization failures
the same as IO errors. In the former case, retrying typically doesn't help
because the processing is deterministic. In the latter case, the failure
may be downstream (e.g. a kafka partition may be temporarily unavailable).
As a user, I would probably want the option to skip over processing
failures, but retry indefinitely if the downstream system is unavailable.
Is that use case supported?

Thanks,
Jason



On Mon, May 21, 2018 at 12:39 PM, Arjun Satish 
wrote:

> Thanks a lot, Ewen! I'll make sure the documentation is clear on the
> differences between retries an tolerance.
>
> Do you think percentage would have the same problem as the one you brought
> up? Also, if we say 10% tolerance, do we have to wait for the duration to
> finish before failing the task, or should we fail as soon as we hit 10%
> error.
>
> Alternatively, do you think making tolerance an Enum would be simpler?
> Where it's values are NONE (any errors kill), ALL (tolerate all errors and
> skip records) and FIRST (tolerate the first error, but fail after that)?
>
> Best,
>
>
> On Mon, May 21, 2018 at 11:28 AM, Ewen Cheslack-Postava  >
> wrote:
>
> > Arjun,
> >
> > Understood on retries vs tolerance -- though I suspect this will end up
> > being a bit confusing to users as well. It's two levels of error handling
> > which is what tripped me up.
> >
> > One last comment on KIP (which otherwise looks good): for the tolerance
> > setting, do we want it to be an absolute value or something like a
> > percentage? Given the current way of setting things, I'm not sure I'd
> ever
> > set it to anything but -1 or 0, with maybe 1 as an easy option for
> > restarting a connector to get it past one bad message, then reverting
> back
> > to -1 or 0.
> >
> > -Ewen
> >
> > On Mon, May 21, 2018 at 11:01 AM Arjun Satish 
> > wrote:
> >
> > > Hey Jason,
> > >
> > > Thanks for your comments. Please find answers inline:
> > >
> > > On Mon, May 21, 2018 at 9:52 AM, Jason Gustafson 
> > > wrote:
> > >
> > > > Hi Arjun,
> > > >
> > > > Thanks for the KIP. Just a few comments/questions:
> > > >
> > > > 1. The proposal allows users to configure the number of retries. I
> > > usually
> > > > find it easier as a user to work with timeouts since it's difficult
> to
> > > know
> > > > how long a retry might take. Have you considered adding a timeout
> > option
> > > > which would retry until the timeout expires?
> > > >
> > >
> > > Good point. Updated the KIP.
> > >
> > > 2. The configs are named very generically (e.g. errors.retries.limit).
> Do
> > > > you think it will be clear to users what operations these configs
> apply
> > > to?
> > > >
> > >
> > > As of now, these configs are applicable to all operations in the
> > connector
> > > pipeline (as mentioned in the proposed changes section). We decided not
> > to
> > > have per operation limit because of the additional config complexity.
> > >
> > >
> > > > 3. I wasn't entirely clear what messages are stored in the dead
> letter
> > > > queue. It sounds like it includes both configs and messages since we
> > have
> > > > errors.dlq.include.configs? Is there a specific schema you have in
> > mind?
> > > >
> > >
> > > This has been addressed in the KIP. The DLQ will now contain only raw
> > > messages (no additional context). We are also supporting DLQs only for
> > sink
> > > connectors now.
> > >
> > >
> > > > 4. I didn't see it mentioned explicitly in the KIP, but I assume the
> > > > tolerance metrics are reset after every task rebalance?
> > > >
> > >
> > > Great question. Yes, we will reset the tolerance metrics on every
> > > rebalance.
> > >
> > >
> > > > 5. I wonder if we can do without errors.tolerance.limit. You can get
> a
> > > > similar effect using errors.tolerance.rate.limit if you allow longer
> > > > durations. I'm not sure how useful an absolute counter is in
> practice.
> > > >
> > >
> > > Yeah, the rate limit does subsume the features offered by the absolute
> > > counter. Removed it.
> > >
> > >
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-6927) Broker uses significant amount of memory during down-conversion

[Dhruvil Shah (JIRA)]

Dhruvil Shah created KAFKA-6927:
---

 Summary: Broker uses significant amount of memory during 
down-conversion
 Key: KAFKA-6927
 URL: https://issues.apache.org/jira/browse/KAFKA-6927
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Dhruvil Shah
Assignee: Dhruvil Shah
 Fix For: 2.0.0


Kafka broker could consume significant amount of memory when down-conversion is 
required. We have seen scenarios where this causes out of memory errors. This 
issue and the proposed fix is described in detail in KIP-283 - 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-283%3A+Efficient+Memory+Usage+for+Down-Conversion]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: kafka-trunk-jdk8 #2661

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Reduce required occurrance from 100 to 10 (#5048)

--
[...truncated 422.86 KB...]
kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotClearAnythingIfOffsetToFirstOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotLetOffsetsGoBackwardsEvenIfEpochsProgress PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldGetFirstOffsetOfSubsequentEpochWhenOffsetRequestedForPreviousEpoch PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest2 PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearEarliestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldPreserveResetOffsetOnClearEarliestIfOneExists PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldUpdateOffsetBetweenEpochBoundariesOnClearEarliest PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldReturnInvalidOffsetIfEpochIsRequestedWhichIsNotCurrentlyTracked PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldFetchEndOffsetOfEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldRetainLatestEpochOnClearAllEarliestAndUpdateItsOffset PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearAllEntries PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > shouldClearLatestOnEmptyCache 
PASSED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed STARTED

kafka.server.epoch.LeaderEpochFileCacheTest > 
shouldNotResetEpochHistoryHeadIfUndefinedPassed PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldIncreaseLeaderEpochBetweenLeaderRestarts PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldAddCurrentLeaderEpochToMessagesAsTheyAreWrittenToLeader PASSED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse STARTED

kafka.server.epoch.LeaderEpochIntegrationTest > 
shouldSendLeaderEpochRequestAndGetAResponse PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica 
STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > shouldGetEpochsFromReplica PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnUnknownTopicOrPartitionIfThrown PASSED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown STARTED

kafka.server.epoch.OffsetsForLeaderEpochTest > 
shouldReturnNoLeaderForPartitionIfThrown PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldSurviveFastLeaderChange PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
offsetsShouldNotGoBackwards PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldFollowLeaderEpochBasicWorkflow STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldFollowLeaderEpochBasicWorkflow PASSED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldNotAllowDivergentLogs STARTED

kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest > 
shouldNotAllowDivergentLogs PASSED

Re: [VOTE] KIP-290: Support for wildcard suffixed ACLs

[Piyush Vijay]

Thanks for the +1s. I'll send out a PR shortly.

Thanks


Piyush Vijay

On Mon, May 21, 2018 at 8:59 AM, Guozhang Wang  wrote:

> Thanks for the KIP, +1 from me (binding).
>
>
> Guozhang
>
> On Mon, May 21, 2018 at 5:22 AM, Damian Guy  wrote:
>
> > +1 (binding)
> >
> > On Sat, 19 May 2018 at 03:51 Piyush Vijay 
> wrote:
> >
> > > Hi everyone,
> > >
> > > I would like to start a vote for
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 290%3A+Support+for+wildcard+suffixed+ACLs
> > > .
> > >
> > > The KIP proposes a way to support wildcard-suffixed resource names in
> > Kafka
> > > ACLs.
> > >
> > > The main challenge was to support it in a backward compatible way
> because
> > > resources like consumer groups don't have defined naming convention and
> > can
> > > have '*' in their names.
> > >
> > > Please take a look.
> > >
> > > Thanks
> > >
> > > Piyush Vijay
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-298: Error Handling in Connect kafka

[Arjun Satish]

All,

Thanks so much for your feedback on this KIP. I've made some small
modifications today. I'll wait till midnight today (PDT) to close this
vote. Please let me know if there are any further comments.

Best,

On Mon, May 21, 2018 at 11:29 AM, Ewen Cheslack-Postava 
wrote:

> +1 binding. I had one last comment in the DISCUSS thread, but not really a
> blocker.
>
> -Ewen
>
> On Mon, May 21, 2018 at 9:48 AM Matthias J. Sax 
> wrote:
>
> > +1 (binding)
> >
> >
> >
> > On 5/21/18 9:30 AM, Randall Hauch wrote:
> > > Thanks, Arjun. +1 (non-binding)
> > >
> > > Regards,
> > > Randall
> > >
> > > On Mon, May 21, 2018 at 11:14 AM, Guozhang Wang 
> > wrote:
> > >
> > >> Thanks for the KIP. +1 (binding)
> > >>
> > >>
> > >> Guozhang
> > >>
> > >> On Fri, May 18, 2018 at 3:36 PM, Gwen Shapira 
> > wrote:
> > >>
> > >>> +1
> > >>>
> > >>> Thank you! Error handling in Connect will be a huge improvement.
> > >>>
> > >>> On Thu, May 17, 2018, 1:58 AM Arjun Satish 
> > >> wrote:
> > >>>
> >  All,
> > 
> >  Many thanks for all the feedback on KIP-298. Highly appreciate the
> > time
> > >>> and
> >  effort you all put into it.
> > 
> >  I've updated the KIP accordingly, and would like to start to start a
> > >> vote
> >  on it.
> > 
> >  KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >  298%3A+Error+Handling+in+Connect
> >  JIRA: https://issues.apache.org/jira/browse/KAFKA-6738
> >  Discussion Thread: https://www.mail-archive.com/
> >  dev@kafka.apache.org/msg87660.html
> > 
> >  Thanks very much!
> > 
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >>
> > >
> >
> >
>

Re: [DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder

[Liquan Pei]

This KIP makes sharing a WindowedKTable among Kafka Stream jobs very easy.
It would be nice to get this into trunk soon.

Best,
Liquan

On Mon, May 21, 2018 at 12:25 PM, Boyang Chen  wrote:

> Hey all,
>
>
> I would like to start a discussion thread on KIP 300, which introduces a
> new API called windowedTable() in StreamsBuilder:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 300%3A+Add+Windowed+KTable+API+in+StreamsBuilder
>
>
> The pull request I'm working on is here: https://github.com/apache/
> kafka/pull/5044
>
>
> I understood that the community is busy working on 2.0 release, but this
> KIP is really important for our internal use case. So if any of you got
> time, please focus on clarifying the use case and reaching the agreement of
> API. Really appreciate your time!
>
>
> Best,
>
> Boyang
>
>
>


-- 
Liquan Pei
Software Engineer, Confluent Inc

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Arjun Satish]

OK. Let's simplify tolerance to simply have NONE or ALL values. For
extensions, we can open a KIP and implement in later versions.

Thanks a lot!

On Mon, May 21, 2018 at 1:18 PM, Ewen Cheslack-Postava 
wrote:

> On Mon, May 21, 2018 at 12:39 PM Arjun Satish 
> wrote:
>
> > Thanks a lot, Ewen! I'll make sure the documentation is clear on the
> > differences between retries an tolerance.
> >
> > Do you think percentage would have the same problem as the one you
> brought
> > up? Also, if we say 10% tolerance, do we have to wait for the duration to
> > finish before failing the task, or should we fail as soon as we hit 10%
> > error.
> >
> Yeah, percent might not be right either. I'd probably only reasonably set
> it to 0% or 100% as something in between seems difficult to justify.
>
>
> >
> > Alternatively, do you think making tolerance an Enum would be simpler?
> > Where it's values are NONE (any errors kill), ALL (tolerate all errors
> and
> > skip records) and FIRST (tolerate the first error, but fail after that)?
> >
>
> I do think the values I would ever use are limited enough to just be an
> enum. Not sure if anyone has use cases for larger positive values.
>
> -Ewen
>
>
> >
> > Best,
> >
> >
> > On Mon, May 21, 2018 at 11:28 AM, Ewen Cheslack-Postava <
> e...@confluent.io
> > >
> > wrote:
> >
> > > Arjun,
> > >
> > > Understood on retries vs tolerance -- though I suspect this will end up
> > > being a bit confusing to users as well. It's two levels of error
> handling
> > > which is what tripped me up.
> > >
> > > One last comment on KIP (which otherwise looks good): for the tolerance
> > > setting, do we want it to be an absolute value or something like a
> > > percentage? Given the current way of setting things, I'm not sure I'd
> > ever
> > > set it to anything but -1 or 0, with maybe 1 as an easy option for
> > > restarting a connector to get it past one bad message, then reverting
> > back
> > > to -1 or 0.
> > >
> > > -Ewen
> > >
> > > On Mon, May 21, 2018 at 11:01 AM Arjun Satish 
> > > wrote:
> > >
> > > > Hey Jason,
> > > >
> > > > Thanks for your comments. Please find answers inline:
> > > >
> > > > On Mon, May 21, 2018 at 9:52 AM, Jason Gustafson  >
> > > > wrote:
> > > >
> > > > > Hi Arjun,
> > > > >
> > > > > Thanks for the KIP. Just a few comments/questions:
> > > > >
> > > > > 1. The proposal allows users to configure the number of retries. I
> > > > usually
> > > > > find it easier as a user to work with timeouts since it's difficult
> > to
> > > > know
> > > > > how long a retry might take. Have you considered adding a timeout
> > > option
> > > > > which would retry until the timeout expires?
> > > > >
> > > >
> > > > Good point. Updated the KIP.
> > > >
> > > > 2. The configs are named very generically (e.g.
> errors.retries.limit).
> > Do
> > > > > you think it will be clear to users what operations these configs
> > apply
> > > > to?
> > > > >
> > > >
> > > > As of now, these configs are applicable to all operations in the
> > > connector
> > > > pipeline (as mentioned in the proposed changes section). We decided
> not
> > > to
> > > > have per operation limit because of the additional config complexity.
> > > >
> > > >
> > > > > 3. I wasn't entirely clear what messages are stored in the dead
> > letter
> > > > > queue. It sounds like it includes both configs and messages since
> we
> > > have
> > > > > errors.dlq.include.configs? Is there a specific schema you have in
> > > mind?
> > > > >
> > > >
> > > > This has been addressed in the KIP. The DLQ will now contain only raw
> > > > messages (no additional context). We are also supporting DLQs only
> for
> > > sink
> > > > connectors now.
> > > >
> > > >
> > > > > 4. I didn't see it mentioned explicitly in the KIP, but I assume
> the
> > > > > tolerance metrics are reset after every task rebalance?
> > > > >
> > > >
> > > > Great question. Yes, we will reset the tolerance metrics on every
> > > > rebalance.
> > > >
> > > >
> > > > > 5. I wonder if we can do without errors.tolerance.limit. You can
> get
> > a
> > > > > similar effect using errors.tolerance.rate.limit if you allow
> longer
> > > > > durations. I'm not sure how useful an absolute counter is in
> > practice.
> > > > >
> > > >
> > > > Yeah, the rate limit does subsume the features offered by the
> absolute
> > > > counter. Removed it.
> > > >
> > > >
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Ewen Cheslack-Postava]

On Mon, May 21, 2018 at 12:39 PM Arjun Satish 
wrote:

> Thanks a lot, Ewen! I'll make sure the documentation is clear on the
> differences between retries an tolerance.
>
> Do you think percentage would have the same problem as the one you brought
> up? Also, if we say 10% tolerance, do we have to wait for the duration to
> finish before failing the task, or should we fail as soon as we hit 10%
> error.
>
Yeah, percent might not be right either. I'd probably only reasonably set
it to 0% or 100% as something in between seems difficult to justify.


>
> Alternatively, do you think making tolerance an Enum would be simpler?
> Where it's values are NONE (any errors kill), ALL (tolerate all errors and
> skip records) and FIRST (tolerate the first error, but fail after that)?
>

I do think the values I would ever use are limited enough to just be an
enum. Not sure if anyone has use cases for larger positive values.

-Ewen


>
> Best,
>
>
> On Mon, May 21, 2018 at 11:28 AM, Ewen Cheslack-Postava  >
> wrote:
>
> > Arjun,
> >
> > Understood on retries vs tolerance -- though I suspect this will end up
> > being a bit confusing to users as well. It's two levels of error handling
> > which is what tripped me up.
> >
> > One last comment on KIP (which otherwise looks good): for the tolerance
> > setting, do we want it to be an absolute value or something like a
> > percentage? Given the current way of setting things, I'm not sure I'd
> ever
> > set it to anything but -1 or 0, with maybe 1 as an easy option for
> > restarting a connector to get it past one bad message, then reverting
> back
> > to -1 or 0.
> >
> > -Ewen
> >
> > On Mon, May 21, 2018 at 11:01 AM Arjun Satish 
> > wrote:
> >
> > > Hey Jason,
> > >
> > > Thanks for your comments. Please find answers inline:
> > >
> > > On Mon, May 21, 2018 at 9:52 AM, Jason Gustafson 
> > > wrote:
> > >
> > > > Hi Arjun,
> > > >
> > > > Thanks for the KIP. Just a few comments/questions:
> > > >
> > > > 1. The proposal allows users to configure the number of retries. I
> > > usually
> > > > find it easier as a user to work with timeouts since it's difficult
> to
> > > know
> > > > how long a retry might take. Have you considered adding a timeout
> > option
> > > > which would retry until the timeout expires?
> > > >
> > >
> > > Good point. Updated the KIP.
> > >
> > > 2. The configs are named very generically (e.g. errors.retries.limit).
> Do
> > > > you think it will be clear to users what operations these configs
> apply
> > > to?
> > > >
> > >
> > > As of now, these configs are applicable to all operations in the
> > connector
> > > pipeline (as mentioned in the proposed changes section). We decided not
> > to
> > > have per operation limit because of the additional config complexity.
> > >
> > >
> > > > 3. I wasn't entirely clear what messages are stored in the dead
> letter
> > > > queue. It sounds like it includes both configs and messages since we
> > have
> > > > errors.dlq.include.configs? Is there a specific schema you have in
> > mind?
> > > >
> > >
> > > This has been addressed in the KIP. The DLQ will now contain only raw
> > > messages (no additional context). We are also supporting DLQs only for
> > sink
> > > connectors now.
> > >
> > >
> > > > 4. I didn't see it mentioned explicitly in the KIP, but I assume the
> > > > tolerance metrics are reset after every task rebalance?
> > > >
> > >
> > > Great question. Yes, we will reset the tolerance metrics on every
> > > rebalance.
> > >
> > >
> > > > 5. I wonder if we can do without errors.tolerance.limit. You can get
> a
> > > > similar effect using errors.tolerance.rate.limit if you allow longer
> > > > durations. I'm not sure how useful an absolute counter is in
> practice.
> > > >
> > >
> > > Yeah, the rate limit does subsume the features offered by the absolute
> > > counter. Removed it.
> > >
> > >
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Arjun Satish]

Thanks a lot, Ewen! I'll make sure the documentation is clear on the
differences between retries an tolerance.

Do you think percentage would have the same problem as the one you brought
up? Also, if we say 10% tolerance, do we have to wait for the duration to
finish before failing the task, or should we fail as soon as we hit 10%
error.

Alternatively, do you think making tolerance an Enum would be simpler?
Where it's values are NONE (any errors kill), ALL (tolerate all errors and
skip records) and FIRST (tolerate the first error, but fail after that)?

Best,


On Mon, May 21, 2018 at 11:28 AM, Ewen Cheslack-Postava 
wrote:

> Arjun,
>
> Understood on retries vs tolerance -- though I suspect this will end up
> being a bit confusing to users as well. It's two levels of error handling
> which is what tripped me up.
>
> One last comment on KIP (which otherwise looks good): for the tolerance
> setting, do we want it to be an absolute value or something like a
> percentage? Given the current way of setting things, I'm not sure I'd ever
> set it to anything but -1 or 0, with maybe 1 as an easy option for
> restarting a connector to get it past one bad message, then reverting back
> to -1 or 0.
>
> -Ewen
>
> On Mon, May 21, 2018 at 11:01 AM Arjun Satish 
> wrote:
>
> > Hey Jason,
> >
> > Thanks for your comments. Please find answers inline:
> >
> > On Mon, May 21, 2018 at 9:52 AM, Jason Gustafson 
> > wrote:
> >
> > > Hi Arjun,
> > >
> > > Thanks for the KIP. Just a few comments/questions:
> > >
> > > 1. The proposal allows users to configure the number of retries. I
> > usually
> > > find it easier as a user to work with timeouts since it's difficult to
> > know
> > > how long a retry might take. Have you considered adding a timeout
> option
> > > which would retry until the timeout expires?
> > >
> >
> > Good point. Updated the KIP.
> >
> > 2. The configs are named very generically (e.g. errors.retries.limit). Do
> > > you think it will be clear to users what operations these configs apply
> > to?
> > >
> >
> > As of now, these configs are applicable to all operations in the
> connector
> > pipeline (as mentioned in the proposed changes section). We decided not
> to
> > have per operation limit because of the additional config complexity.
> >
> >
> > > 3. I wasn't entirely clear what messages are stored in the dead letter
> > > queue. It sounds like it includes both configs and messages since we
> have
> > > errors.dlq.include.configs? Is there a specific schema you have in
> mind?
> > >
> >
> > This has been addressed in the KIP. The DLQ will now contain only raw
> > messages (no additional context). We are also supporting DLQs only for
> sink
> > connectors now.
> >
> >
> > > 4. I didn't see it mentioned explicitly in the KIP, but I assume the
> > > tolerance metrics are reset after every task rebalance?
> > >
> >
> > Great question. Yes, we will reset the tolerance metrics on every
> > rebalance.
> >
> >
> > > 5. I wonder if we can do without errors.tolerance.limit. You can get a
> > > similar effect using errors.tolerance.rate.limit if you allow longer
> > > durations. I'm not sure how useful an absolute counter is in practice.
> > >
> >
> > Yeah, the rate limit does subsume the features offered by the absolute
> > counter. Removed it.
> >
> >
> > >
> > > Thanks,
> > > Jason
> > >
> > >
> >
>

[DISCUSSION] KIP-300: Add Windowed KTable API in StreamsBuilder

[Boyang Chen]

Hey all,


I would like to start a discussion thread on KIP 300, which introduces a new 
API called windowedTable() in StreamsBuilder:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-300%3A+Add+Windowed+KTable+API+in+StreamsBuilder


The pull request I'm working on is here: 
https://github.com/apache/kafka/pull/5044


I understood that the community is busy working on 2.0 release, but this KIP is 
really important for our internal use case. So if any of you got time, please 
focus on clarifying the use case and reaching the agreement of API. Really 
appreciate your time!


Best,

Boyang

[jira] [Resolved] (KAFKA-3665) Default ssl.endpoint.identification.algorithm should be https

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3665.

Resolution: Fixed

> Default ssl.endpoint.identification.algorithm should be https
> -
>
> Key: KAFKA-3665
> URL: https://issues.apache.org/jira/browse/KAFKA-3665
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Ismael Juma
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.0.0
>
>
> The default `ssl.endpoint.identification.algorithm` is `null` which is not a 
> secure default (man in the middle attacks are possible).
> We should probably use `https` instead. A more conservative alternative would 
> be to update the documentation instead of changing the default.
> A paper on the topic (thanks to Ryan Pridgeon for the reference): 
> http://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (KAFKA-3665) Default ssl.endpoint.identification.algorithm should be https

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reopened KAFKA-3665:


> Default ssl.endpoint.identification.algorithm should be https
> -
>
> Key: KAFKA-3665
> URL: https://issues.apache.org/jira/browse/KAFKA-3665
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Ismael Juma
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.0.0
>
>
> The default `ssl.endpoint.identification.algorithm` is `null` which is not a 
> secure default (man in the middle attacks are possible).
> We should probably use `https` instead. A more conservative alternative would 
> be to update the documentation instead of changing the default.
> A paper on the topic (thanks to Ryan Pridgeon for the reference): 
> http://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Matthias J. Sax]

I think that the risk of the change is moderate as I expect most people
to use the DefaultStreamPartitioner.

However, there would still be possibility to define a new interface
instead of changing the old:

> public interface DynamicStreamPartitioner {
> Integer partition(String topic, K key, V value, int numPartitions);
> }

The newly added methods `Topology#addSink` and `KStream#to` would take
this new interface instead of the old.

Maybe `DynamicStreamPartitioner` must extend `StreamPartitioner` to make
runtime code work though...

WDYT?

-Matthias

On 5/21/18 11:47 AM, Guozhang Wang wrote:
> Hello everyone,
> 
> While implementing the PR for this KIP I realized there is once place which
> we should consider modifying on public APIs as well:
> StreamPartitioner#partition, to add the topic name string. Note it will be
> a incompatible change that requires users who have customized
> StreamPartitioner implementations.
> 
> I've updated the wiki page of KIP-303, please recast your vote on this
> thread. Thanks!!!
> 
> 
> Guozhang
> 
> 
> On Thu, May 17, 2018 at 3:15 PM, John Roesler  wrote:
> 
>> +1 non-binding
>>
>> On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax 
>> wrote:
>>
>>> +1 (binding)
>>>
>>>
>>> On 5/17/18 12:18 PM, Ted Yu wrote:
 +1
  Original message From: Gwen Shapira <
>> g...@confluent.io>
>>> Date: 5/17/18  11:53 AM  (GMT-08:00) To: dev 
>>> Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka
>> Streams'
>>> Topology Sink
 Yay, its about time :)

 +1

 On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang 
>>> wrote:

> Hello folks,
>
> I'd like to start a voting thread on adding dynamic routing
>>> functionality
> in Streams sink node. Please find a KIP here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 303%3A+Add+Dynamic+Routing+in+Streams+Sink
>
>
> And the PR itself ready for review as well under KAFKA-4936:
>
> https://github.com/apache/kafka/pull/5018
>
>
>
> Thanks!
> -- Guozhang
>



>>>
>>>
>>
> 
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

[Guozhang Wang]

Hello everyone,

While implementing the PR for this KIP I realized there is once place which
we should consider modifying on public APIs as well:
StreamPartitioner#partition, to add the topic name string. Note it will be
a incompatible change that requires users who have customized
StreamPartitioner implementations.

I've updated the wiki page of KIP-303, please recast your vote on this
thread. Thanks!!!


Guozhang


On Thu, May 17, 2018 at 3:15 PM, John Roesler  wrote:

> +1 non-binding
>
> On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax 
> wrote:
>
> > +1 (binding)
> >
> >
> > On 5/17/18 12:18 PM, Ted Yu wrote:
> > > +1
> > >  Original message From: Gwen Shapira <
> g...@confluent.io>
> > Date: 5/17/18  11:53 AM  (GMT-08:00) To: dev 
> > Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka
> Streams'
> > Topology Sink
> > > Yay, its about time :)
> > >
> > > +1
> > >
> > > On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang 
> > wrote:
> > >
> > >> Hello folks,
> > >>
> > >> I'd like to start a voting thread on adding dynamic routing
> > functionality
> > >> in Streams sink node. Please find a KIP here:
> > >>
> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >> 303%3A+Add+Dynamic+Routing+in+Streams+Sink
> > >>
> > >>
> > >> And the PR itself ready for review as well under KAFKA-4936:
> > >>
> > >> https://github.com/apache/kafka/pull/5018
> > >>
> > >>
> > >>
> > >> Thanks!
> > >> -- Guozhang
> > >>
> > >
> > >
> > >
> >
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-298: Error Handling in Connect kafka

[Ewen Cheslack-Postava]

+1 binding. I had one last comment in the DISCUSS thread, but not really a
blocker.

-Ewen

On Mon, May 21, 2018 at 9:48 AM Matthias J. Sax 
wrote:

> +1 (binding)
>
>
>
> On 5/21/18 9:30 AM, Randall Hauch wrote:
> > Thanks, Arjun. +1 (non-binding)
> >
> > Regards,
> > Randall
> >
> > On Mon, May 21, 2018 at 11:14 AM, Guozhang Wang 
> wrote:
> >
> >> Thanks for the KIP. +1 (binding)
> >>
> >>
> >> Guozhang
> >>
> >> On Fri, May 18, 2018 at 3:36 PM, Gwen Shapira 
> wrote:
> >>
> >>> +1
> >>>
> >>> Thank you! Error handling in Connect will be a huge improvement.
> >>>
> >>> On Thu, May 17, 2018, 1:58 AM Arjun Satish 
> >> wrote:
> >>>
>  All,
> 
>  Many thanks for all the feedback on KIP-298. Highly appreciate the
> time
> >>> and
>  effort you all put into it.
> 
>  I've updated the KIP accordingly, and would like to start to start a
> >> vote
>  on it.
> 
>  KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>  298%3A+Error+Handling+in+Connect
>  JIRA: https://issues.apache.org/jira/browse/KAFKA-6738
>  Discussion Thread: https://www.mail-archive.com/
>  dev@kafka.apache.org/msg87660.html
> 
>  Thanks very much!
> 
> >>>
> >>
> >>
> >>
> >> --
> >> -- Guozhang
> >>
> >
>
>

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Ewen Cheslack-Postava]

Arjun,

Understood on retries vs tolerance -- though I suspect this will end up
being a bit confusing to users as well. It's two levels of error handling
which is what tripped me up.

One last comment on KIP (which otherwise looks good): for the tolerance
setting, do we want it to be an absolute value or something like a
percentage? Given the current way of setting things, I'm not sure I'd ever
set it to anything but -1 or 0, with maybe 1 as an easy option for
restarting a connector to get it past one bad message, then reverting back
to -1 or 0.

-Ewen

On Mon, May 21, 2018 at 11:01 AM Arjun Satish 
wrote:

> Hey Jason,
>
> Thanks for your comments. Please find answers inline:
>
> On Mon, May 21, 2018 at 9:52 AM, Jason Gustafson 
> wrote:
>
> > Hi Arjun,
> >
> > Thanks for the KIP. Just a few comments/questions:
> >
> > 1. The proposal allows users to configure the number of retries. I
> usually
> > find it easier as a user to work with timeouts since it's difficult to
> know
> > how long a retry might take. Have you considered adding a timeout option
> > which would retry until the timeout expires?
> >
>
> Good point. Updated the KIP.
>
> 2. The configs are named very generically (e.g. errors.retries.limit). Do
> > you think it will be clear to users what operations these configs apply
> to?
> >
>
> As of now, these configs are applicable to all operations in the connector
> pipeline (as mentioned in the proposed changes section). We decided not to
> have per operation limit because of the additional config complexity.
>
>
> > 3. I wasn't entirely clear what messages are stored in the dead letter
> > queue. It sounds like it includes both configs and messages since we have
> > errors.dlq.include.configs? Is there a specific schema you have in mind?
> >
>
> This has been addressed in the KIP. The DLQ will now contain only raw
> messages (no additional context). We are also supporting DLQs only for sink
> connectors now.
>
>
> > 4. I didn't see it mentioned explicitly in the KIP, but I assume the
> > tolerance metrics are reset after every task rebalance?
> >
>
> Great question. Yes, we will reset the tolerance metrics on every
> rebalance.
>
>
> > 5. I wonder if we can do without errors.tolerance.limit. You can get a
> > similar effect using errors.tolerance.rate.limit if you allow longer
> > durations. I'm not sure how useful an absolute counter is in practice.
> >
>
> Yeah, the rate limit does subsume the features offered by the absolute
> counter. Removed it.
>
>
> >
> > Thanks,
> > Jason
> >
> >
>

Re: [VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Bill Bejeck]

Thanks for the KIP.

+1

-Bill

On Mon, May 21, 2018 at 1:41 PM, Ted Yu  wrote:

> +1
>
> On Mon, May 21, 2018 at 9:02 AM, Guozhang Wang  wrote:
>
> > Thanks Dhruvil for the KIP, +1 from me (binding).
> >
> >
> > Guozhang
> >
> > On Mon, May 21, 2018 at 7:52 AM, Dhruvil Shah 
> > wrote:
> >
> > > Hi,
> > >
> > > I would like to start a vote on KIP-306 which proposes to add a
> > > configuration to delay responses to failed authentication.
> > >
> > > Link to the KIP:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 306%3A+Configuration+for+Delaying+Response+to+Failed+
> > Client+Authentication
> > >
> > > Thanks,
> > > Dhruvil
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Arjun Satish]

Hey Jason,

Thanks for your comments. Please find answers inline:

On Mon, May 21, 2018 at 9:52 AM, Jason Gustafson  wrote:

> Hi Arjun,
>
> Thanks for the KIP. Just a few comments/questions:
>
> 1. The proposal allows users to configure the number of retries. I usually
> find it easier as a user to work with timeouts since it's difficult to know
> how long a retry might take. Have you considered adding a timeout option
> which would retry until the timeout expires?
>

Good point. Updated the KIP.

2. The configs are named very generically (e.g. errors.retries.limit). Do
> you think it will be clear to users what operations these configs apply to?
>

As of now, these configs are applicable to all operations in the connector
pipeline (as mentioned in the proposed changes section). We decided not to
have per operation limit because of the additional config complexity.


> 3. I wasn't entirely clear what messages are stored in the dead letter
> queue. It sounds like it includes both configs and messages since we have
> errors.dlq.include.configs? Is there a specific schema you have in mind?
>

This has been addressed in the KIP. The DLQ will now contain only raw
messages (no additional context). We are also supporting DLQs only for sink
connectors now.


> 4. I didn't see it mentioned explicitly in the KIP, but I assume the
> tolerance metrics are reset after every task rebalance?
>

Great question. Yes, we will reset the tolerance metrics on every
rebalance.


> 5. I wonder if we can do without errors.tolerance.limit. You can get a
> similar effect using errors.tolerance.rate.limit if you allow longer
> durations. I'm not sure how useful an absolute counter is in practice.
>

Yeah, the rate limit does subsume the features offered by the absolute
counter. Removed it.


>
> Thanks,
> Jason
>
>

Re: [VOTE] KIP-285: Connect Rest Extension Plugin

[Magesh Nandakumar]

Thanks a lot Guozhang. I have updated the KIP to reflect that minor change.

On Mon, May 21, 2018 at 9:08 AM, Guozhang Wang  wrote:

> Thanks for the KIP, +1 from me (binding)
>
> One minor suggestion: in "class TaskState" constructor, the passed in
> parameter of `worker` could be `workerId`, to be consistent with others in
> ConnectorState?
>
>
> Guozhang
>
>
> On Fri, May 18, 2018 at 9:44 AM, Randall Hauch  wrote:
>
> > +1 (non-binding)
> >
> > Regards,
> > Randall
> >
> > On Fri, May 18, 2018 at 11:32 AM, Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > +1 (non-binding)
> > >
> > > - Konstantine
> > >
> > > On Thu, May 17, 2018 at 10:05 PM, Ewen Cheslack-Postava <
> > e...@confluent.io
> > > >
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Ewen
> > > >
> > > > On Thu, May 17, 2018 at 12:16 PM Ted Yu  wrote:
> > > >
> > > > > +1
> > > > >  Original message From: Gwen Shapira <
> > > g...@confluent.io>
> > > > > Date: 5/17/18  12:02 PM  (GMT-08:00) To: dev  >
> > > > > Subject: Re: [VOTE] KIP-285: Connect Rest Extension Plugin
> > > > > LGTM. +1.
> > > > >
> > > > > On Wed, May 16, 2018 at 8:19 PM, Magesh Nandakumar <
> > > mage...@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > After a good round of discussions with excellent feedback and no
> > > major
> > > > > > objections, I would like to start a vote on KIP-285: Connect Rest
> > > > > Extension
> > > > > > Plugin.
> > > > > >
> > > > > > KIP: <
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 285%3A+Connect+Rest+Extension+Plugin
> > > > > > >
> > > > > >
> > > > > >
> > > > > > JIRA: <*https://issues.apache.org/jira/browse/KAFKA-6776
> > > > > > *>
> > > > > >
> > > > > > Discussion thread: <
> > > > > > https://www.mail-archive.com/dev@kafka.apache.org/msg86910.html>
> > > > > >
> > > > > > Thanks,
> > > > > > Magesh
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *Gwen Shapira*
> > > > > Product Manager | Confluent
> > > > > 650.450.2760 | @gwenshap
> > > > > Follow us: Twitter  | blog
> > > > > 
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

Build failed in Jenkins: kafka-trunk-jdk8 #2660

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-6868; Fix buffer underflow and expose group state in the consumer

--
[...truncated 423.95 KB...]
kafka.zookeeper.ZooKeeperClientTest > testPipelinedGetData STARTED

kafka.zookeeper.ZooKeeperClientTest > testPipelinedGetData PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChildChangeHandlerForChildChange 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren 
PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline STARTED

kafka.zookeeper.ZooKeeperClientTest > testMixedPipeline PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetDataExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry STARTED

kafka.zookeeper.ZooKeeperClientTest > testSessionExpiry PASSED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testSetDataNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testDeleteNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testExistsExistingZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testZooKeeperStateChangeRateMetrics PASSED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion STARTED

kafka.zookeeper.ZooKeeperClientTest > testZNodeChangeHandlerForDeletion PASSED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode STARTED

kafka.zookeeper.ZooKeeperClientTest > testGetAclNonExistentZNode PASSED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
STARTED

kafka.zookeeper.ZooKeeperClientTest > testStateChangeHandlerForAuthFailure 
PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > controlThrowable STARTED

kafka.network.SocketServerTest > controlThrowable PASSED

kafka.network.SocketServerTest > testRequestMetricsAfterStop STARTED

kafka.network.SocketServerTest > testRequestMetricsAfterStop PASSED

kafka.network.SocketServerTest > testConnectionIdReuse STARTED

kafka.network.SocketServerTest > testConnectionIdReuse PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testConnectionId STARTED

kafka.network.SocketServerTest > testConnectionId PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > testNoOpAction STARTED

kafka.network.SocketServerTest > testNoOpAction PASSED

kafka.network.SocketServerTest > simpleRequest STARTED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > closingChannelException STARTED

kafka.network.SocketServerTest > closingChannelException PASSED

kafka.network.SocketServerTest > testIdleConnection STARTED

kafka.network.SocketServerTest > testIdleConnection PASSED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed STARTED

kafka.network.SocketServerTest > 
testClientDisconnectionWithStagedReceivesFullyProcessed PASSED

kafka.network.SocketServerTest > testZeroMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testZeroMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown STARTED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown PASSED

kafka.network.SocketServerTest > testSessionPrincipal STARTED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > configureNewConnectionException STARTED

kafka.network.SocketServerTest > configureNewConnectionException PASSED

Re: [VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Ted Yu]

+1

On Mon, May 21, 2018 at 9:02 AM, Guozhang Wang  wrote:

> Thanks Dhruvil for the KIP, +1 from me (binding).
>
>
> Guozhang
>
> On Mon, May 21, 2018 at 7:52 AM, Dhruvil Shah 
> wrote:
>
> > Hi,
> >
> > I would like to start a vote on KIP-306 which proposes to add a
> > configuration to delay responses to failed authentication.
> >
> > Link to the KIP:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 306%3A+Configuration+for+Delaying+Response+to+Failed+
> Client+Authentication
> >
> > Thanks,
> > Dhruvil
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

[Ismael Juma]

Thanks for the KIP, +1 (binding). Can you also please describe the
compatibility impact of changing the error code from
CLUSTER_AUTHORIZATION_FAILED to TOPIC_AUTHORIZATION_FAILED?

Ismael

On Wed, Apr 25, 2018 at 2:45 AM Edoardo Comar  wrote:

> Hi,
>
> The discuss thread on KIP-277 (
> https://www.mail-archive.com/dev@kafka.apache.org/msg86540.html )
> seems to have been fruitful and concerns have been addressed, please allow
> me start a vote on it:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+Fine+Grained+ACL+for+CreateTopics+API
>
> I will update the small PR to the latest KIP semantics if the vote passes
> (as I hope :-).
>
> cheers
> Edo
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

[jira] [Resolved] (KAFKA-6873) Broker is not returning data including requested offset

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-6873.

Resolution: Not A Problem

> Broker is not returning data including requested offset
> ---
>
> Key: KAFKA-6873
> URL: https://issues.apache.org/jira/browse/KAFKA-6873
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 1.1.0
> Environment: ubuntu
>Reporter: Adam Dratwinski
>Priority: Minor
>
> After upgrading Kafka to 1.1.0 from 0.9.x I experience issues related with 
> broker returning incomplete responses. This happens for my all log compacted 
> topics. I am using Golang client (Sarama).
> I debugged the issue and found that for some requests brokers return 
> FetchResponse with all messages having offsets lower then requested. For 
> example, I request for offset 1078831, I get FetchResponse with only one 
> message having offset 1078830, which produces missing blocks error. If I 
> request the next offset (1078832), then I get a block with many messages, 
> starting with much higher offset (e.g 1083813). There is a big gap in offsets 
> between these records, probably because I am using log compacted topics, but 
> all expected messages are there.
> Sarama client treats this as consumer error:
> {quote}kafka: response did not contain all the expected topic/partition blocks
> {quote}
> For build-in java client this issue is not happening. Looks like it is less 
> restrict regarding the data order, and when the offset is missing in the 
> returned block, it just simply request the next offset.
> I reported this issue at Shopify/sarama Github project (see 
> [https://github.com/Shopify/sarama/issues/1087)], where I got response, that 
> this seems to be Kafka bug, as according to the documentation, in this 
> situation broker should never return only messages having lower offsets.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-242: Mask password fields in Kafka Connect REST response

[Randall Hauch]

See also
https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations,
which just passed.

On Mon, Mar 19, 2018 at 11:16 PM, Ewen Cheslack-Postava 
wrote:

> SSL authentication was added in KIP-208, which will be included in Kafka
> 1.1.0:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
>
> Connect isn't much different from the core Kafka/client configs currently
> where in some security setups you need to pass in passwords directly, and
> since there's various dynamic broker config improvements in the works, the
> fact that Connect exposes these in a REST API doesn't make it any
> different. I think the real long term solution to this is to add pluggable
> password support where you could, e.g., get these values out of a separate
> secrets management system instead of specifying them directly. Masking
> passwords as described in this solution feels like it's more of a temporary
> workaround and in order to be able to edit and update these connector
> configs by working with the REST API, we'd have to address these issues
> anyway.
>
> -Ewen
>
> On Mon, Mar 19, 2018 at 2:33 PM, Matt Farmer  wrote:
>
> > What’s the status of this? This is a pretty hard blocker for us to meet
> > requirements internally to deploy connect in a distributed fashion.
> >
> > @Ewen - Regarding the concern of accessing information securely - has
> > there been any consideration of adding authentication to the connect api?
> >
> > > On Jan 17, 2018, at 3:55 PM, Randall Hauch  wrote:
> > >
> > > Vincent,
> > >
> > > Can the KIP more explicitly say that this is opt-in, and that by
> default
> > > nothing will change?
> > >
> > > Randall
> > >
> > > On Tue, Jan 16, 2018 at 11:18 PM, Ewen Cheslack-Postava <
> > e...@confluent.io>
> > > wrote:
> > >
> > >> Vincent,
> > >>
> > >> I think with the addition of a configuration to control this for
> > >> compatibility, people would generally be ok with it. If you want to
> > start a
> > >> VOTE thread, the KIP deadline is coming up and the PR looks pretty
> > small. I
> > >> will take a pass at reviewing the PR so we'll be ready to merge if we
> > can
> > >> get the KIP voted through.
> > >>
> > >> Thanks,
> > >> Ewen
> > >>
> > >> On Fri, Jan 12, 2018 at 10:18 AM, Vincent Meng 
> wrote:
> > >>
> > >>> @Ted: The issue is kinda hard to reproduce. It's just something we
> > >> observe
> > >>> over time.
> > >>>
> > >>> @Ewen: I agree. Opt-in seems to be a good solution to me. To your
> > >> question,
> > >>> if there is no ConfDef that defines which fields are Passwords we can
> > >> just
> > >>> return the config as is.
> > >>>
> > >>> There is a PR for this KIP already. Comments/Discussions are welcome.
> > >>> https://github.com/apache/kafka/pull/4269
> > >>>
> > >>> On Tue, Jan 2, 2018 at 8:52 PM, Ewen Cheslack-Postava <
> > e...@confluent.io
> > >>>
> > >>> wrote:
> > >>>
> >  Vincent,
> > 
> >  Thanks for the KIP. This is definitely an issue we know is a problem
> > >> for
> >  some users.
> > 
> >  I think the major problem with the KIP as-is is that it makes it
> > >>> impossible
> >  to get the original value back out of the API. This KIP probably
> ties
> > >> in
> >  significantly with ideas for securing the REST API (SSL) and adding
> > >> ACLs
> > >>> to
> >  it. Both are things we know people want, but haven't happened yet.
> > >>> However,
> >  it also interacts with other approaches to adding those features,
> e.g.
> >  layering proxies on top of the existing API (e.g. nginx, apache,
> etc).
> > >>> Just
> >  doing a blanket replacement of password values with a constant would
> > >>> likely
> >  break things for people who secure things via a proxy (and may just
> > not
> >  allow reads of configs unless the user is authorized for the
> > particular
> >  connector). These are the types of concerns we like to think through
> > in
> > >>> the
> >  compatibility section. One option to get the masking functionality
> in
> >  without depending on a bunch of other security improvements might be
> > to
> >  make this configurable so users that need this (and can forgo
> seeing a
> >  valid config via the API) can opt-in.
> > 
> >  Regarding your individual points:
> > 
> >  * I don't think the particular value for the masked content matters
> > >> much.
> >  Any constant indicating a password field is good. Your value seems
> > fine
> > >>> to
> >  me.
> >  * I don't think ConnectorInfo has enough info on its own to do
> proper
> >  masking. In fact, I think you need to parse the config enough to get
> > >> the
> >  Connector-specific ConfigDef out in order to determine which fields
> > are
> >  Password fields. I would probably try to push this to be as central
> as
> >  possible, 

[jira] [Resolved] (KAFKA-6839) ZK session retry with cname record

[Tyler Monahan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tyler Monahan resolved KAFKA-6839.
--
Resolution: Invalid

> ZK session retry with cname record
> --
>
> Key: KAFKA-6839
> URL: https://issues.apache.org/jira/browse/KAFKA-6839
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Tyler Monahan
>Priority: Major
>
> I have a 3 node kafka cluster setup in aws that talks to a 3 node zk cluster 
> behind an elb. I am giving the kafka instances a dns cname record that points 
> to the aws elb which is another cname record pointing to two A records. When 
> the aws elb cname record changes the two A records it is pointing at and 
> kafka trys to reconnect to zk after losing a session it uses the old A 
> records and not the new ones so the reconnect attempt fails. There appears to 
> be some kind of caching instead of using the record that is set in the config 
> file.
> This is the error message I am seeing in the broker logs.
> {code:java}
> [2018-04-30 20:09:21,449] INFO Opening socket connection to server 
> ip-10-65-68-244.us-west-2.compute.internal/10.65.68.244:2181. Will not 
> attempt to authenticate using SASL (unknown error) 
> (org.apache.zookeeper.ClientCnxn)
> [2018-04-30 20:09:24,450] WARN Client session timed out, have not heard from 
> server in 3962ms for sessionid 0x263094512190001 
> (org.apache.zookeeper.ClientCnxn)
> [2018-04-30 20:09:24,451] INFO Client session timed out, have not heard from 
> server in 3962ms for sessionid 0x263094512190001, closing socket connection 
> and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2018-04-30 20:09:26,532] INFO Opening socket connection to server 
> ip-10-65-84-102.us-west-2.compute.internal/10.65.84.102:2181. Will not 
> attempt to authenticate using SASL (unknown error) 
> (org.apache.zookeeper.ClientCnxn)
> [2018-04-30 20:09:29,531] WARN Session 0x263094512190001 for server null, 
> unexpected error, closing socket connection and attempting reconnect 
> (org.apache.zookeeper.ClientCnxn)
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at 
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] KIP-298: Error Handling in Connect

[Jason Gustafson]

Hi Arjun,

Thanks for the KIP. Just a few comments/questions:

1. The proposal allows users to configure the number of retries. I usually
find it easier as a user to work with timeouts since it's difficult to know
how long a retry might take. Have you considered adding a timeout option
which would retry until the timeout expires?
2. The configs are named very generically (e.g. errors.retries.limit). Do
you think it will be clear to users what operations these configs apply to?
3. I wasn't entirely clear what messages are stored in the dead letter
queue. It sounds like it includes both configs and messages since we have
errors.dlq.include.configs? Is there a specific schema you have in mind?
4. I didn't see it mentioned explicitly in the KIP, but I assume the
tolerance metrics are reset after every task rebalance?
5. I wonder if we can do without errors.tolerance.limit. You can get a
similar effect using errors.tolerance.rate.limit if you allow longer
durations. I'm not sure how useful an absolute counter is in practice.

Thanks,
Jason


On Fri, May 18, 2018 at 2:55 PM, Arjun Satish 
wrote:

> Super! Thanks, Magesh!
>
> On Fri, May 18, 2018 at 2:53 PM, Magesh Nandakumar 
> wrote:
>
> > Arjun,
> >
> > Thanks for all the updates. I think it looks great and I don't have any
> > other concerns.
> >
> > Thanks
> > Magesh
> >
> > On Fri, May 18, 2018 at 2:36 PM, Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > The updated version of the KIP that uses the dead-letter-queue only for
> > > sink records and only to store the raw record data looks better and
> > easier
> > > to understand.
> > > I think it's moving to the right direction.
> > >
> > > No further comments from my side.
> > >
> > > Thanks Arjun!
> > >
> > > - Konstantine
> > >
> > > On Fri, May 18, 2018 at 1:07 AM, Arjun Satish 
> > > wrote:
> > >
> > > > Ewen,
> > > >
> > > > Thanks a lot for your comments!
> > > >
> > > > 1. For errors.retry.delay.max.ms, yes we plan to use exponential
> > > backoffs
> > > > with an fixed initial value. Updated the KIP to say this.
> > > >
> > > > 2. A failed operation will be retried (potentially multiple times).
> If
> > > all
> > > > the retries fail, we declare this to be an error. This is where
> > tolerance
> > > > kicks in. Hence, you can have 0 retries, but infinite tolerance (
> > > > errors.tolerance.limit = -1), where we will never retry any failure,
> > but
> > > > all of bad records will be skipped. Updated the KIP. Hopefully this
> is
> > > > clear now.
> > > >
> > > > 3. Yes, for error messages we have some base information (what
> > operation
> > > > failed and with what exception and stacktrace, for example). Hence,
> the
> > > > three configs. The main reason for having properties for disabling
> > > messages
> > > > and configs is to avoid logging sensitive information to unsecured
> > > > locations (for example, the file logs). Updated the KIP to describe
> > this.
> > > >
> > > > I think topic name should be mandatory: if we have a default topic,
> > then
> > > > all the connectors in a cluster will produce messages into it, making
> > it
> > > > confusing to read from. We could have a default pattern for
> > constructing
> > > > topic names, for example: a format like ${connector-name}-errors.
> > > >
> > > > 4. The reason for multiple clusters is to allow users with sensitive
> > data
> > > > to log errors into secure clusters. There are defaults for these
> > > > properties, but if you think this is making the config too complex,
> we
> > > can
> > > > drop the errors.deadletterqueue.producer.* properties from this
> > > > implementation.
> > > >
> > > > 5. I had mentioned that the format is in JSON in the proposed changes
> > > > section. Updated the public interface section to say this again. We
> > could
> > > > provide overrides for the Converter used here, and use an
> AvroConverter
> > > > instead, which should preserve the structure and schema of the data.
> > The
> > > > avro binary would be base64 encoded in the logged records. But yes,
> > this
> > > > brings in configurable converters and their configurations which
> > > introduces
> > > > a new level of complexity (AvroConverters and their dependency on
> > Schema
> > > > Registry, for instance). Hence, they were not included in this
> > proposal.
> > > >
> > > > Another option is to add a StructSerializer and StructDeserializer,
> > which
> > > > can retain the schema and structure of the Structs in the schema. If
> we
> > > do
> > > > this, non-Java clients which need to read these error records would
> > need
> > > to
> > > > port the deserialization logic. Ultimately, we need to indicate what
> > the
> > > > record looks like, and
> > > >
> > > > Could you point out what is unclear w.r.t reprocessing?
> > > >
> > > > Let me know what you think.
> > > >
> > > >
> > > > On Thu, May 17, 2018 at 11:02 PM, Ewen Cheslack-Postava <
> > > 

Re: [VOTE] KIP-298: Error Handling in Connect kafka

[Matthias J. Sax]

+1 (binding)



On 5/21/18 9:30 AM, Randall Hauch wrote:
> Thanks, Arjun. +1 (non-binding)
> 
> Regards,
> Randall
> 
> On Mon, May 21, 2018 at 11:14 AM, Guozhang Wang  wrote:
> 
>> Thanks for the KIP. +1 (binding)
>>
>>
>> Guozhang
>>
>> On Fri, May 18, 2018 at 3:36 PM, Gwen Shapira  wrote:
>>
>>> +1
>>>
>>> Thank you! Error handling in Connect will be a huge improvement.
>>>
>>> On Thu, May 17, 2018, 1:58 AM Arjun Satish 
>> wrote:
>>>
 All,

 Many thanks for all the feedback on KIP-298. Highly appreciate the time
>>> and
 effort you all put into it.

 I've updated the KIP accordingly, and would like to start to start a
>> vote
 on it.

 KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
 298%3A+Error+Handling+in+Connect
 JIRA: https://issues.apache.org/jira/browse/KAFKA-6738
 Discussion Thread: https://www.mail-archive.com/
 dev@kafka.apache.org/msg87660.html

 Thanks very much!

>>>
>>
>>
>>
>> --
>> -- Guozhang
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

[Edoardo Comar]

Hi All

I think I can close the vote as KIP-277  is approved with the
following +1 votes (and no -1s)

binding: Guozhang, Rajini, Damian
non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt

Thanks to everyone who reviewed and voted on the KIP.


On 21 May 2018 at 13:12, Damian Guy  wrote:
> +1 (binding)
>
> Thanks
>
> On Mon, 21 May 2018 at 11:26 Edoardo Comar  wrote:
>
>> Hi, KIP freeze is tomorrow. Anyone else would like to cast their vote ?
>>
>> voting so far :
>> binding: Guozhang, Rajini
>> non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt
>>
>> On 18 May 2018 at 09:38, Edoardo Comar  wrote:
>> > bumping again as this KIP is still missing one binding vote ...
>> >
>> > voting so far :
>> > binding: Guozhang, Rajini
>> > non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt
>> >
>> > Edo
>> >
>> > On 17 May 2018 at 03:15, Matt Farmer  wrote:
>> >> +1 (non-binding)
>> >>
>> >> On Tue, May 15, 2018 at 4:26 AM, Edoardo Comar 
>> wrote:
>> >>
>> >>> Hi,
>> >>> bumping the thread as the current vote count for this KIP is
>> >>> 2 binding +1
>> >>> 5 non-binding +1
>> >>>
>> >>> thanks, Edo
>> >>>
>> >>> On 8 May 2018 at 16:14, Edoardo Comar  wrote:
>> >>> > Hi,
>> >>> > bumping the thread as the current vote count for this KIP is
>> >>> > 2 binding +1
>> >>> > 5 non-binding +1
>> >>> >
>> >>> > so still missing a binding vote please
>> >>> >
>> >>> > thanks,
>> >>> > Edo
>> >>> >
>> >>> >
>> >>> > On 30 April 2018 at 12:49, Manikumar 
>> wrote:
>> >>> >>
>> >>> >> +1 (non-binding)
>> >>> >>
>> >>> >> Thanks
>> >>> >>
>> >>> >> On Thu, Apr 26, 2018 at 9:59 PM, Colin McCabe 
>> >>> wrote:
>> >>> >>
>> >>> >> > +1 (non-binding)
>> >>> >> >
>> >>> >> > best,
>> >>> >> > Colin
>> >>> >> >
>> >>> >> >
>> >>> >> > On Wed, Apr 25, 2018, at 02:45, Edoardo Comar wrote:
>> >>> >> > > Hi,
>> >>> >> > >
>> >>> >> > > The discuss thread on KIP-277 (
>> >>> >> > > https://www.mail-archive.com/dev@kafka.apache.org/msg86540.html
>> )
>> >>> >> > > seems to have been fruitful and concerns have been addressed,
>> please
>> >>> >> > allow
>> >>> >> > > me start a vote on it:
>> >>> >> > >
>> >>> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> >>> >> > 277+-+Fine+Grained+ACL+for+CreateTopics+API
>> >>> >> > >
>> >>> >> > > I will update the small PR to the latest KIP semantics if the
>> vote
>> >>> >> > passes
>> >>> >> > > (as I hope :-).
>> >>> >> > >
>> >>> >> > > cheers
>> >>> >> > > Edo
>> >>> >> > > --
>> >>> >> > >
>> >>> >> > > Edoardo Comar
>> >>> >> > >
>> >>> >> > > IBM Message Hub
>> >>> >> > >
>> >>> >> > > IBM UK Ltd, Hursley Park, SO21 2JN
>> >>> >> > > Unless stated otherwise above:
>> >>> >> > > IBM United Kingdom Limited - Registered in England and Wales
>> with
>> >>> >> > > number
>> >>> >> > > 741598.
>> >>> >> > > Registered office: PO Box 41, North Harbour, Portsmouth,
>> Hampshire
>> >>> PO6
>> >>> >> > 3AU
>> >>> >> >
>> >>> >
>> >>> >
>> >>> >
>> >>> >
>> >>> > --
>> >>> > "When the people fear their government, there is tyranny; when the
>> >>> > government fears the people, there is liberty." [Thomas Jefferson]
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> "When the people fear their government, there is tyranny; when the
>> >>> government fears the people, there is liberty." [Thomas Jefferson]
>> >>>
>> >
>> >
>> >
>> > --
>> > "When the people fear their government, there is tyranny; when the
>> > government fears the people, there is liberty." [Thomas Jefferson]
>>
>>
>>
>> --
>> "When the people fear their government, there is tyranny; when the
>> government fears the people, there is liberty." [Thomas Jefferson]
>>



-- 
"When the people fear their government, there is tyranny; when the
government fears the people, there is liberty." [Thomas Jefferson]

Re: [VOTE] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

[Ron Dagostino]

Thank you, Damian.  This concludes the vote for KIP 255.  The vote passes
with three binding + 1 votes, from Rajini, Jun, and Damian, and three
non-binding +1 votes, from Mickael, Manikumar, and myself.

I have marked the KIP as "Accepted".

The pull request is available at https://github.com/apache/kafka/pull/4994
and is squashed/rebased onto trunk.

Ron

On Mon, May 21, 2018 at 8:25 AM, Damian Guy  wrote:

> +1 (binding)
>
> Thanks
>
> On Mon, 21 May 2018 at 04:59 Ron Dagostino  wrote:
>
> > Hi Committers.  One more binding affirmative vote is required if KIP 255
> > is to have a chance of being included in the 2.0.0 release.  Please vote
> > today.
> >
> > Ron
> >
> > > On May 18, 2018, at 9:27 PM, Ron Dagostino  wrote:
> > >
> > > Hi committers.  KIP 255 still needs 1 more binding vote.  Currently
> > there are two binding + 1 votes, from Rajini and Jun, and three
> non-binding
> > +1 votes, from Mickael, Manikumar, and myself.
> > >
> > > Please vote by the Monday deadline.
> > >
> > > Ron
> > >
> > >> On Thu, May 17, 2018 at 10:59 AM, Ron Dagostino 
> > wrote:
> > >> Hi Jun.  I've updated the KIP to add a new section titled "Summary for
> > Production Use" that includes this information along with a consolidated
> > set of references to the applicable specifications.  Thanks for the
> > questions.
> > >>
> > >> *We still need another binding vote* (currently there are two binding
> +
> > 1 votes, from Rajini and Jun, and three non-binding +1 votes, from
> Mickael,
> > Manikumar, and myself).
> > >>
> > >> Please vote before the May 22nd KIP Freeze deadline so this KIP can be
> > included in the 2.0.0 release.
> > >>
> > >> A pull request is available and includes additional commits reflecting
> > initial review comments: https://github.com/apache/kafka/pull/4994
> > >>
> > >> Ron
> > >>
> > >>> On Wed, May 16, 2018 at 8:09 PM, Jun Rao  wrote:
> > >>> Hi, Ron,
> > >>>
> > >>> Thanks. I understand now. It may be useful to add a reference to JWT
> > in the
> > >>> KIP.
> > >>>
> > >>> Jun
> > >>>
> > >>> On Tue, May 15, 2018 at 6:51 PM, Ron Dagostino 
> > wrote:
> > >>>
> > >>> > Hi Jun.  I think you are getting at the fact that OAuth 2 is a
> > flexible
> > >>> > framework that allows different installations to do things
> > differently.  It
> > >>> > is true that the principal name in Kafka could come from any claim
> > in the
> > >>> > token.  Most of the time it would come from the 'sub' claim, but it
> > could
> > >>> > certainly come from another claim, or it could be only indirectly
> > based on
> > >>> > a claim value (maybe certain text would be trimmed or
> > prefixed/suffixed).
> > >>> > The point, which I think you are getting at, is that because the
> > framework
> > >>> > is flexible we need to accommodate that flexibility.  The callback
> > handler
> > >>> > class defined by the listener.name.sasl_ssl.
> oauthbearer.sasl.server.
> > >>> > callback.handler.class configuration value gives us the required
> > >>> > flexibility.  As an example, I have an implementation that
> leverages
> > a
> > >>> > popular open source JOSE library to parse the compact
> serialization,
> > >>> > retrieve the public key if it has not yet been retrieved, verify
> the
> > >>> > digital signature, and map the 'sub' claim to the
> OAuthBearerToken's
> > >>> > principal name (which becomes the SASL authorization ID, which
> > becomes the
> > >>> > Kafka principal name).  I could just as easily have mapped a
> > different
> > >>> > claim to the OAuthBearerToken's principal name, manipulated the
> > 'sub' claim
> > >>> > value in some way, etc.  I write the callback handler code, so I
> > complete
> > >>> > flexibility to do whatever my OAuth 2 installation requires me to
> do.
> > >>> >
> > >>> > Ron
> > >>> >
> > >>> > On Tue, May 15, 2018 at 1:39 PM, Jun Rao  wrote:
> > >>> >
> > >>> > > Hi, Ron,
> > >>> > >
> > >>> > > Thanks for the reply. I understood your answers to #2 and #3.
> > >>> > >
> > >>> > > For #1, will the server map all clients' principal name to the
> > value
> > >>> > > associated with "sub" claim? How do we support mapping different
> > clients
> > >>> > to
> > >>> > > different principal names?
> > >>> > >
> > >>> > > Jun
> > >>> > >
> > >>> > > On Mon, May 14, 2018 at 7:02 PM, Ron Dagostino <
> rndg...@gmail.com>
> > >>> > wrote:
> > >>> > >
> > >>> > > > Hi Jun.  Thanks for the +1 vote.
> > >>> > > >
> > >>> > > > Regarding the first question about token claims, yes, you have
> it
> > >>> > correct
> > >>> > > > about translating the OAuth token to a principle name via a
> JAAS
> > module
> > >>> > > > option in the default unsecured case.  Specifically, the OAuth
> > SASL
> > >>> > > Server
> > >>> > > > implementation is responsible for setting the authorization ID,
> > and it
> > >>> > > gets
> > >>> > > > the authorization ID from the 

Re: [VOTE] KIP-298: Error Handling in Connect kafka

[Randall Hauch]

Thanks, Arjun. +1 (non-binding)

Regards,
Randall

On Mon, May 21, 2018 at 11:14 AM, Guozhang Wang  wrote:

> Thanks for the KIP. +1 (binding)
>
>
> Guozhang
>
> On Fri, May 18, 2018 at 3:36 PM, Gwen Shapira  wrote:
>
> > +1
> >
> > Thank you! Error handling in Connect will be a huge improvement.
> >
> > On Thu, May 17, 2018, 1:58 AM Arjun Satish 
> wrote:
> >
> > > All,
> > >
> > > Many thanks for all the feedback on KIP-298. Highly appreciate the time
> > and
> > > effort you all put into it.
> > >
> > > I've updated the KIP accordingly, and would like to start to start a
> vote
> > > on it.
> > >
> > > KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 298%3A+Error+Handling+in+Connect
> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-6738
> > > Discussion Thread: https://www.mail-archive.com/
> > > dev@kafka.apache.org/msg87660.html
> > >
> > > Thanks very much!
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

[Gwen Shapira]

+1 from me too. Can't wait to use this feature.

On Mon, May 21, 2018 at 9:11 AM, Guozhang Wang  wrote:

> Thanks for the KIP. +1 from me (binding).
>
>
> Guozhang
>
> On Fri, May 18, 2018 at 9:46 AM, Randall Hauch  wrote:
>
> > Looks great.
> >
> > +1 (non-binding)
> >
> > Regards,
> > Randall
> >
> > On Fri, May 18, 2018 at 10:23 AM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> > wrote:
> >
> > > Thanks, Robert! Sounds good. And thanks for the KIP.
> > >
> > > +1 (binding)
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Fri, May 18, 2018 at 4:17 PM, Robert Yokota 
> > wrote:
> > >
> > > > HI Rajini,
> > > >
> > > > Good questions.
> > > >
> > > > First, if no ConfigProviders are configured, then config values of
> the
> > > form
> > > > ${vault:mypassword} will remain as is.
> > > >
> > > > Second, I mention in the KIP that if a provider does not have a value
> > > for a
> > > > given key, the variable will remain unresolved and the final value
> will
> > > be
> > > > of the form ${vault:mypassword} still.
> > > >
> > > > If one wants to use a config value ${vault:mypassword}, as well as
> the
> > > > VaultConfigProvider, one can choose to use a different prefix besides
> > > > "vault" when referring to the VaultConfigProvider since the prefixes
> > are
> > > > arbitrary and specified in a config file.
> > > >
> > > > Finally, if one want to use a config value ${vault:mypassword}, as
> well
> > > as
> > > > the VaultConfigProvider, and one wants to use the prefix "vault" and
> > not
> > > > something else, then yes, one could use a LiteralConfigProvider as
> you
> > > > described, or even put the ${vault:mypassword} in a different file
> and
> > > use
> > > > the FileConfigProvider to pull in the value (since there is only one
> > > level
> > > > of indirection).
> > > >
> > > > Thanks,
> > > > Robert
> > > >
> > > >
> > > >
> > > > On Fri, May 18, 2018 at 3:42 AM, Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Robert,
> > > > >
> > > > > A couple of questions:
> > > > >
> > > > >
> > > > >1. Since we always expand config values, don't we also need a
> way
> > to
> > > > >include values that never get expanded? I may want to use
> > > > >"${vault:mypassword}" as my literal password without a lookup.
> > Since
> > > > we
> > > > >allow only level of indirection, perhaps all we need is a
> > > > ConfigProvider
> > > > >that uses the string inside, for example:
> > > > ${literal:${vault:mypassword}}
> > > > > ?
> > > > >It would avoid having restrictions on what passwords can look
> > like.
> > > > >2. What is the behaviour if I specify a password that is
> > > > >"${notavault:something}" that matches the config provider
> syntax,
> > > but
> > > > > for
> > > > >which there is no config provider?
> > > > >
> > > > >
> > > > >
> > > > > On Fri, May 18, 2018 at 5:41 AM, Ewen Cheslack-Postava <
> > > > e...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Thanks for addressing this Robert, it's a pretty common user
> need.
> > > > > >
> > > > > > First, +1 (binding) generally.
> > > > > >
> > > > > > Two very minor comments that I think could be clarified but
> > wouldn't
> > > > > affect
> > > > > > votes:
> > > > > >
> > > > > > * Let's list in the KIP what package the ConfigProvider,
> > > > > > ConfigChangeCallback, ConfigData and ConfigTransformer interfaces
> > are
> > > > > > defined in. Very, very minor, but given the aim to possibly reuse
> > > > > elsewhere
> > > > > > and the fact that it'll likely end up in the common packages
> might
> > > mean
> > > > > > devs focused more on the common/core packages will have strong
> > > opinions
> > > > > > where they should be. I think it'd definitely be good to get
> input
> > > from
> > > > > > folks focusing on the broker on where they think it should go
> > since I
> > > > > think
> > > > > > it would be very natural to extend this to security settings
> there.
> > > > > (Also,
> > > > > > I think ConfigData is left out of the list of new interfaces by
> > > > accident,
> > > > > > but I think it's clear what's being added anyway.)
> > > > > > * I may have glanced past it, but we're not shipping any
> > > > ConfigProviders
> > > > > > out of the box? This mentions file and vault, but just as
> examples.
> > > > Just
> > > > > > want to make sure everyone knows up front that this is a
> pluggable
> > > API,
> > > > > but
> > > > > > you need to add more jars to take advantage of it. I think this
> is
> > > fine
> > > > > as
> > > > > > I don't think there are truly common secrets provider
> > > > > > formats/apis/protocols, just want to make sure it is clear.
> > > > > >
> > > > > > Thanks,
> > > > > > Ewen
> > > > > >
> > > > > > On Thu, May 17, 2018 at 6:19 PM Ted Yu 
> > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >  Original message From: Magesh Nandakumar <
> > > 

Re: [VOTE] KIP-298: Error Handling in Connect kafka

[Guozhang Wang]

Thanks for the KIP. +1 (binding)


Guozhang

On Fri, May 18, 2018 at 3:36 PM, Gwen Shapira  wrote:

> +1
>
> Thank you! Error handling in Connect will be a huge improvement.
>
> On Thu, May 17, 2018, 1:58 AM Arjun Satish  wrote:
>
> > All,
> >
> > Many thanks for all the feedback on KIP-298. Highly appreciate the time
> and
> > effort you all put into it.
> >
> > I've updated the KIP accordingly, and would like to start to start a vote
> > on it.
> >
> > KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 298%3A+Error+Handling+in+Connect
> > JIRA: https://issues.apache.org/jira/browse/KAFKA-6738
> > Discussion Thread: https://www.mail-archive.com/
> > dev@kafka.apache.org/msg87660.html
> >
> > Thanks very much!
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

[Guozhang Wang]

Thanks for the KIP. +1 from me (binding).


Guozhang

On Fri, May 18, 2018 at 9:46 AM, Randall Hauch  wrote:

> Looks great.
>
> +1 (non-binding)
>
> Regards,
> Randall
>
> On Fri, May 18, 2018 at 10:23 AM, Rajini Sivaram 
> wrote:
>
> > Thanks, Robert! Sounds good. And thanks for the KIP.
> >
> > +1 (binding)
> >
> > Regards,
> >
> > Rajini
> >
> > On Fri, May 18, 2018 at 4:17 PM, Robert Yokota 
> wrote:
> >
> > > HI Rajini,
> > >
> > > Good questions.
> > >
> > > First, if no ConfigProviders are configured, then config values of the
> > form
> > > ${vault:mypassword} will remain as is.
> > >
> > > Second, I mention in the KIP that if a provider does not have a value
> > for a
> > > given key, the variable will remain unresolved and the final value will
> > be
> > > of the form ${vault:mypassword} still.
> > >
> > > If one wants to use a config value ${vault:mypassword}, as well as the
> > > VaultConfigProvider, one can choose to use a different prefix besides
> > > "vault" when referring to the VaultConfigProvider since the prefixes
> are
> > > arbitrary and specified in a config file.
> > >
> > > Finally, if one want to use a config value ${vault:mypassword}, as well
> > as
> > > the VaultConfigProvider, and one wants to use the prefix "vault" and
> not
> > > something else, then yes, one could use a LiteralConfigProvider as you
> > > described, or even put the ${vault:mypassword} in a different file and
> > use
> > > the FileConfigProvider to pull in the value (since there is only one
> > level
> > > of indirection).
> > >
> > > Thanks,
> > > Robert
> > >
> > >
> > >
> > > On Fri, May 18, 2018 at 3:42 AM, Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Robert,
> > > >
> > > > A couple of questions:
> > > >
> > > >
> > > >1. Since we always expand config values, don't we also need a way
> to
> > > >include values that never get expanded? I may want to use
> > > >"${vault:mypassword}" as my literal password without a lookup.
> Since
> > > we
> > > >allow only level of indirection, perhaps all we need is a
> > > ConfigProvider
> > > >that uses the string inside, for example:
> > > ${literal:${vault:mypassword}}
> > > > ?
> > > >It would avoid having restrictions on what passwords can look
> like.
> > > >2. What is the behaviour if I specify a password that is
> > > >"${notavault:something}" that matches the config provider syntax,
> > but
> > > > for
> > > >which there is no config provider?
> > > >
> > > >
> > > >
> > > > On Fri, May 18, 2018 at 5:41 AM, Ewen Cheslack-Postava <
> > > e...@confluent.io>
> > > > wrote:
> > > >
> > > > > Thanks for addressing this Robert, it's a pretty common user need.
> > > > >
> > > > > First, +1 (binding) generally.
> > > > >
> > > > > Two very minor comments that I think could be clarified but
> wouldn't
> > > > affect
> > > > > votes:
> > > > >
> > > > > * Let's list in the KIP what package the ConfigProvider,
> > > > > ConfigChangeCallback, ConfigData and ConfigTransformer interfaces
> are
> > > > > defined in. Very, very minor, but given the aim to possibly reuse
> > > > elsewhere
> > > > > and the fact that it'll likely end up in the common packages might
> > mean
> > > > > devs focused more on the common/core packages will have strong
> > opinions
> > > > > where they should be. I think it'd definitely be good to get input
> > from
> > > > > folks focusing on the broker on where they think it should go
> since I
> > > > think
> > > > > it would be very natural to extend this to security settings there.
> > > > (Also,
> > > > > I think ConfigData is left out of the list of new interfaces by
> > > accident,
> > > > > but I think it's clear what's being added anyway.)
> > > > > * I may have glanced past it, but we're not shipping any
> > > ConfigProviders
> > > > > out of the box? This mentions file and vault, but just as examples.
> > > Just
> > > > > want to make sure everyone knows up front that this is a pluggable
> > API,
> > > > but
> > > > > you need to add more jars to take advantage of it. I think this is
> > fine
> > > > as
> > > > > I don't think there are truly common secrets provider
> > > > > formats/apis/protocols, just want to make sure it is clear.
> > > > >
> > > > > Thanks,
> > > > > Ewen
> > > > >
> > > > > On Thu, May 17, 2018 at 6:19 PM Ted Yu 
> wrote:
> > > > >
> > > > > > +1
> > > > > >  Original message From: Magesh Nandakumar <
> > > > > > mage...@confluent.io> Date: 5/17/18  6:05 PM  (GMT-08:00) To:
> > > > > > dev@kafka.apache.org Subject: Re: [VOTE] KIP-297: Externalizing
> > > > Secrets
> > > > > > for Connect Configurations
> > > > > > Thanks Robert, this looks great
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On Thu, May 17, 2018 at 5:35 PM, Colin McCabe <
> cmcc...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Thanks, Robert!
> > > > > > >

Re: [VOTE] KIP-285: Connect Rest Extension Plugin

[Guozhang Wang]

Thanks for the KIP, +1 from me (binding)

One minor suggestion: in "class TaskState" constructor, the passed in
parameter of `worker` could be `workerId`, to be consistent with others in
ConnectorState?


Guozhang


On Fri, May 18, 2018 at 9:44 AM, Randall Hauch  wrote:

> +1 (non-binding)
>
> Regards,
> Randall
>
> On Fri, May 18, 2018 at 11:32 AM, Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > +1 (non-binding)
> >
> > - Konstantine
> >
> > On Thu, May 17, 2018 at 10:05 PM, Ewen Cheslack-Postava <
> e...@confluent.io
> > >
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Ewen
> > >
> > > On Thu, May 17, 2018 at 12:16 PM Ted Yu  wrote:
> > >
> > > > +1
> > > >  Original message From: Gwen Shapira <
> > g...@confluent.io>
> > > > Date: 5/17/18  12:02 PM  (GMT-08:00) To: dev 
> > > > Subject: Re: [VOTE] KIP-285: Connect Rest Extension Plugin
> > > > LGTM. +1.
> > > >
> > > > On Wed, May 16, 2018 at 8:19 PM, Magesh Nandakumar <
> > mage...@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > After a good round of discussions with excellent feedback and no
> > major
> > > > > objections, I would like to start a vote on KIP-285: Connect Rest
> > > > Extension
> > > > > Plugin.
> > > > >
> > > > > KIP: <
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 285%3A+Connect+Rest+Extension+Plugin
> > > > > >
> > > > >
> > > > >
> > > > > JIRA: <*https://issues.apache.org/jira/browse/KAFKA-6776
> > > > > *>
> > > > >
> > > > > Discussion thread: <
> > > > > https://www.mail-archive.com/dev@kafka.apache.org/msg86910.html>
> > > > >
> > > > > Thanks,
> > > > > Magesh
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Gwen Shapira*
> > > > Product Manager | Confluent
> > > > 650.450.2760 | @gwenshap
> > > > Follow us: Twitter  | blog
> > > > 
> > > >
> > >
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Guozhang Wang]

Thanks Dhruvil for the KIP, +1 from me (binding).


Guozhang

On Mon, May 21, 2018 at 7:52 AM, Dhruvil Shah  wrote:

> Hi,
>
> I would like to start a vote on KIP-306 which proposes to add a
> configuration to delay responses to failed authentication.
>
> Link to the KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 306%3A+Configuration+for+Delaying+Response+to+Failed+Client+Authentication
>
> Thanks,
> Dhruvil
>



-- 
-- Guozhang

Re: [VOTE] KIP-290: Support for wildcard suffixed ACLs

[Guozhang Wang]

Thanks for the KIP, +1 from me (binding).


Guozhang

On Mon, May 21, 2018 at 5:22 AM, Damian Guy  wrote:

> +1 (binding)
>
> On Sat, 19 May 2018 at 03:51 Piyush Vijay  wrote:
>
> > Hi everyone,
> >
> > I would like to start a vote for
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 290%3A+Support+for+wildcard+suffixed+ACLs
> > .
> >
> > The KIP proposes a way to support wildcard-suffixed resource names in
> Kafka
> > ACLs.
> >
> > The main challenge was to support it in a backward compatible way because
> > resources like consumer groups don't have defined naming convention and
> can
> > have '*' in their names.
> >
> > Please take a look.
> >
> > Thanks
> >
> > Piyush Vijay
> >
>



-- 
-- Guozhang

[jira] [Created] (KAFKA-6926) Reduce NPath exceptions in Connect

[Randall Hauch (JIRA)]

Randall Hauch created KAFKA-6926:


 Summary: Reduce NPath exceptions in Connect
 Key: KAFKA-6926
 URL: https://issues.apache.org/jira/browse/KAFKA-6926
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 2.0.0
Reporter: Randall Hauch


The [recent upgrade of Checkstyle and move to Java 
8|https://github.com/apache/kafka/pull/5046/files/1a83b7d04bd3cecbb68d033211b1bcdfbe085d47#diff-6869a8c771257f000c3cefb045e9d289]
 has caused some existing code to not pass the NPath rule. Look at the classes 
to see what might need to be changed to remove the class from the suppression 
rule:

AbstractStatus|ConnectRecord|ConnectSchema|DistributedHerder|FileStreamSourceTask|JsonConverter|KafkaConfigBackingStore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[VOTE] KIP-306: Configuration for Delaying Response to Failed Client Authentication

[Dhruvil Shah]

Hi,

I would like to start a vote on KIP-306 which proposes to add a
configuration to delay responses to failed authentication.

Link to the KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-306%3A+Configuration+for+Delaying+Response+to+Failed+Client+Authentication

Thanks,
Dhruvil

Re: [VOTE] KIP-244: Add Record Header support to Kafka Streams

[Jorge Esteban Quilcate Otoya]

Thanks for your votes and your feedback.

This KIP has been approved with the following results:

Binding +1s: 3 (Matthias, Damina, Guozhang)
Non-biniding +1s: (Bill, Ted)

Jorge.

El mar., 15 may. 2018 a las 20:01, Bill Bejeck ()
escribió:

> Thanks for the KIP!
>
> +1
>
> -Bill
>
> On Tue, May 15, 2018 at 1:47 PM, Damian Guy  wrote:
>
> > Thanks. +1 (binding)
> >
> > On Tue, 15 May 2018 at 01:04 Jorge Esteban Quilcate Otoya <
> > quilcate.jo...@gmail.com> wrote:
> >
> > > @Guozhang added. Thanks!
> > >
> > > El mar., 15 may. 2018 a las 5:50, Matthias J. Sax (<
> > matth...@confluent.io
> > > >)
> > > escribió:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks a lot for the KIP!
> > > >
> > > > -Matthias
> > > >
> > > > On 5/14/18 10:17 AM, Guozhang Wang wrote:
> > > > > +1 from me
> > > > >
> > > > > One more comment on the wiki: while reviewing the PR I realized
> that
> > > in `
> > > > > MockProcessorContext.java
> > > > > <
> > > >
> > > https://github.com/apache/kafka/pull/4955/files#diff-
> > d5440e7338f775230019a86e6bcacccb
> > > > >`
> > > > > we are also adding one additional API plus modifying the existing
> > > > > `setRecordMetadata` API. Since this class is part of the public
> > > > test-utils
> > > > > package we should claim it in the wiki as well.
> > > > >
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, May 14, 2018 at 8:43 AM, Ted Yu 
> wrote:
> > > > >
> > > > >> +1
> > > > >>
> > > > >> On Mon, May 14, 2018 at 8:31 AM, Jorge Esteban Quilcate Otoya <
> > > > >> quilcate.jo...@gmail.com> wrote:
> > > > >>
> > > > >>> Hi everyone,
> > > > >>>
> > > > >>> I would like to start a vote on KIP-244: Add Record Header
> support
> > to
> > > > >> Kafka
> > > > >>> Streams
> > > > >>>
> > > > >>> KIP wiki page:
> > > > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > >>> 244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> > > > >>>
> > > > >>> The discussion thread is here:
> > > > >>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201805.
> > > > >>>
> mbox/%3CCAC3UcJvrgcBfe6%3DiW6%2BuTWsLB%2B4CsHgRmDx9TvCzJQrWvfg7_w%
> > > > >>> 40mail.gmail.com%3E
> > > > >>>
> > > > >>> Cheers,
> > > > >>> Jorge.
> > > > >>>
> > > > >>
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

[Bill Bejeck]

All,

The 72 hours has passed so I will close this vote.

KIP-295 has been accepted with 3 binding votes (Damian Guy, Matthias Sax,
and Guozhang Wang) 2 non-binding (Ted Yu and myself)

Thanks,
Bill

On Wed, May 16, 2018 at 12:20 PM, Damian Guy  wrote:

> +1
> On Tue, 15 May 2018 at 15:04, Ted Yu  wrote:
>
> > +1
> >  Original message From: Guozhang Wang <
> wangg...@gmail.com>
> > Date: 5/15/18  2:34 PM  (GMT-08:00) To: dev@kafka.apache.org Subject:
> Re:
> > [VOTE] KIP-295: Add Streams Configuration Allowing for Optional Topology
> > Optimization
> > +1 (binding).
> >
> > On Tue, May 15, 2018 at 2:16 PM, Matthias J. Sax 
> > wrote:
> >
> > > +1 (binding)
> > >
> > >
> > > On 5/15/18 1:45 PM, Bill Bejeck wrote:
> > > > Hi all,
> > > >
> > > > I'd like to start a vote on KIP-295: Add Streams Configuration
> Allowing
> > > for
> > > > Optional Topology Optimization.
> > > >
> > > > KIP wiki page:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 295%3A+Add+Streams+Configuration+Allowing+for+
> > > Optional+Topology+Optimization
> > > >
> > > > Discussion thread:
> > > > https://www.mail-archive.com/dev@kafka.apache.org/msg87593.html
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [DISCUSS] Apache Kafka 2.0.0 Release Plan

[Jorge Esteban Quilcate Otoya]

Ok, thanks!

On Mon, 21 May 2018, 15:17 Rajini Sivaram,  wrote:

> Jorge,
>
> KIP-244 has binding votes from Guozhang, Matthias and Damian. So you can
> close the vote and move the KIP to Accepted state.
>
>
> On Mon, May 21, 2018 at 2:13 PM, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > I think there is one missing vote in KIP-244 too. Pull request is almost
> > ready to be merged also.
> >
> > On Mon, 21 May 2018, 11:48 Rajini Sivaram, 
> > wrote:
> >
> > > Hi all,
> > >
> > > This is a reminder that KIP freeze for 2.0.0 release is tomorrow. KIPs
> > that
> > > are not yet in voting stage will be postponed to the next release. We
> > have
> > > several KIPs that need more votes to be accepted. Please participate in
> > > reviews and votes to enable these to be added to the release (or
> > postponed
> > > if required). We need more committers to review and vote since binding
> > > votes are needed by tomorrow to include these in the release, but
> reviews
> > > and votes from everyone in the community are welcome.
> > >
> > > KIPs that require one more binding vote:
> > >
> > >- KIP-235: https://cwiki.apache.org/confluence/display/KAFKA/K
> > >IP-235%3A+Add+DNS+alias+support+for+secured+connection
> > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%
> > 3A+Add+DNS+alias+support+for+secured+connection
> > > >
> > >- KIP-255: https://cwiki.apache.org/confluence/pages/viewpage.
> > >action?pageId=75968876
> > >- KIP-277: https://cwiki.apache.org/confluence/display/KAFKA/K
> > >IP-277+-+Fine+Grained+ACL+for+CreateTopics+API
> > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+
> > Fine+Grained+ACL+for+CreateTopics+API
> > > >
> > >- KIP-285: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >285%3A+Connect+Rest+Extension+Plugin
> > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-285%
> > 3A+Connect+Rest+Extension+Plugin
> > > >
> > >- KIP-297:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%
> > >3A+Externalizing+Secrets+for+Connect+Configurations
> > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%
> > 3A+Externalizing+Secrets+for+Connect+Configurations
> > > >
> > >
> > > KIPs that require two more binding votes:
> > >
> > >- KIP-248: https://cwiki.apache.org/confluence/display/KAFKA/K
> > >IP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+
> > Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> > > >
> > >- KIP-290: https://cwiki.apache.org/confl
> > uence/display/KAFKA/KIP-290%3A
> > >+Support+for+wildcard+suffixed+ACLs
> > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%
> > 3A+Support+for+wildcard+suffixed+ACLs
> > > >
> > >- KIP-298: https://cwiki.apache.org/confluence/display/KAFKA/K
> > >IP-298%3A+Error+Handling+in+Connect
> > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%
> > 3A+Error+Handling+in+Connect
> > > >
> > >
> > > KIPs that require three binding votes
> > >
> > >
> > >- KIP-275: https://cwiki.apache.org/confluence/pages/viewpage.
> > >action?pageId=75977607
> > >
> > >
> > > Thanks,
> > >
> > > Rajini
> > >
> > > On Tue, May 15, 2018 at 11:23 AM, Rajini Sivaram <
> > rajinisiva...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > This is just a reminder that KIP freeze for 2.0.0 release is in a
> > week's
> > > > time and we still have a lot of KIPs in progress. KIPs that are
> > currently
> > > > being discussed should start the voting process soon to get voting
> > > complete
> > > > by 22nd of May. Please participate in discussions and votes to enable
> > > > these to be added to the release (or postponed if required).
> > > >
> > > > Voting is in progress for the following KIPs:
> > > >
> > > >
> > > >- KIP-235: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >235%3A+Add+DNS+alias+support+for+secured+connection
> > > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%
> > 3A+Add+DNS+alias+support+for+secured+connection
> > > >
> > > >- KIP-244: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> > > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%
> > 3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> > > >
> > > >- KIP-248: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> > > ><
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+
> > Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> > > >
> > > >- KIP-255: https://cwiki.apache.org/confluence/pages/viewpage.
> > > >action?pageId=75968876
> > > >- 

Re: [DISCUSS] Apache Kafka 2.0.0 Release Plan

[Rajini Sivaram]

Jorge,

KIP-244 has binding votes from Guozhang, Matthias and Damian. So you can
close the vote and move the KIP to Accepted state.


On Mon, May 21, 2018 at 2:13 PM, Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> I think there is one missing vote in KIP-244 too. Pull request is almost
> ready to be merged also.
>
> On Mon, 21 May 2018, 11:48 Rajini Sivaram, 
> wrote:
>
> > Hi all,
> >
> > This is a reminder that KIP freeze for 2.0.0 release is tomorrow. KIPs
> that
> > are not yet in voting stage will be postponed to the next release. We
> have
> > several KIPs that need more votes to be accepted. Please participate in
> > reviews and votes to enable these to be added to the release (or
> postponed
> > if required). We need more committers to review and vote since binding
> > votes are needed by tomorrow to include these in the release, but reviews
> > and votes from everyone in the community are welcome.
> >
> > KIPs that require one more binding vote:
> >
> >- KIP-235: https://cwiki.apache.org/confluence/display/KAFKA/K
> >IP-235%3A+Add+DNS+alias+support+for+secured+connection
> ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%
> 3A+Add+DNS+alias+support+for+secured+connection
> > >
> >- KIP-255: https://cwiki.apache.org/confluence/pages/viewpage.
> >action?pageId=75968876
> >- KIP-277: https://cwiki.apache.org/confluence/display/KAFKA/K
> >IP-277+-+Fine+Grained+ACL+for+CreateTopics+API
> ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+
> Fine+Grained+ACL+for+CreateTopics+API
> > >
> >- KIP-285: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >285%3A+Connect+Rest+Extension+Plugin
> ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-285%
> 3A+Connect+Rest+Extension+Plugin
> > >
> >- KIP-297: https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%
> >3A+Externalizing+Secrets+for+Connect+Configurations
> ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%
> 3A+Externalizing+Secrets+for+Connect+Configurations
> > >
> >
> > KIPs that require two more binding votes:
> >
> >- KIP-248: https://cwiki.apache.org/confluence/display/KAFKA/K
> >IP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+
> Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> > >
> >- KIP-290: https://cwiki.apache.org/confl
> uence/display/KAFKA/KIP-290%3A
> >+Support+for+wildcard+suffixed+ACLs
> ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%
> 3A+Support+for+wildcard+suffixed+ACLs
> > >
> >- KIP-298: https://cwiki.apache.org/confluence/display/KAFKA/K
> >IP-298%3A+Error+Handling+in+Connect
> ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%
> 3A+Error+Handling+in+Connect
> > >
> >
> > KIPs that require three binding votes
> >
> >
> >- KIP-275: https://cwiki.apache.org/confluence/pages/viewpage.
> >action?pageId=75977607
> >
> >
> > Thanks,
> >
> > Rajini
> >
> > On Tue, May 15, 2018 at 11:23 AM, Rajini Sivaram <
> rajinisiva...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > This is just a reminder that KIP freeze for 2.0.0 release is in a
> week's
> > > time and we still have a lot of KIPs in progress. KIPs that are
> currently
> > > being discussed should start the voting process soon to get voting
> > complete
> > > by 22nd of May. Please participate in discussions and votes to enable
> > > these to be added to the release (or postponed if required).
> > >
> > > Voting is in progress for the following KIPs:
> > >
> > >
> > >- KIP-235: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >235%3A+Add+DNS+alias+support+for+secured+connection
> > ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%
> 3A+Add+DNS+alias+support+for+secured+connection
> > >
> > >- KIP-244: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> > ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%
> 3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> > >
> > >- KIP-248: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> > ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+
> Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> > >
> > >- KIP-255: https://cwiki.apache.org/confluence/pages/viewpage.
> > >action?pageId=75968876
> > >- KIP-275: https://cwiki.apache.org/confluence/pages/viewpage.
> > >action?pageId=75977607
> > >- KIP-277: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >277+-+Fine+Grained+ACL+for+CreateTopics+API
> > ><
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+
> Fine+Grained+ACL+for+CreateTopics+API
> > >
> > >- 

Re: [DISCUSS] Apache Kafka 2.0.0 Release Plan

[Jorge Esteban Quilcate Otoya]

I think there is one missing vote in KIP-244 too. Pull request is almost
ready to be merged also.

On Mon, 21 May 2018, 11:48 Rajini Sivaram,  wrote:

> Hi all,
>
> This is a reminder that KIP freeze for 2.0.0 release is tomorrow. KIPs that
> are not yet in voting stage will be postponed to the next release. We have
> several KIPs that need more votes to be accepted. Please participate in
> reviews and votes to enable these to be added to the release (or postponed
> if required). We need more committers to review and vote since binding
> votes are needed by tomorrow to include these in the release, but reviews
> and votes from everyone in the community are welcome.
>
> KIPs that require one more binding vote:
>
>- KIP-235: https://cwiki.apache.org/confluence/display/KAFKA/K
>IP-235%3A+Add+DNS+alias+support+for+secured+connection
><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection
> >
>- KIP-255: https://cwiki.apache.org/confluence/pages/viewpage.
>action?pageId=75968876
>- KIP-277: https://cwiki.apache.org/confluence/display/KAFKA/K
>IP-277+-+Fine+Grained+ACL+for+CreateTopics+API
><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+Fine+Grained+ACL+for+CreateTopics+API
> >
>- KIP-285: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>285%3A+Connect+Rest+Extension+Plugin
><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-285%3A+Connect+Rest+Extension+Plugin
> >
>- KIP-297: https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%
>3A+Externalizing+Secrets+for+Connect+Configurations
><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations
> >
>
> KIPs that require two more binding votes:
>
>- KIP-248: https://cwiki.apache.org/confluence/display/KAFKA/K
>IP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> >
>- KIP-290: https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A
>+Support+for+wildcard+suffixed+ACLs
><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A+Support+for+wildcard+suffixed+ACLs
> >
>- KIP-298: https://cwiki.apache.org/confluence/display/KAFKA/K
>IP-298%3A+Error+Handling+in+Connect
><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect
> >
>
> KIPs that require three binding votes
>
>
>- KIP-275: https://cwiki.apache.org/confluence/pages/viewpage.
>action?pageId=75977607
>
>
> Thanks,
>
> Rajini
>
> On Tue, May 15, 2018 at 11:23 AM, Rajini Sivaram 
> wrote:
>
> > Hi all,
> >
> > This is just a reminder that KIP freeze for 2.0.0 release is in a week's
> > time and we still have a lot of KIPs in progress. KIPs that are currently
> > being discussed should start the voting process soon to get voting
> complete
> > by 22nd of May. Please participate in discussions and votes to enable
> > these to be added to the release (or postponed if required).
> >
> > Voting is in progress for the following KIPs:
> >
> >
> >- KIP-235: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >235%3A+Add+DNS+alias+support+for+secured+connection
> ><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection
> >
> >- KIP-244: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> ><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> >
> >- KIP-248: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> ><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
> >
> >- KIP-255: https://cwiki.apache.org/confluence/pages/viewpage.
> >action?pageId=75968876
> >- KIP-275: https://cwiki.apache.org/confluence/pages/viewpage.
> >action?pageId=75977607
> >- KIP-277: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >277+-+Fine+Grained+ACL+for+CreateTopics+API
> ><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+Fine+Grained+ACL+for+CreateTopics+API
> >
> >- KIP-278: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >278+-+Add+version+option+to+Kafka%27s+commands
> ><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-278+-+Add+version+option+to+Kafka%27s+commands
> >
> >- KIP-282: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >282%3A+Add+the+listener+name+to+the+authentication+context
> ><
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-282%3A+Add+the+listener+name+to+the+authentication+context
> 

Re: [VOTE] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

[Damian Guy]

+1 (binding)

Thanks

On Mon, 21 May 2018 at 04:59 Ron Dagostino  wrote:

> Hi Committers.  One more binding affirmative vote is required if KIP 255
> is to have a chance of being included in the 2.0.0 release.  Please vote
> today.
>
> Ron
>
> > On May 18, 2018, at 9:27 PM, Ron Dagostino  wrote:
> >
> > Hi committers.  KIP 255 still needs 1 more binding vote.  Currently
> there are two binding + 1 votes, from Rajini and Jun, and three non-binding
> +1 votes, from Mickael, Manikumar, and myself.
> >
> > Please vote by the Monday deadline.
> >
> > Ron
> >
> >> On Thu, May 17, 2018 at 10:59 AM, Ron Dagostino 
> wrote:
> >> Hi Jun.  I've updated the KIP to add a new section titled "Summary for
> Production Use" that includes this information along with a consolidated
> set of references to the applicable specifications.  Thanks for the
> questions.
> >>
> >> *We still need another binding vote* (currently there are two binding +
> 1 votes, from Rajini and Jun, and three non-binding +1 votes, from Mickael,
> Manikumar, and myself).
> >>
> >> Please vote before the May 22nd KIP Freeze deadline so this KIP can be
> included in the 2.0.0 release.
> >>
> >> A pull request is available and includes additional commits reflecting
> initial review comments: https://github.com/apache/kafka/pull/4994
> >>
> >> Ron
> >>
> >>> On Wed, May 16, 2018 at 8:09 PM, Jun Rao  wrote:
> >>> Hi, Ron,
> >>>
> >>> Thanks. I understand now. It may be useful to add a reference to JWT
> in the
> >>> KIP.
> >>>
> >>> Jun
> >>>
> >>> On Tue, May 15, 2018 at 6:51 PM, Ron Dagostino 
> wrote:
> >>>
> >>> > Hi Jun.  I think you are getting at the fact that OAuth 2 is a
> flexible
> >>> > framework that allows different installations to do things
> differently.  It
> >>> > is true that the principal name in Kafka could come from any claim
> in the
> >>> > token.  Most of the time it would come from the 'sub' claim, but it
> could
> >>> > certainly come from another claim, or it could be only indirectly
> based on
> >>> > a claim value (maybe certain text would be trimmed or
> prefixed/suffixed).
> >>> > The point, which I think you are getting at, is that because the
> framework
> >>> > is flexible we need to accommodate that flexibility.  The callback
> handler
> >>> > class defined by the listener.name.sasl_ssl.oauthbearer.sasl.server.
> >>> > callback.handler.class configuration value gives us the required
> >>> > flexibility.  As an example, I have an implementation that leverages
> a
> >>> > popular open source JOSE library to parse the compact serialization,
> >>> > retrieve the public key if it has not yet been retrieved, verify the
> >>> > digital signature, and map the 'sub' claim to the OAuthBearerToken's
> >>> > principal name (which becomes the SASL authorization ID, which
> becomes the
> >>> > Kafka principal name).  I could just as easily have mapped a
> different
> >>> > claim to the OAuthBearerToken's principal name, manipulated the
> 'sub' claim
> >>> > value in some way, etc.  I write the callback handler code, so I
> complete
> >>> > flexibility to do whatever my OAuth 2 installation requires me to do.
> >>> >
> >>> > Ron
> >>> >
> >>> > On Tue, May 15, 2018 at 1:39 PM, Jun Rao  wrote:
> >>> >
> >>> > > Hi, Ron,
> >>> > >
> >>> > > Thanks for the reply. I understood your answers to #2 and #3.
> >>> > >
> >>> > > For #1, will the server map all clients' principal name to the
> value
> >>> > > associated with "sub" claim? How do we support mapping different
> clients
> >>> > to
> >>> > > different principal names?
> >>> > >
> >>> > > Jun
> >>> > >
> >>> > > On Mon, May 14, 2018 at 7:02 PM, Ron Dagostino 
> >>> > wrote:
> >>> > >
> >>> > > > Hi Jun.  Thanks for the +1 vote.
> >>> > > >
> >>> > > > Regarding the first question about token claims, yes, you have it
> >>> > correct
> >>> > > > about translating the OAuth token to a principle name via a JAAS
> module
> >>> > > > option in the default unsecured case.  Specifically, the OAuth
> SASL
> >>> > > Server
> >>> > > > implementation is responsible for setting the authorization ID,
> and it
> >>> > > gets
> >>> > > > the authorization ID from the OAuthBearerToken's principalName()
> >>> > method.
> >>> > > > The listener.name.sasl_ssl.oauthbearer.sasl.server.
> >>> > > callback.handler.class
> >>> > > > is responsible for handling an instance of
> OAuthBearerValidatorCallback
> >>> > > to
> >>> > > > accept a token compact serialization from the client and return
> an
> >>> > > instance
> >>> > > > of OAuthBearerToken (assuming the compact serialization
> validates), and
> >>> > > in
> >>> > > > the default unsecured case the builtin unsecured validator
> callback
> >>> > > handler
> >>> > > > defines the OAuthBearerToken.principalName() method to return the
> >>> > 'sub'
> >>> > > > claim value by default (with the actual claim it uses being
> 

Re: [VOTE] KIP-290: Support for wildcard suffixed ACLs

[Damian Guy]

+1 (binding)

On Sat, 19 May 2018 at 03:51 Piyush Vijay  wrote:

> Hi everyone,
>
> I would like to start a vote for
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A+Support+for+wildcard+suffixed+ACLs
> .
>
> The KIP proposes a way to support wildcard-suffixed resource names in Kafka
> ACLs.
>
> The main challenge was to support it in a backward compatible way because
> resources like consumer groups don't have defined naming convention and can
> have '*' in their names.
>
> Please take a look.
>
> Thanks
>
> Piyush Vijay
>

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

[Damian Guy]

+1 (binding)

Thanks

On Mon, 21 May 2018 at 11:26 Edoardo Comar  wrote:

> Hi, KIP freeze is tomorrow. Anyone else would like to cast their vote ?
>
> voting so far :
> binding: Guozhang, Rajini
> non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt
>
> On 18 May 2018 at 09:38, Edoardo Comar  wrote:
> > bumping again as this KIP is still missing one binding vote ...
> >
> > voting so far :
> > binding: Guozhang, Rajini
> > non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt
> >
> > Edo
> >
> > On 17 May 2018 at 03:15, Matt Farmer  wrote:
> >> +1 (non-binding)
> >>
> >> On Tue, May 15, 2018 at 4:26 AM, Edoardo Comar 
> wrote:
> >>
> >>> Hi,
> >>> bumping the thread as the current vote count for this KIP is
> >>> 2 binding +1
> >>> 5 non-binding +1
> >>>
> >>> thanks, Edo
> >>>
> >>> On 8 May 2018 at 16:14, Edoardo Comar  wrote:
> >>> > Hi,
> >>> > bumping the thread as the current vote count for this KIP is
> >>> > 2 binding +1
> >>> > 5 non-binding +1
> >>> >
> >>> > so still missing a binding vote please
> >>> >
> >>> > thanks,
> >>> > Edo
> >>> >
> >>> >
> >>> > On 30 April 2018 at 12:49, Manikumar 
> wrote:
> >>> >>
> >>> >> +1 (non-binding)
> >>> >>
> >>> >> Thanks
> >>> >>
> >>> >> On Thu, Apr 26, 2018 at 9:59 PM, Colin McCabe 
> >>> wrote:
> >>> >>
> >>> >> > +1 (non-binding)
> >>> >> >
> >>> >> > best,
> >>> >> > Colin
> >>> >> >
> >>> >> >
> >>> >> > On Wed, Apr 25, 2018, at 02:45, Edoardo Comar wrote:
> >>> >> > > Hi,
> >>> >> > >
> >>> >> > > The discuss thread on KIP-277 (
> >>> >> > > https://www.mail-archive.com/dev@kafka.apache.org/msg86540.html
> )
> >>> >> > > seems to have been fruitful and concerns have been addressed,
> please
> >>> >> > allow
> >>> >> > > me start a vote on it:
> >>> >> > >
> >>> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> >> > 277+-+Fine+Grained+ACL+for+CreateTopics+API
> >>> >> > >
> >>> >> > > I will update the small PR to the latest KIP semantics if the
> vote
> >>> >> > passes
> >>> >> > > (as I hope :-).
> >>> >> > >
> >>> >> > > cheers
> >>> >> > > Edo
> >>> >> > > --
> >>> >> > >
> >>> >> > > Edoardo Comar
> >>> >> > >
> >>> >> > > IBM Message Hub
> >>> >> > >
> >>> >> > > IBM UK Ltd, Hursley Park, SO21 2JN
> >>> >> > > Unless stated otherwise above:
> >>> >> > > IBM United Kingdom Limited - Registered in England and Wales
> with
> >>> >> > > number
> >>> >> > > 741598.
> >>> >> > > Registered office: PO Box 41, North Harbour, Portsmouth,
> Hampshire
> >>> PO6
> >>> >> > 3AU
> >>> >> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > "When the people fear their government, there is tyranny; when the
> >>> > government fears the people, there is liberty." [Thomas Jefferson]
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> "When the people fear their government, there is tyranny; when the
> >>> government fears the people, there is liberty." [Thomas Jefferson]
> >>>
> >
> >
> >
> > --
> > "When the people fear their government, there is tyranny; when the
> > government fears the people, there is liberty." [Thomas Jefferson]
>
>
>
> --
> "When the people fear their government, there is tyranny; when the
> government fears the people, there is liberty." [Thomas Jefferson]
>

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

[Andy Coates]

Hey Piyush,

Thanks for the updated KIP! Couple of minor points from me:

When storing wildcard-suffixed Acls in ZK, drop the asterisk of the end for
the path, e.g. change "*/kafka-wildcard-acl/Topic/teamA*" * to "*/*
*kafka-wildcard-acl**/Topic/**teamA"*. This reduces error conditions, i.e.
this is a place for storing wildcard-suffixed Acls, so it implicitly ends
in an asterisk. If you include the asterisk in the path then you need to
validate each entry, when reading, ends with an asterisk, and do something
if they don't. If you don't include the training asterisk then there is
nothing to validate and you can treat the prefix as a literal, (i.e. no
escaping needed).  TBH I'd probably drop the asterisk from the in-memory
representation as well, for the same reason.

Rather than creating an enum to indicate the type of a resource, you could
instead use polymorphism, e.g. make Resource an interface and have two
implementations: LiteralResource and WildcardSuffixedResource.  This is
also extensible, but may also allow for a cleaner implementation.

Andy

On 21 May 2018 at 01:58, Rajini Sivaram  wrote:

> Hi Piyush, Thanks for the KIP!
>
> +1 (binding)
>
> Regards,
>
> Rajini
>
> On Sun, May 20, 2018 at 2:53 PM, Andy Coates  wrote:
>
> > Awesome last minute effort Piyush.
> >
> > Really appreciate your time and input,
> >
> > Andy
> >
> > Sent from my iPhone
> >
> > > On 19 May 2018, at 03:43, Piyush Vijay  wrote:
> > >
> > > Updated the KIP.
> > >
> > > 1. New enum field 'ResourceNameType' in Resource and ResourceFilter
> > classes.
> > > 2. modify getAcls() and rely on ResourceNameType' field in Resource to
> > > return either exact matches or all matches based on wildcard-suffix.
> > > 3. CLI changes to identify if resource name is literal or
> wildcard-suffix
> > > 4. Escaping doesn't work and isn't required if we're keeping a separate
> > > path on ZK (kafka-wildcard-acl) to store wildcard-suffix ACLs.
> > > 5. New API keys for Create / Delete / Describe Acls request with a new
> > > field in schemas for 'ResourceNameType'.
> > >
> > > Looks ready to me for the vote, will start voting thread now. Thanks
> > > everyone for the valuable feedback.
> > >
> > >
> > >
> > >
> > > Piyush Vijay
> > >
> > >
> > > Piyush Vijay
> > >
> > >> On Fri, May 18, 2018 at 6:07 PM, Andy Coates 
> wrote:
> > >>
> > >> Hi Piyush,
> > >>
> > >> We're fast approaching the KIP deadline. Are you actively working on
> > this?
> > >> If you're not I can take over.
> > >>
> > >> Thanks,
> > >>
> > >> Andy
> > >>
> > >>> On 18 May 2018 at 14:25, Andy Coates  wrote:
> > >>>
> > >>> OK I've read it now.
> > >>>
> > >>> 1. I see you have an example:
> >  For example: If I want to fetch all ACLs that match ’topicA*’, it’s
> > not
> > >>> possible without introducing new API AND maintaining backwards
> > >>> compatibility.
> > >>> getAcls takes a Resource, right, which would be either a full
> resource
> > >>> name or 'ALL', i.e. '*', right?  The point of the call is to get all
> > ACLs
> > >>> relating to a specific resource, not a partial resource like
> 'topicA*'.
> > >>> Currently, I'm guessing / half-remembering that if you ask it for
> ACLs
> > >> for
> > >>> topic 'foo' it doesn't include global 'ALL' ACLs in the list - that
> > would
> > >>> be a different call.  With the introduction of partial wildcards I
> > think
> > >>> the _most_ backwards compatible change would be to have
> > >>> getAcls("topic:foo") to return all the ACLs, including that affect
> this
> > >>> topic. This could include any '*'/ALL Acls, (which would be a small
> > >>> backwards compatible change), or exclude them as it current does.
> > >>> Excluding any matching partial wildcard acl, e.g. 'f*' would break
> > >>> compatibility IMHO.
> > >>>
> > >>> 2. Example command lines, showing how to add ACLs to specific
> resources
> > >>> that *end* with an asterisk char and adding wildcard-suffixed ACLs,
> > would
> > >>> really help clarify the KIP. e.g.
> > >>>
> > >>> bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:
> > >> 2181
> > >>> --add --allow-principal User:Bob --allow-principal User:Alice
> > >> --allow-host
> > >>> 198.51.100.0 --allow-host 198.51.100.1 --operation Read --group
> > my-app-*
> > >>>
> > >>> With the above command I can't see how the code can know if the user
> > >> means
> > >>> a literal group called 'my-app-*', or a wildcard suffix for any group
> > >>> starting with 'my-app-'. Escaping isn't enough as the escape char can
> > >> clash
> > >>> too, e.g. escaping a literal to 'my-app-\*' can still clash with
> > someone
> > >>> wanting a wildcard sufiix matching any group starting with
> 'my-app-\'.
> > >>>
> > >>> So there needs to be a syntax change here, I think.  Maybe some new
> > >>> command line switch to either explicitly enable or disable
> > >>> 'wildcard-suffix' support?  Probably defaulting 

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

[Ted Yu]

+1
 Original message From: Stephane Maarek 
 Date: 5/21/18  3:15 AM  (GMT-08:00) To: 
dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-290: Support for wildcard 
suffixed ACLs 
+1 non binding

On Mon., 21 May 2018, 2:44 pm Rajini Sivaram, 
wrote:

> Hi Piyush, Thanks for the KIP!
>
> +1 (binding)
>
> Regards,
>
> Rajini
>
> On Sun, May 20, 2018 at 2:53 PM, Andy Coates  wrote:
>
> > Awesome last minute effort Piyush.
> >
> > Really appreciate your time and input,
> >
> > Andy
> >
> > Sent from my iPhone
> >
> > > On 19 May 2018, at 03:43, Piyush Vijay  wrote:
> > >
> > > Updated the KIP.
> > >
> > > 1. New enum field 'ResourceNameType' in Resource and ResourceFilter
> > classes.
> > > 2. modify getAcls() and rely on ResourceNameType' field in Resource to
> > > return either exact matches or all matches based on wildcard-suffix.
> > > 3. CLI changes to identify if resource name is literal or
> wildcard-suffix
> > > 4. Escaping doesn't work and isn't required if we're keeping a separate
> > > path on ZK (kafka-wildcard-acl) to store wildcard-suffix ACLs.
> > > 5. New API keys for Create / Delete / Describe Acls request with a new
> > > field in schemas for 'ResourceNameType'.
> > >
> > > Looks ready to me for the vote, will start voting thread now. Thanks
> > > everyone for the valuable feedback.
> > >
> > >
> > >
> > >
> > > Piyush Vijay
> > >
> > >
> > > Piyush Vijay
> > >
> > >> On Fri, May 18, 2018 at 6:07 PM, Andy Coates 
> wrote:
> > >>
> > >> Hi Piyush,
> > >>
> > >> We're fast approaching the KIP deadline. Are you actively working on
> > this?
> > >> If you're not I can take over.
> > >>
> > >> Thanks,
> > >>
> > >> Andy
> > >>
> > >>> On 18 May 2018 at 14:25, Andy Coates  wrote:
> > >>>
> > >>> OK I've read it now.
> > >>>
> > >>> 1. I see you have an example:
> >  For example: If I want to fetch all ACLs that match ’topicA*’, it’s
> > not
> > >>> possible without introducing new API AND maintaining backwards
> > >>> compatibility.
> > >>> getAcls takes a Resource, right, which would be either a full
> resource
> > >>> name or 'ALL', i.e. '*', right?  The point of the call is to get all
> > ACLs
> > >>> relating to a specific resource, not a partial resource like
> 'topicA*'.
> > >>> Currently, I'm guessing / half-remembering that if you ask it for
> ACLs
> > >> for
> > >>> topic 'foo' it doesn't include global 'ALL' ACLs in the list - that
> > would
> > >>> be a different call.  With the introduction of partial wildcards I
> > think
> > >>> the _most_ backwards compatible change would be to have
> > >>> getAcls("topic:foo") to return all the ACLs, including that affect
> this
> > >>> topic. This could include any '*'/ALL Acls, (which would be a small
> > >>> backwards compatible change), or exclude them as it current does.
> > >>> Excluding any matching partial wildcard acl, e.g. 'f*' would break
> > >>> compatibility IMHO.
> > >>>
> > >>> 2. Example command lines, showing how to add ACLs to specific
> resources
> > >>> that *end* with an asterisk char and adding wildcard-suffixed ACLs,
> > would
> > >>> really help clarify the KIP. e.g.
> > >>>
> > >>> bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:
> > >> 2181
> > >>> --add --allow-principal User:Bob --allow-principal User:Alice
> > >> --allow-host
> > >>> 198.51.100.0 --allow-host 198.51.100.1 --operation Read --group
> > my-app-*
> > >>>
> > >>> With the above command I can't see how the code can know if the user
> > >> means
> > >>> a literal group called 'my-app-*', or a wildcard suffix for any group
> > >>> starting with 'my-app-'. Escaping isn't enough as the escape char can
> > >> clash
> > >>> too, e.g. escaping a literal to 'my-app-\*' can still clash with
> > someone
> > >>> wanting a wildcard sufiix matching any group starting with
> 'my-app-\'.
> > >>>
> > >>> So there needs to be a syntax change here, I think.  Maybe some new
> > >>> command line switch to either explicitly enable or disable
> > >>> 'wildcard-suffix' support?  Probably defaulting to wildcard-suffix
> > being
> > >>> on, (better experience going forward), though off is more backwards
> > >>> compatible.
> > >>>
> > >>>
> > >>> 3. Again, examples of how to store ACLs for specific resources that
> > >> *end* with
> > >>> an asterisk and wildcard-suffix ACLs, with any escaping would really
> > >> help.
> > >>>
> > >>>
> > >>>
> >  On 18 May 2018 at 13:55, Andy Coates  wrote:
> > 
> >  Hey Piyush,
> > 
> >  Thanks for getting this in! :D
> > 
> >  About to read now. But just quickly...
> > 
> >  1. I'll read up on the need for getMatchingAcls - but just playing
> > >> devils
> >  advocate for a moment - if a current caller of getAcls() expects it
> to
> >  return the full set of ACLs for a given resource, would post this
> > change

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

[Edoardo Comar]

Hi, KIP freeze is tomorrow. Anyone else would like to cast their vote ?

voting so far :
binding: Guozhang, Rajini
non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt

On 18 May 2018 at 09:38, Edoardo Comar  wrote:
> bumping again as this KIP is still missing one binding vote ...
>
> voting so far :
> binding: Guozhang, Rajini
> non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt
>
> Edo
>
> On 17 May 2018 at 03:15, Matt Farmer  wrote:
>> +1 (non-binding)
>>
>> On Tue, May 15, 2018 at 4:26 AM, Edoardo Comar  wrote:
>>
>>> Hi,
>>> bumping the thread as the current vote count for this KIP is
>>> 2 binding +1
>>> 5 non-binding +1
>>>
>>> thanks, Edo
>>>
>>> On 8 May 2018 at 16:14, Edoardo Comar  wrote:
>>> > Hi,
>>> > bumping the thread as the current vote count for this KIP is
>>> > 2 binding +1
>>> > 5 non-binding +1
>>> >
>>> > so still missing a binding vote please
>>> >
>>> > thanks,
>>> > Edo
>>> >
>>> >
>>> > On 30 April 2018 at 12:49, Manikumar  wrote:
>>> >>
>>> >> +1 (non-binding)
>>> >>
>>> >> Thanks
>>> >>
>>> >> On Thu, Apr 26, 2018 at 9:59 PM, Colin McCabe 
>>> wrote:
>>> >>
>>> >> > +1 (non-binding)
>>> >> >
>>> >> > best,
>>> >> > Colin
>>> >> >
>>> >> >
>>> >> > On Wed, Apr 25, 2018, at 02:45, Edoardo Comar wrote:
>>> >> > > Hi,
>>> >> > >
>>> >> > > The discuss thread on KIP-277 (
>>> >> > > https://www.mail-archive.com/dev@kafka.apache.org/msg86540.html )
>>> >> > > seems to have been fruitful and concerns have been addressed, please
>>> >> > allow
>>> >> > > me start a vote on it:
>>> >> > >
>>> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> >> > 277+-+Fine+Grained+ACL+for+CreateTopics+API
>>> >> > >
>>> >> > > I will update the small PR to the latest KIP semantics if the vote
>>> >> > passes
>>> >> > > (as I hope :-).
>>> >> > >
>>> >> > > cheers
>>> >> > > Edo
>>> >> > > --
>>> >> > >
>>> >> > > Edoardo Comar
>>> >> > >
>>> >> > > IBM Message Hub
>>> >> > >
>>> >> > > IBM UK Ltd, Hursley Park, SO21 2JN
>>> >> > > Unless stated otherwise above:
>>> >> > > IBM United Kingdom Limited - Registered in England and Wales with
>>> >> > > number
>>> >> > > 741598.
>>> >> > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
>>> PO6
>>> >> > 3AU
>>> >> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > "When the people fear their government, there is tyranny; when the
>>> > government fears the people, there is liberty." [Thomas Jefferson]
>>> >
>>>
>>>
>>>
>>> --
>>> "When the people fear their government, there is tyranny; when the
>>> government fears the people, there is liberty." [Thomas Jefferson]
>>>
>
>
>
> --
> "When the people fear their government, there is tyranny; when the
> government fears the people, there is liberty." [Thomas Jefferson]



-- 
"When the people fear their government, there is tyranny; when the
government fears the people, there is liberty." [Thomas Jefferson]

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

[Stephane Maarek]

+1 non binding

On Mon., 21 May 2018, 2:44 pm Rajini Sivaram, 
wrote:

> Hi Piyush, Thanks for the KIP!
>
> +1 (binding)
>
> Regards,
>
> Rajini
>
> On Sun, May 20, 2018 at 2:53 PM, Andy Coates  wrote:
>
> > Awesome last minute effort Piyush.
> >
> > Really appreciate your time and input,
> >
> > Andy
> >
> > Sent from my iPhone
> >
> > > On 19 May 2018, at 03:43, Piyush Vijay  wrote:
> > >
> > > Updated the KIP.
> > >
> > > 1. New enum field 'ResourceNameType' in Resource and ResourceFilter
> > classes.
> > > 2. modify getAcls() and rely on ResourceNameType' field in Resource to
> > > return either exact matches or all matches based on wildcard-suffix.
> > > 3. CLI changes to identify if resource name is literal or
> wildcard-suffix
> > > 4. Escaping doesn't work and isn't required if we're keeping a separate
> > > path on ZK (kafka-wildcard-acl) to store wildcard-suffix ACLs.
> > > 5. New API keys for Create / Delete / Describe Acls request with a new
> > > field in schemas for 'ResourceNameType'.
> > >
> > > Looks ready to me for the vote, will start voting thread now. Thanks
> > > everyone for the valuable feedback.
> > >
> > >
> > >
> > >
> > > Piyush Vijay
> > >
> > >
> > > Piyush Vijay
> > >
> > >> On Fri, May 18, 2018 at 6:07 PM, Andy Coates 
> wrote:
> > >>
> > >> Hi Piyush,
> > >>
> > >> We're fast approaching the KIP deadline. Are you actively working on
> > this?
> > >> If you're not I can take over.
> > >>
> > >> Thanks,
> > >>
> > >> Andy
> > >>
> > >>> On 18 May 2018 at 14:25, Andy Coates  wrote:
> > >>>
> > >>> OK I've read it now.
> > >>>
> > >>> 1. I see you have an example:
> >  For example: If I want to fetch all ACLs that match ’topicA*’, it’s
> > not
> > >>> possible without introducing new API AND maintaining backwards
> > >>> compatibility.
> > >>> getAcls takes a Resource, right, which would be either a full
> resource
> > >>> name or 'ALL', i.e. '*', right?  The point of the call is to get all
> > ACLs
> > >>> relating to a specific resource, not a partial resource like
> 'topicA*'.
> > >>> Currently, I'm guessing / half-remembering that if you ask it for
> ACLs
> > >> for
> > >>> topic 'foo' it doesn't include global 'ALL' ACLs in the list - that
> > would
> > >>> be a different call.  With the introduction of partial wildcards I
> > think
> > >>> the _most_ backwards compatible change would be to have
> > >>> getAcls("topic:foo") to return all the ACLs, including that affect
> this
> > >>> topic. This could include any '*'/ALL Acls, (which would be a small
> > >>> backwards compatible change), or exclude them as it current does.
> > >>> Excluding any matching partial wildcard acl, e.g. 'f*' would break
> > >>> compatibility IMHO.
> > >>>
> > >>> 2. Example command lines, showing how to add ACLs to specific
> resources
> > >>> that *end* with an asterisk char and adding wildcard-suffixed ACLs,
> > would
> > >>> really help clarify the KIP. e.g.
> > >>>
> > >>> bin/kafka-acls.sh --authorizer-properties
> zookeeper.connect=localhost:
> > >> 2181
> > >>> --add --allow-principal User:Bob --allow-principal User:Alice
> > >> --allow-host
> > >>> 198.51.100.0 --allow-host 198.51.100.1 --operation Read --group
> > my-app-*
> > >>>
> > >>> With the above command I can't see how the code can know if the user
> > >> means
> > >>> a literal group called 'my-app-*', or a wildcard suffix for any group
> > >>> starting with 'my-app-'. Escaping isn't enough as the escape char can
> > >> clash
> > >>> too, e.g. escaping a literal to 'my-app-\*' can still clash with
> > someone
> > >>> wanting a wildcard sufiix matching any group starting with
> 'my-app-\'.
> > >>>
> > >>> So there needs to be a syntax change here, I think.  Maybe some new
> > >>> command line switch to either explicitly enable or disable
> > >>> 'wildcard-suffix' support?  Probably defaulting to wildcard-suffix
> > being
> > >>> on, (better experience going forward), though off is more backwards
> > >>> compatible.
> > >>>
> > >>>
> > >>> 3. Again, examples of how to store ACLs for specific resources that
> > >> *end* with
> > >>> an asterisk and wildcard-suffix ACLs, with any escaping would really
> > >> help.
> > >>>
> > >>>
> > >>>
> >  On 18 May 2018 at 13:55, Andy Coates  wrote:
> > 
> >  Hey Piyush,
> > 
> >  Thanks for getting this in! :D
> > 
> >  About to read now. But just quickly...
> > 
> >  1. I'll read up on the need for getMatchingAcls - but just playing
> > >> devils
> >  advocate for a moment - if a current caller of getAcls() expects it
> to
> >  return the full set of ACLs for a given resource, would post this
> > change
> >  only returning a sub set and requiring them to return
> getMatchingAcls
> > to
> >  get the full set not itself be a break in compatibility? I'm
> thinking
> > >> about
> >  any tooling / UI / etc people 

RE: [VOTE] KIP-235 Add DNS alias support for secured connection

[Skrzypek, Jonathan]

Hi,

What would be the next step here ?
I know there's a discussion going on around KIP-302, but I'm also conscious 
that the 2.0.0 deadline for KIPs is tomorrow.
I've opened this KIP in January and discussions have been productive with an 
end solution I had the impression was reasonable, so I am keen to see it make 
it the next release.


Jonathan Skrzypek

-Original Message-
From: Skrzypek, Jonathan [Tech]
Sent: 14 May 2018 13:48
To: dev
Subject: RE: [VOTE] KIP-235 Add DNS alias support for secured connection

Sure, I modified the KIP to add more details

https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection


Jonathan Skrzypek


-Original Message-
From: Ismael Juma [mailto:ism...@juma.me.uk]
Sent: 14 May 2018 11:53
To: dev
Subject: Re: [VOTE] KIP-235 Add DNS alias support for secured connection

Thanks for the KIP, Jonathan. It would be helpful to have more detail on
how SSL authentication could be broken if the new behaviour is the default.
I know this was discussed in the mailing list thread, but it's important to
include it in the KIP since it's the main reason why a new config is needed
(and configs should be avoided whenever we can just do the right thing).

Ismael

On Fri, Mar 23, 2018 at 12:05 PM Skrzypek, Jonathan <
jonathan.skrzy...@gs.com> wrote:

> Hi,
>
> I would like to start a vote for KIP-235
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D235-253A-2BAdd-2BDNS-2Balias-2Bsupport-2Bfor-2Bsecured-2Bconnection=DwIBaQ=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4=nNmJlu1rR_QFAPdxGlafmDu9_r6eaCbPOM0NM1EHo-E=FM_uCHnnO2dqxWC0bi7_QOJKfKmQI80-Xduvb-URWOw=RpGkijfK-WHcU0s8ZtMXEkIr69QraJhYKaGSC9V_rnI=
>
> This is a proposition to add an option for reverse dns lookup of
> bootstrap.servers hosts, allowing the use of dns aliases on clusters using
> SASL authentication.
>
>
>
>



Your Personal Data: We may collect and process information about you that may 
be subject to data protection laws. For more information about how we use and 
disclose your personal data, how we protect your information, our legal basis 
to use your information, your rights and who you can contact, please refer to: 
www.gs.com/privacy-notices

Re: [DISCUSS] Apache Kafka 2.0.0 Release Plan

[Rajini Sivaram]

Hi all,

This is a reminder that KIP freeze for 2.0.0 release is tomorrow. KIPs that
are not yet in voting stage will be postponed to the next release. We have
several KIPs that need more votes to be accepted. Please participate in
reviews and votes to enable these to be added to the release (or postponed
if required). We need more committers to review and vote since binding
votes are needed by tomorrow to include these in the release, but reviews
and votes from everyone in the community are welcome.

KIPs that require one more binding vote:

   - KIP-235: https://cwiki.apache.org/confluence/display/KAFKA/K
   IP-235%3A+Add+DNS+alias+support+for+secured+connection
   

   - KIP-255: https://cwiki.apache.org/confluence/pages/viewpage.
   action?pageId=75968876
   - KIP-277: https://cwiki.apache.org/confluence/display/KAFKA/K
   IP-277+-+Fine+Grained+ACL+for+CreateTopics+API
   

   - KIP-285: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
   285%3A+Connect+Rest+Extension+Plugin
   

   - KIP-297: https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%
   3A+Externalizing+Secrets+for+Connect+Configurations
   


KIPs that require two more binding votes:

   - KIP-248: https://cwiki.apache.org/confluence/display/KAFKA/K
   IP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
   

   - KIP-290: https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A
   +Support+for+wildcard+suffixed+ACLs
   

   - KIP-298: https://cwiki.apache.org/confluence/display/KAFKA/K
   IP-298%3A+Error+Handling+in+Connect
   


KIPs that require three binding votes


   - KIP-275: https://cwiki.apache.org/confluence/pages/viewpage.
   action?pageId=75977607


Thanks,

Rajini

On Tue, May 15, 2018 at 11:23 AM, Rajini Sivaram 
wrote:

> Hi all,
>
> This is just a reminder that KIP freeze for 2.0.0 release is in a week's
> time and we still have a lot of KIPs in progress. KIPs that are currently
> being discussed should start the voting process soon to get voting complete
> by 22nd of May. Please participate in discussions and votes to enable
> these to be added to the release (or postponed if required).
>
> Voting is in progress for the following KIPs:
>
>
>- KIP-235: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>235%3A+Add+DNS+alias+support+for+secured+connection
>
> 
>- KIP-244: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
>
> 
>- KIP-248: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
>
> 
>- KIP-255: https://cwiki.apache.org/confluence/pages/viewpage.
>action?pageId=75968876
>- KIP-275: https://cwiki.apache.org/confluence/pages/viewpage.
>action?pageId=75977607
>- KIP-277: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>277+-+Fine+Grained+ACL+for+CreateTopics+API
>
> 
>- KIP-278: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>278+-+Add+version+option+to+Kafka%27s+commands
>
> 
>- KIP-282: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>282%3A+Add+the+listener+name+to+the+authentication+context
>
> 
>- KIP-283: https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>283%3A+Efficient+Memory+Usage+for+Down-Conversion
>
> 
>
> Regards,
>
> Rajini
>
> On Thu, May 10, 2018 at 9:31 AM, Rajini Sivaram 
> wrote:
>
>> Hi all,
>>
>> A reminder that 

[jira] [Created] (KAFKA-6925) Memory leak in org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl

[Marcin Kuthan (JIRA)]

Marcin Kuthan created KAFKA-6925:


 Summary: Memory leak in 
org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl
 Key: KAFKA-6925
 URL: https://issues.apache.org/jira/browse/KAFKA-6925
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 1.0.1
Reporter: Marcin Kuthan


The retained heap of 
org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl
 is surprisingly high for long running job. Over 100MB of heap for every stream 
after a week of uptime, when for the same application a few hours after start 
takes 2MB.

For the problematic instance majority of memory StreamsMetricsThreadImpl is 
occupied by hash map entries in parentSensors, over 8000 elements 100+kB each. 
For fresh instance there are less than 200 elements.

Below you could find retained set report generated from Eclipse Mat but I'm not 
fully sure about correctness due to complex object graph in the metrics related 
code.

 
{code:java}
Class Name | Objects | Shallow Heap
---
org.apache.kafka.common.metrics.KafkaMetric | 140,476 | 4,495,232
org.apache.kafka.common.MetricName | 140,476 | 4,495,232
org.apache.kafka.common.metrics.stats.SampledStat$Sample | 73,599 | 3,532,752
org.apache.kafka.common.metrics.stats.Meter | 42,104 | 1,347,328
org.apache.kafka.common.metrics.stats.Count | 42,104 | 1,347,328
org.apache.kafka.common.metrics.stats.Rate | 42,104 | 1,010,496
org.apache.kafka.common.metrics.stats.Total | 42,104 | 1,010,496
org.apache.kafka.common.metrics.stats.Max | 28,134 | 900,288
org.apache.kafka.common.metrics.stats.Avg | 28,134 | 900,288
org.apache.kafka.common.metrics.Sensor | 3,164 | 202,496
org.apache.kafka.common.metrics.Sensor[] | 3,164 | 71,088
org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl|
 1 | 56
---
{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-278: Add version option to Kafka's commands

[Rajini Sivaram]

Hi Sasaki, Thanks for the KIP!

+1 (binding)

On Mon, May 14, 2018 at 4:30 PM, Sasaki Toru 
wrote:

> Rajini,
>
> Thank you for your appropriate advice, I misunderstood some non-binding
> votes.
> I get this KIP back to the state waiting to vote.
>
>
> Thanks,
> Sasaki
>
>
> 2018-05-14 20:03 GMT+09:00 Mickael Maison :
>>
>> +1 (non binding)
>>> Thanks for the KIP
>>>
>>> On Mon, May 14, 2018 at 10:36 AM, Rajini Sivaram
>>>  wrote:
>>>
 Sasaki,

 I believe this has 2 binding votes and six non-binding votes, so needs
 another binding vote to pass.

 On Mon, May 14, 2018 at 9:00 AM, Sasaki Toru 
 wrote:

 Hi all,
>
> This KIP would be accepted with 4 binding and 4 non-binding votes.
>
> Thank you for participating.
>
>
> Thanks,
> Sasaki.
>
> 2018-05-12 9:31 GMT+09:00 Colin McCabe :
>
>> +1 (non-binding)
>>
>>> Colin
>>>
>>> On Fri, May 11, 2018, at 12:35, Attila Sasvári wrote:
>>>
>>> +1 (non-binding)

 Thomas Crayford  ezt írta (időpont: 2018.
 máj.
 11., P 18:20):

 +1 (non-binding)

> On Fri, May 11, 2018 at 5:17 PM, Guozhang Wang  >
>
> wrote:
 Thanks Toru-san, +1 (binding)

> On Fri, May 11, 2018 at 8:30 AM, Jason Gustafson <
>>
> ja...@confluent.io
>>>
 wrote:
>>
>> Thanks for the KIP! +1 (binding)
>>
>>> On Fri, May 11, 2018 at 12:35 AM, Manikumar <
>>>
>>> manikumar.re...@gmail.com
>>
> wrote:

> +1 (non-binding)
>>>
 Thanks for the KIP.

 On Fri, May 11, 2018 at 12:56 PM, zhenya Sun 

 wrote:
>>>
>> +1 building
>>
>>> 在 2018年5月11日，上午9:51，Ted Yu  写道：
>>
>> +1
>>
>> On Thu, May 10, 2018 at 6:42 PM, Sasaki Toru <
>>
>> sasaki...@oss.nttdata.com>
> wrote:
>
>> Hi all,
>>
>>> I would like to start the vote on KIP-278: Add version
>>>
>>> option to
>>
> Kafka's

> commands.
>
>> The link to this KIP is here:
>>> >>
>>> 278+-
>>
> +Add+version+option+to+Kafka%27s+commands>

> The discussion thread is here:
>>> <
>>>
>>> https://www.mail-archive.com/dev@kafka.apache.org/msg86688.
>>
> html>
>>>
 Many thanks,
>>> Sasaki
>>>
>>> --
>>> Sasaki Toru(sasaki...@oss.nttdata.com) NTT DATA CORPORATION
>>>
>>>
>>>
>>> --
>> -- Guozhang
>>
>>
>> --
> Sasaki Toru(sasaki...@oss.nttdata.com) NTT DATA CORPORATION
>
>
>
> --
> Sasaki Toru(sasaki...@oss.nttdata.com) NTT DATA CORPORATION
>
>

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

[Rajini Sivaram]

Hi Piyush, Thanks for the KIP!

+1 (binding)

Regards,

Rajini

On Sun, May 20, 2018 at 2:53 PM, Andy Coates  wrote:

> Awesome last minute effort Piyush.
>
> Really appreciate your time and input,
>
> Andy
>
> Sent from my iPhone
>
> > On 19 May 2018, at 03:43, Piyush Vijay  wrote:
> >
> > Updated the KIP.
> >
> > 1. New enum field 'ResourceNameType' in Resource and ResourceFilter
> classes.
> > 2. modify getAcls() and rely on ResourceNameType' field in Resource to
> > return either exact matches or all matches based on wildcard-suffix.
> > 3. CLI changes to identify if resource name is literal or wildcard-suffix
> > 4. Escaping doesn't work and isn't required if we're keeping a separate
> > path on ZK (kafka-wildcard-acl) to store wildcard-suffix ACLs.
> > 5. New API keys for Create / Delete / Describe Acls request with a new
> > field in schemas for 'ResourceNameType'.
> >
> > Looks ready to me for the vote, will start voting thread now. Thanks
> > everyone for the valuable feedback.
> >
> >
> >
> >
> > Piyush Vijay
> >
> >
> > Piyush Vijay
> >
> >> On Fri, May 18, 2018 at 6:07 PM, Andy Coates  wrote:
> >>
> >> Hi Piyush,
> >>
> >> We're fast approaching the KIP deadline. Are you actively working on
> this?
> >> If you're not I can take over.
> >>
> >> Thanks,
> >>
> >> Andy
> >>
> >>> On 18 May 2018 at 14:25, Andy Coates  wrote:
> >>>
> >>> OK I've read it now.
> >>>
> >>> 1. I see you have an example:
>  For example: If I want to fetch all ACLs that match ’topicA*’, it’s
> not
> >>> possible without introducing new API AND maintaining backwards
> >>> compatibility.
> >>> getAcls takes a Resource, right, which would be either a full resource
> >>> name or 'ALL', i.e. '*', right?  The point of the call is to get all
> ACLs
> >>> relating to a specific resource, not a partial resource like 'topicA*'.
> >>> Currently, I'm guessing / half-remembering that if you ask it for ACLs
> >> for
> >>> topic 'foo' it doesn't include global 'ALL' ACLs in the list - that
> would
> >>> be a different call.  With the introduction of partial wildcards I
> think
> >>> the _most_ backwards compatible change would be to have
> >>> getAcls("topic:foo") to return all the ACLs, including that affect this
> >>> topic. This could include any '*'/ALL Acls, (which would be a small
> >>> backwards compatible change), or exclude them as it current does.
> >>> Excluding any matching partial wildcard acl, e.g. 'f*' would break
> >>> compatibility IMHO.
> >>>
> >>> 2. Example command lines, showing how to add ACLs to specific resources
> >>> that *end* with an asterisk char and adding wildcard-suffixed ACLs,
> would
> >>> really help clarify the KIP. e.g.
> >>>
> >>> bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:
> >> 2181
> >>> --add --allow-principal User:Bob --allow-principal User:Alice
> >> --allow-host
> >>> 198.51.100.0 --allow-host 198.51.100.1 --operation Read --group
> my-app-*
> >>>
> >>> With the above command I can't see how the code can know if the user
> >> means
> >>> a literal group called 'my-app-*', or a wildcard suffix for any group
> >>> starting with 'my-app-'. Escaping isn't enough as the escape char can
> >> clash
> >>> too, e.g. escaping a literal to 'my-app-\*' can still clash with
> someone
> >>> wanting a wildcard sufiix matching any group starting with 'my-app-\'.
> >>>
> >>> So there needs to be a syntax change here, I think.  Maybe some new
> >>> command line switch to either explicitly enable or disable
> >>> 'wildcard-suffix' support?  Probably defaulting to wildcard-suffix
> being
> >>> on, (better experience going forward), though off is more backwards
> >>> compatible.
> >>>
> >>>
> >>> 3. Again, examples of how to store ACLs for specific resources that
> >> *end* with
> >>> an asterisk and wildcard-suffix ACLs, with any escaping would really
> >> help.
> >>>
> >>>
> >>>
>  On 18 May 2018 at 13:55, Andy Coates  wrote:
> 
>  Hey Piyush,
> 
>  Thanks for getting this in! :D
> 
>  About to read now. But just quickly...
> 
>  1. I'll read up on the need for getMatchingAcls - but just playing
> >> devils
>  advocate for a moment - if a current caller of getAcls() expects it to
>  return the full set of ACLs for a given resource, would post this
> change
>  only returning a sub set and requiring them to return getMatchingAcls
> to
>  get the full set not itself be a break in compatibility? I'm thinking
> >> about
>  any tooling / UI / etc people may have built on top of this.  If Im
> >> missing
>  the point, then maybe a concrete example, (if you've not already added
> >> one
>  to the doc), may help.
> 
>  2. Something must change on the command line, surely? e.g. as command
>  line user how would the command differ if I wanted to add an ACL onto
> a
>  group called 'foo*' as opposed to a all groups 

Kafka consumer to unzip stream of .gz files and read

[mayur shah]

 HI Team,

Greeting!

I am facing one issue on kafka consumer using python hope you guys help us
to resolve this issue

Kafka consumer to unzip stream of .gz files and read


Kafka producer is sending .gz files but not able to decompress and read the
files at the consumer end. Getting error as "IOError: Not a gzipped file"

Producer -

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic Airport
< ~/Downloads/stocks.json.gz

Consumer -

import sys import gzipimport StringIOfrom kafka import KafkaConsumer

consumer = KafkaConsumer(KAFKA_TOPIC, bootstrap_servers=KAFKA_BROKERS)
try:
for message in consumer:
f = StringIO.StringIO(message.value)
gzip_f = gzip.GzipFile(fileobj=f)
unzipped_content = gzip_f.read()
content = unzipped_content.decode('utf8')
print (content)except KeyboardInterrupt:
sys.exit()

Error at consumer -

Traceback (most recent call last):
  File "consumer.py", line 18, in 
unzipped_content = gzip_f.read()
  File "/usr/lib64/python2.6/gzip.py", line 212, in read
self._read(readsize)
  File "/usr/lib64/python2.6/gzip.py", line 255, in _read
self._read_gzip_header()
  File "/usr/lib64/python2.6/gzip.py", line 156, in _read_gzip_header
raise IOError, 'Not a gzipped file'IOError: Not a gzipped file

Regards,
Mayur

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

[Viktor Somogyi]

Hi All,

I'd like to ask the community to please vote for this as the KIP
freeze is tomorrow.

Thank you very much,
Viktor

On Mon, May 21, 2018 at 9:39 AM, Viktor Somogyi  wrote:
> Hi Colin,
>
> Sure, I'll add a note.
> Thanks for your vote.
>
> Viktor
>
> On Sat, May 19, 2018 at 1:01 AM, Colin McCabe  wrote:
>> Hi Viktor,
>>
>> Thanks, this looks good.
>>
>> The boolean should default to false if not set, to ensure that existing 
>> clients continue to work as-is, right?  Might be good to add a note 
>> specifying that.
>>
>> +1 (non-binding)
>>
>> best,
>> Colin
>>
>> On Fri, May 18, 2018, at 08:16, Viktor Somogyi wrote:
>>> Updated KIP-248:
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
>>>
>>> I'd like to ask project members, committers and contributors to vote
>>> as this would be a useful improvement in Kafka.
>>>
>>> Sections changed:
>>> - Public interfaces: added the bin/scram-credentials.sh command that
>>> we discussed with Colin.
>>> - Wire format types: removed AlterOperations. As discussed with Colin,
>>> we don't actually need this: we should behave in an incremental way in
>>> AlterQuotas. For AlterConfig we'll implement this behavior with an
>>> extra flag on the protocol (and incrementing the version).
>>> - AlterQuotas protocol: removed AlterOperations. Added some more
>>> description to the behavior of the protocol. Removing quotas will
>>> happen by sending a NaN instead of the AlterOperations. (Since IEEE
>>> 754 covers NaNs and it is not a valid config for any quota, I think it
>>> is a good notation.)
>>> - SCRAM: so it will be done by the scram-credentials command that uses
>>> direct zookeeper connection. I think further modes, like doing it
>>> through the broker is not necessary. The idea here is that zookeeper
>>> in this case acts as a credentials store. This should be decoupled
>>> from the broker as we manage broker credentials as well. The new
>>> command acts as a client to the store.
>>> - AlterConfigs will have an incremental_update flag as discussed. By
>>> default it is false to provide the backward compatible behavior. When
>>> it is true it will merge the configs with what's there in the node.
>>> Deletion in incremental mode is done by sending an empty string as
>>> config value.
>>> - Other compatibility changes: this KIP doesn't scope listing multiple
>>> users and client's quotas. As per a conversation with Rajini, it is
>>> not a common use case and we can add it back later if it is needed. If
>>> this functionality is needed, the old code should be still available
>>> through run-kafka-class. (Removed the USE_OLD_KAFKA_CONFIG_COMMAND as
>>> it doesn't make sense anymore.)
>>>
>>> On Fri, May 18, 2018 at 12:33 PM, Viktor Somogyi
>>>  wrote:
>>> > Ok, ignore my previous mail (except the last sentence), gmail didn't
>>> > update me about your last email :/.
>>> >
>>> >> I think we should probably just create a flag for alterConfigs which 
>>> >> marks it as incremental, like we discussed earlier, and do this as a 
>>> >> compatible change that is needed for the shell command.
>>> >
>>> > Alright, I missed that about the sensitive configs too, so in this
>>> > case I can agree with this. I'll update the KIP this afternoon and
>>> > update this thread.
>>> > Thanks again for your contribution.
>>> >
>>> > Viktor
>>> >
>>> > On Fri, May 18, 2018 at 2:34 AM, Colin McCabe  wrote:
>>> >> Actually, I just realized that this won't work.  The AlterConfigs API is 
>>> >> kind of broken right now.  DescribeConfigs won't return the "sensitive" 
>>> >> configurations like passwords.  So doing describe + edit + alter will 
>>> >> wipe out all sensitive configs. :(
>>> >>
>>> >> I think we should probably just create a flag for alterConfigs which 
>>> >> marks it as incremental, like we discussed earlier, and do this as a 
>>> >> compatible change that is needed for the shell command.
>>> >>
>>> >> best,
>>> >> Colin
>>> >>
>>> >>
>>> >> On Thu, May 17, 2018, at 09:32, Colin McCabe wrote:
>>> >>> Hi Viktor,
>>> >>>
>>> >>> Since the KIP freeze is coming up really soon, maybe we should just drop
>>> >>> the section about changes to AlterConfigs from KIP-248.  We don't really
>>> >>> need it here, since ConfigCommand can use AlterConfigs as-is.
>>> >>>
>>> >>> We can pick up the discussion about improving AlterConfigs in a future 
>>> >>> KIP.
>>> >>>
>>> >>> cheers,
>>> >>> Colin
>>> >>>
>>> >>> On Wed, May 16, 2018, at 22:06, Colin McCabe wrote:
>>> >>> > Hi Viktor,
>>> >>> >
>>> >>> > The shell command isn’t that easy to integrate into applications.
>>> >>> > AdminClient will get integrated  into a lot more stuff, which
>>> >>> > increases the potential for conflicts.  I would argue that we should
>>> >>> > fix this soon.
>>> >>> > If we do want to reduce the scope in this KIP, we could do the merge 
>>> >>> > in
>>> 

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

[Viktor Somogyi]

Hi Colin,

Sure, I'll add a note.
Thanks for your vote.

Viktor

On Sat, May 19, 2018 at 1:01 AM, Colin McCabe  wrote:
> Hi Viktor,
>
> Thanks, this looks good.
>
> The boolean should default to false if not set, to ensure that existing 
> clients continue to work as-is, right?  Might be good to add a note 
> specifying that.
>
> +1 (non-binding)
>
> best,
> Colin
>
> On Fri, May 18, 2018, at 08:16, Viktor Somogyi wrote:
>> Updated KIP-248:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient
>>
>> I'd like to ask project members, committers and contributors to vote
>> as this would be a useful improvement in Kafka.
>>
>> Sections changed:
>> - Public interfaces: added the bin/scram-credentials.sh command that
>> we discussed with Colin.
>> - Wire format types: removed AlterOperations. As discussed with Colin,
>> we don't actually need this: we should behave in an incremental way in
>> AlterQuotas. For AlterConfig we'll implement this behavior with an
>> extra flag on the protocol (and incrementing the version).
>> - AlterQuotas protocol: removed AlterOperations. Added some more
>> description to the behavior of the protocol. Removing quotas will
>> happen by sending a NaN instead of the AlterOperations. (Since IEEE
>> 754 covers NaNs and it is not a valid config for any quota, I think it
>> is a good notation.)
>> - SCRAM: so it will be done by the scram-credentials command that uses
>> direct zookeeper connection. I think further modes, like doing it
>> through the broker is not necessary. The idea here is that zookeeper
>> in this case acts as a credentials store. This should be decoupled
>> from the broker as we manage broker credentials as well. The new
>> command acts as a client to the store.
>> - AlterConfigs will have an incremental_update flag as discussed. By
>> default it is false to provide the backward compatible behavior. When
>> it is true it will merge the configs with what's there in the node.
>> Deletion in incremental mode is done by sending an empty string as
>> config value.
>> - Other compatibility changes: this KIP doesn't scope listing multiple
>> users and client's quotas. As per a conversation with Rajini, it is
>> not a common use case and we can add it back later if it is needed. If
>> this functionality is needed, the old code should be still available
>> through run-kafka-class. (Removed the USE_OLD_KAFKA_CONFIG_COMMAND as
>> it doesn't make sense anymore.)
>>
>> On Fri, May 18, 2018 at 12:33 PM, Viktor Somogyi
>>  wrote:
>> > Ok, ignore my previous mail (except the last sentence), gmail didn't
>> > update me about your last email :/.
>> >
>> >> I think we should probably just create a flag for alterConfigs which 
>> >> marks it as incremental, like we discussed earlier, and do this as a 
>> >> compatible change that is needed for the shell command.
>> >
>> > Alright, I missed that about the sensitive configs too, so in this
>> > case I can agree with this. I'll update the KIP this afternoon and
>> > update this thread.
>> > Thanks again for your contribution.
>> >
>> > Viktor
>> >
>> > On Fri, May 18, 2018 at 2:34 AM, Colin McCabe  wrote:
>> >> Actually, I just realized that this won't work.  The AlterConfigs API is 
>> >> kind of broken right now.  DescribeConfigs won't return the "sensitive" 
>> >> configurations like passwords.  So doing describe + edit + alter will 
>> >> wipe out all sensitive configs. :(
>> >>
>> >> I think we should probably just create a flag for alterConfigs which 
>> >> marks it as incremental, like we discussed earlier, and do this as a 
>> >> compatible change that is needed for the shell command.
>> >>
>> >> best,
>> >> Colin
>> >>
>> >>
>> >> On Thu, May 17, 2018, at 09:32, Colin McCabe wrote:
>> >>> Hi Viktor,
>> >>>
>> >>> Since the KIP freeze is coming up really soon, maybe we should just drop
>> >>> the section about changes to AlterConfigs from KIP-248.  We don't really
>> >>> need it here, since ConfigCommand can use AlterConfigs as-is.
>> >>>
>> >>> We can pick up the discussion about improving AlterConfigs in a future 
>> >>> KIP.
>> >>>
>> >>> cheers,
>> >>> Colin
>> >>>
>> >>> On Wed, May 16, 2018, at 22:06, Colin McCabe wrote:
>> >>> > Hi Viktor,
>> >>> >
>> >>> > The shell command isn’t that easy to integrate into applications.
>> >>> > AdminClient will get integrated  into a lot more stuff, which
>> >>> > increases the potential for conflicts.  I would argue that we should
>> >>> > fix this soon.
>> >>> > If we do want to reduce the scope in this KIP, we could do the merge in
>> >>> > the ConfigCommand  tool for now, and leave AC unchanged.
>> >>> > Best,
>> >>> > Colin
>> >>> >
>> >>> >
>> >>> > On Wed, May 16, 2018, at 04:57, Viktor Somogyi wrote:
>> >>> > > Hi Colin,
>> >>> > >
>> >>> > > > Doing get-merge-set is buggy, though.  If someone else does 
>> >>> > > > get-merge-
>> >>> > > > set at the