Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

t; For > > >> > > > >>> adding new > > >> > > > >>> > > partitions request, this is proposed replica assignment > for > > >> new > > >> > > > >>> partitions. > > >> > > > >>> > > For replica re-assignment case, this is proposed new > > >> >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

AlterTopicRequest interface > >> > > > >>> proposal, but > >> > > > >>> > > the idea is basically to return only the parts which were > >> > > changed. > >> > > > >>

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

;> > >> be >> > > > >>> > >> > much better for the policy writer to just be able to >> write: >> > > > >>> > >> > >> > > > >>> > >> &g

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

t; > > > >>> > >> > > > > >>> > >> If no further comments a forthcoming in the next day or two > > > then I > > > > >>> will > > > > >>> > >> start a vote. > > > > >>> > >> > > > > >>&g

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

gt; > >> > should behave. > > > >>> > >> > > > > >>> > >> > The current CreateTopicPolicy gets passed either the request > > > >>> partition > > > >>> > >> > count and replication factor, or the requested assignm

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

>> > >> >>> > > com_ppatierno=DwIBaQ=jf_iaSHvJObTbx-siA1ZOg= > > >>> > >> >>> > > > > >>> EzRhmSah4IHsUZVekRUIINhltZK7U0OaeRo7hgW4_tQ

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

so less symmetric between its use for > >>> createTopic() > >>> > >> and > >>> > >> > for alterTopic(). This can make it harder to write a policy. For > >>> > >> example, > >>> > >> > if I want the policy &q

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

> would >>> > >> be >>> > >> > much better for the policy writer to just be able to write: >>> > >> > >>> > >> > if (request.requestedState().numPartitions() >= 100) >>> > >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

t; >> >>> > wrote: >> > >> >>> > > >> > >> >>> > > > Hi Ismael, >> > >> >>> > > > >> > >> >>> > > > >> > >> >>> > > >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

te: > > >> > > > >> >> Good point. Then I guess I can do those items too. I would also > need to > > >> >> do the same changes for DeleteRecordsRequest and Response. > > >> >> > > >> >> On 4 October 2017 at 10:37,

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

licy that can't send errors to clients is much less useful > >> >>> 2. Testing policies is much easier with `validateOnly` > >> >>> > >> >>> Ismael > >> >>> > >> >>> On Wed, Oct 4, 2017 at 9:20 AM, Tom Bentley > >> >>> wrote: > >>

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

ding an error message to >> DeleteTopicResponse. >> >>> > Since those are not policy-related I think they're best left out of >> >>> > KIP-201. I suppose it is up to you and Mickael whether to narrow the >> >>> scope >> >>> > o

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

t; > > > >>> > > Tom > >>> > > > >>> > > On 27 September 2017 at 11:45, Paolo Patierno > >>> > wrote: > >>> > > > >>> > > > Hi Ismael, > >>> > > > > >>> > > > > >>> > > > 1. I don't have a real requirement

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

ivation bullet that was behind >>> > > KIP-170: >>> > > >>> > > introducing ClusterState to allow validation of create/alter topic >>> > request >>> > > >>> > > not just against the request metadata but al

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

ion of create/alter topic >> > request >> > > >> > > not just against the request metadata but also >> > > against the current amount of resources already used in the cluster >> (eg >> > > number of partitions). >> > > >>

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

--------------------- > > > > > > Edoardo Comar > > > > > > IBM Message Hub > > > > > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > > > > > > > > From: Tom Bentley <t.j.bent...@gmail.com> >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

related to the offset > for > > > > allowing or not deletion on time base. > > > > > > > > > > > > Paolo Patierno > > > > Senior Software Engineer (IoT) @ Red Hat > > > > Microsoft MVP on Azure & IoT > > > > Microsoft Azure Advisor > > > > > > > &g

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

ber of partitions). > > thanks > Edo > -- > > Edoardo Comar > > IBM Message Hub > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > From: Tom Bentley <t.j.bent...@gmail.com> > To: dev@kafka.apache.org > Date:

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

15:15 Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces Hi All, I've updated KIP-201 again so there is now a single policy interface (and thus a single key by which to configure it) for topic creation, modification, deletion and record deletion, which each have their own vali

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

nkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > > > From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma < > ism...@juma.me.uk> > Sent: Wednesday, September 27, 2017 10:30 AM > To: dev@kafka.apache.org >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Ismael Juma <ism...@juma.me.uk> Sent: Wednesday, September 27, 2017 10:30 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces A couple of questions: 1. Is this a concrete requirement from a user or is it hypothetical? 2. You sure you would want to do this in t

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Ismael, Thanks for looking at the KIP and explaining the thinking behind the original API. Looking at the updated KIP, I notice that we actually have a > TopicDeletionPolicy with a separate config. That seems to be a halfway > house. Not sure about that. > I can certainly see that point of

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

_ > From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma < > ism...@juma.me.uk> > Sent: Wednesday, September 27, 2017 10:18 AM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces > > A couple more commen

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

il.com> Sent: Wednesday, September 27, 2017 9:47 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces I have updated the KIP to add a common policy interface for topic and message deletion. This included pulling ClusterState and TopicState interfaces up to

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

vExperience<http://paolopatierno.wordpress.com/> From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma <ism...@juma.me.uk> Sent: Wednesday, September 27, 2017 10:18 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-201: Rationalisin

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

server will pass > > to the policy.validate method > > would just lazily tap into MetadataCache. No need for big upfront > > allocations. > > > > ciao, > > Edo > > -- > > > > Edoardo Comar > > > > IBM Message Hub >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

need for big upfront >> > allocations. >> > >> > ciao, >> > Edo >> > -- >> > >> > Edoardo Comar >> > >> > IBM Message Hub >> > >> > IBM UK Ltd, Hursley Par

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

> IBM UK Ltd, Hursley Park, SO21 2JN > > > > From: Tom Bentley <t.j.bent...@gmail.com> > To: dev@kafka.apache.org > Date: 26/09/2017 17:39 > Subject:Re: [DISCUSS] KIP-201: Rationalising Policy interfaces > > > > Hi Edoardo, > > > what about

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

DISCUSS] KIP-201: Rationalising Policy interfaces Hi Edoardo, what about a single method in ClusterState > > interface ClusterState { > public Map<String,TopicState> topicsState(); > > } > > which could return a read-only snapshot of the cluster

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Edoardo, what about a single method in ClusterState > > interface ClusterState { > public Map topicsState(); > > } > > which could return a read-only snapshot of the cluster metadata ? > Sure that would work too. A concern with that is that we end up

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

atierno> Blog : DevExperience<http://paolopatierno.wordpress.com/> From: Edoardo Comar <eco...@uk.ibm.com> Sent: Tuesday, September 26, 2017 4:18 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces Thanks Tom,

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

ge Hub IBM UK Ltd, Hursley Park, SO21 2JN From: Tom Bentley <t.j.bent...@gmail.com> To: dev@kafka.apache.org Date: 26/09/2017 16:41 Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces Hi Edoardo and Ismael, Thanks for the comments. If we're going to

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

esponse > (to tell why it's failed). > > I'm happy if you incorporate the enhancements to create/alter that allow a > > check against the cluster metadata > and leave out - to anther KIP, or maybe I'll rewrite 170 the changes to > delete. > > thanks > Edo > > -------

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

dev@kafka.apache.org Date: 26/09/2017 14:01 Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces Hi Tom, yes it makes sense to have a single KIP for enhancing these policies. As Mickael pointed out, we need that the create/alter topic policy are able to assess the current c

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

ou incorporate the enhancements to create/alter that allow a > check against the cluster metadata > and leave out - to anther KIP, or maybe I'll rewrite 170 the changes to > delete. > > thanks > Edo > > -------------- > > Edoardo Comar > > IBM Message Hub > &

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

sley Park, SO21 2JN From: Tom Bentley <t.j.bent...@gmail.com> To: dev@kafka.apache.org Date: 25/09/2017 18:11 Subject: Re: [DISCUSS] KIP-201: Rationalising Policy interfaces Hi Mickael, Thanks for the reply. Thanks for the KIP. Is this meant to superseed KIP-170 ? >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Ismael, On 25 September 2017 at 17:51, Ismael Juma wrote: > We don't have this policy today for what it's worth. > Thanks for the clarification. On re-reading I realise I misinterpreted Guozhang Wang's suggestion when 1.0.0 was first mooted: > Just to clarify, my

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Mickael, Thanks for the reply. Thanks for the KIP. Is this meant to superseed KIP-170 ? > If so, one of our key requirements was to be able to access the > topics/partitions list from the policy, so an administrator could > enforce a partition limit for example. > It's not meant to replace

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

On Mon, Sep 25, 2017 at 5:32 PM, Tom Bentley wrote: > > bq. If this KIP is accepted for Kafka 1.1.0 this removal could happen in > > Kafka 3.0.0 > > > > There would be no Kafka 2.0 ? > > > > As I understand it, a deprecation has to exist for a complete major version >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Tom, Thanks for the KIP. Is this meant to superseed KIP-170 ? If so, one of our key requirements was to be able to access the topics/partitions list from the policy, so an administrator could enforce a partition limit for example. Also instead of simply having the Java Principal object, could

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

bq. deprecations that are added in 1.x (x>0) have to remain in all 2.y Makes sense. It is fine to exclude KIP-113 from your KIP. Thanks On Mon, Sep 25, 2017 at 9:32 AM, Tom Bentley wrote: > Hi Ted, > > Thanks for the feedback! > > bq. topic.action.policy.class.name > >

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

Hi Ted, Thanks for the feedback! bq. topic.action.policy.class.name > > Since the policy would cover more than one action, how about using actions > for the second word ? > Good point, done. > For TopicState interface, the abstract modifier for its methods are not > needed. > Fixed. bq.

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

bq. topic.action.policy.class.name Since the policy would cover more than one action, how about using actions for the second word ? For TopicState interface, the abstract modifier for its methods are not needed. bq. KIP-113 Mind adding more to the above bullet ? bq. If this KIP is accepted