Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2021-01-12 Thread Boyang Chen
In my offline discussion with Rajini and other folks, we basically had a better understanding for the following problems: 1. Whether there should be a clear disabling mechanism for periodical refreshing. We thought about using -1 for a special refreshing interval value to disable it, but as a

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2021-01-08 Thread Boyang Chen
On Fri, Jan 8, 2021 at 4:46 AM Rajini Sivaram wrote: > Hi Boyang, > > Thanks for the responses. Follow up comments on a couple of those: > > 4) Can you provide some more details on the scenarios where file watcher is > useful. You mentioned hybrid, but it is not clear to me why a watcher that >

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2021-01-08 Thread Rajini Sivaram
Hi Boyang, Thanks for the responses. Follow up comments on a couple of those: 4) Can you provide some more details on the scenarios where file watcher is useful. You mentioned hybrid, but it is not clear to me why a watcher that reloads 99% of the time would be useful. There are a few cases to

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2021-01-07 Thread Boyang Chen
Hey David, thanks for the feedback. On Thu, Jan 7, 2021 at 2:37 AM David Jacot wrote: > Hi Boyang, > > Thanks for the KIP. I am fine with it in general. I just have a few > comments. > > With the proposal, we don't have the guarantee that both the new keystore > and the new truststore will be

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2021-01-07 Thread Boyang Chen
Thanks Rajini for the comments. On Thu, Jan 7, 2021 at 2:27 AM Rajini Sivaram wrote: > Hi Boyang, > > Thanks for the KIP, I have a few questions: > > 1) Will it be possible to enable/disable automatic file reloading? If not, > we should mention in the compatibility section. > I don't think we

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2021-01-07 Thread David Jacot
Hi Boyang, Thanks for the KIP. I am fine with it in general. I just have a few comments. With the proposal, we don't have the guarantee that both the new keystore and the new truststore will be picked up together so we may end up with the new keystore and the old truststore for a short period of

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2021-01-07 Thread Rajini Sivaram
Hi Boyang, Thanks for the KIP, I have a few questions: 1) Will it be possible to enable/disable automatic file reloading? If not, we should mention in the compatibility section. 2) We are introducing new common SSL configs and updating common code to perform automated updates. What does this

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2021-01-06 Thread Jason Gustafson
Thanks Boyang. Someone mentioned my email never showed up, but basically I suggested tying the refresh configuration more directly to the configurations it would affect. I'm happy with the updates. -Jason On Tue, Jan 5, 2021 at 8:34 PM Boyang Chen wrote: > Thanks Jason for the feedback. I

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2021-01-05 Thread Boyang Chen
Thanks Jason for the feedback. I separated the time configs for key store and trust store, and rename the configs as you proposed. Best, Boyang On Mon, Dec 14, 2020 at 3:47 PM Boyang Chen wrote: > Hey there, > > bumping up this thread to see if there are further questions regarding the >

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-14 Thread Boyang Chen
Hey there, bumping up this thread to see if there are further questions regarding the updated proposal. Best, Boyang On Thu, Dec 10, 2020 at 11:52 AM Boyang Chen wrote: > After some offline discussions, we believe that it's the right direction > to go by doing a hybrid approach which includes

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-10 Thread Boyang Chen
After some offline discussions, we believe that it's the right direction to go by doing a hybrid approach which includes both file-watch trigger and interval based reloading. The former guarantees a swift change in 99% time, while the latter provides a time-based guarantee in the worst case when

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-08 Thread Boyang Chen
Hey Gwen, thanks for the feedback. On Sun, Dec 6, 2020 at 10:06 PM Gwen Shapira wrote: > Agree with Igor. IIRC, we also encountered cases where filewatch was > not triggered as expected. An interval will give us a better > worse-case scenario that is easily controlled by the Kafka admin. > >

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-06 Thread Gwen Shapira
Agree with Igor. IIRC, we also encountered cases where filewatch was not triggered as expected. An interval will give us a better worse-case scenario that is easily controlled by the Kafka admin. Gwen On Sun, Dec 6, 2020 at 8:17 AM Igor Soarez wrote: > > > > > The proposed change relies on a

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-06 Thread Igor Soarez
> > The proposed change relies on a file watch, why not also have a polling > > interval to check the file for changes? > > > > The periodical check could work, the slight downside is that we need > additional configurations to schedule the interval. Do you think the > file-watch approach has

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-06 Thread Noa Resare
The benefit of the solution I mention is simply that it can be implemented without changing Kafka, and I provided it more as a side note for people reading this list that might not have time to wait for this KIP to land into a released version. I do think that the KIP proposal would be very

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-04 Thread Boyang Chen
Thanks Noa for the suggested path. Like you mentioned, I feel this mechanism is a little bit overkill for a simple security file reloading case. Could you provide more context on the benefit of doing a customized KeyManager setup? TBH, I don't see Kafka going deep into these low level security

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-04 Thread Boyang Chen
Thanks for the feedback Nikolay. I think our proposals are solving orthogonal issues. The KIP is proposing to deprecate the reloading path on AlterConfig, which has nothing to do with general certification expire issue. Best, Boyang On Fri, Dec 4, 2020 at 4:26 AM Nikolay Izhikov wrote: >

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-04 Thread Boyang Chen
Hey Igor, thanks for the feedback. On Fri, Dec 4, 2020 at 5:24 AM Igor Soarez wrote: > Hi Boyang, > > The proposed change relies on a file watch, why not also have a polling > interval to check the file for changes? > > The periodical check could work, the slight downside is that we need

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-04 Thread Igor Soarez
Hi Boyang, The proposed change relies on a file watch, why not also have a polling interval to check the file for changes? What happens if the file is changed into an invalid store? Does the previous store stay in use? Thanks, -- Igor On Fri, Dec 4, 2020, at 1:28 AM, Boyang Chen wrote: >

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-04 Thread Nikolay Izhikov
Hello, Boyang Chen. I think this KIP overlaps with my idea [1] of exposing information about certificates Kafka uses. Kafka administrator should initiate renewal certificates procedure not long before the certificate expires. But, for now, there is no way for administrators to know the

Re: [DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-04 Thread Noa Resare
Hi Boyang, I think that it would improve the ergonomics of dealing with short lived certificates to have this be the default behaviour. It should be noted that transparently reloading certificates and keys when they changed on disk can be implemented right now registering a custom

[DISCUSS] KIP-687: Automatic Reloading of Security Store

2020-12-03 Thread Boyang Chen
Hey there, I would like to start the discussion thread for KIP-687: https://cwiki.apache.org/confluence/display/KAFKA/KIP-687%3A+Automatic+Reloading+of+Security+Store This KIP is trying to deprecate the AlterConfigs API support of updating the security store by reloading path in-place, and