r
> duplicating this file to be within the RAT project. The place to ask is
> probably in the RAT project itself -- I don't know if this is something
> they'd like to include or not (hopefully yes.)
>
> cheers,
> Colin
>
>
> On Thu, Aug 15, 2024, at 04:54, Claud
Followup2: your answer speaks directly to "WRITE" access. My example was
READ access. So the question method is answering then is: Does the user
have access to READ any TOPIC? And that is further restricted by the
requestContext host is it not?
On Tue, Sep 3, 2024 at 2:10 PM Claude
Followup: If ALLOW_EVERYONE_IF_NO_ACL_IS_FOUND_CONFIG = "true" then
authorizeByResourceType should return true in all cases since the user
would have access for any operation on any undefined topic?
On Tue, Sep 3, 2024 at 2:08 PM Claude Warren, Jr
wrote:
> I am working on a re
t;true", so `User:alice` can
> write to a topic that doesn't start with `foo` and hence
> `authorizeByResourceType` should be ALLOWED. What was the behaviour you
> observed?
>
> Regards,
>
> Rajini
>
>
> On Tue, Sep 3, 2024 at 12:22 PM Claude Warren wrote:
*Setup:*
Superuser = "User:superman"
ACLs added to system
new StandardAcl(TOPIC, "foo", PREFIXED, "User:alice", WILDCARD, READ, DENY)
new StandardAcl(TOPIC, "foobar", LITERAL, "User:alice", WILDCARD, READ,
ALLOW)
new StandardAcl(TOPIC, "foo", PREFIXED, "User:bob", WILDCARD, READ, ALLOW)
ALLOW_EVE
a pull request to satisfy KAFKA-17423 that contains only
the new implementation.
Claude
On Mon, Sep 2, 2024 at 9:09 AM Claude Warren, Jr
wrote:
> I have been working on implementing a Trie structure to store ACLs and
> improve the performance in the metadata/authorization code. The ups
After some discussion about the earlier KIP-1042 we have rewritten it to
focus on the implementation of a GLOB pattern type. Please review and
comment.
We have removed all discussion of the Trie implementation and focus on what
is required for the GLOB implementation. The KIP does assume that th
I have been working on implementing a Trie structure to store ACLs and
improve the performance in the metadata/authorization code. The upshot of
this was that I found it very difficult to determine if the implementation
was correctly reimplementing the current implementation.
My goal was to simpl
r these things. Does that make sense? I don't
> see another path to doing it compatibly. I certainly wouldn't want to
> create a "Python 2 vs. Python 3" type situation where people get stuck on
> an older authorizer fork because the new one requires globs and they can'
y have had,
Claude
On Thu, Aug 29, 2024 at 6:51 PM Colin McCabe wrote:
> On Thu, Aug 29, 2024, at 01:34, Claude Warren, Jr wrote:
>
> Colin,
>
> Thanks for your insightful comments. I came to the same conclusion.
>
> I do have 2 Jira tickets to simplify some of t
aude
On Fri, Aug 23, 2024 at 9:43 PM Colin McCabe wrote:
> On Sat, Jul 27, 2024, at 04:20, Claude Warren, Jr wrote:
> > I have updated the KIP with results from the Trie implementation and they
> > are dramatic to say the least. For most searches they are at least an
> > order
URL: https://issues.apache.org/jira/browse/KAFKA-17423
The above is an improvement to Kafka to replace the sorted list ACL
implementation with a Trie based implementation. I have an implementation
that passes all the tests, including the new ones in
KAFKA-17316 (pull request https://github.com/ap
Claude Warren created KAFKA-17423:
-
Summary: Replace StandardAuthorizer with Trie implementation
Key: KAFKA-17423
URL: https://issues.apache.org/jira/browse/KAFKA-17423
Project: Kafka
Issue
Should a ResourcePatternFilter that has a PatternType of ANY and a name of
WILDCARD_RESOURCE not match any Acls? I think this is a bug.I am writing
a series of tests to ensure that I have implemented everything correctly in
the Trie implementation and this has come up.
public boolean matches(Res
Greetings,
I have been working on Apache RAT recently and I noticed that Kafka has a
very nice XSLT to convert the Rat output to an HTML document.
I know there is not a legal or licensing issue but I am asking if there are
any objections to my taking the .gradle/resources/rat-output-to-html.xsl
f
:DOH: Nevermind. Problem between keyboard and seat.
On Thu, Aug 15, 2024 at 8:36 AM Claude Warren, Jr
wrote:
> If there is an authorizer with no ACLs and
> authorizeByResourceType(AuthorizableRequestContext
> requestContext, AclOperation op, ResourceType resourceType) is called
If there is an authorizer with no ACLs and
authorizeByResourceType(AuthorizableRequestContext
requestContext, AclOperation op, ResourceType resourceType) is called with
op = UNKNOWN or ANY, or resourceType = UKNOWN or ANY should an
IllegalArgumentException be thrown as it is when there are ACLs?
I
Claude Warren created KAFKA-17316:
-
Summary: Refactor StandardAuthorizer for easier extension
Key: KAFKA-17316
URL: https://issues.apache.org/jira/browse/KAFKA-17316
Project: Kafka
Issue
StandardAuthorizor.
-
https://github.com/Claudenw/kafka/compare/StandardAuthorizer_refactor...KIP-1042_trie_simplification
applies
the Trie changes to the authorizer refactor.
On Fri, Aug 2, 2024 at 10:07 AM Claude Warren, Jr
wrote:
> Proposed Changes
>> This KIP suggests to suppo
RAL and PREFIXED acls.
>
>
>- Define the Trie structure
>
>
>- Populate the Trie with ACLs
>
>
>- Retrieve ACLs using the Trie
>
> With this optimization, we hope to have a drastic reduced latency in the
> matchingAcls method, and it's much more efficient.
29, 2024 at 8:36 AM Claude Warren, Jr
wrote:
> I have updated the KIP with results from the Trie implementation and they
> are dramatic to say the least. For most searches they are at least an
> order of magnitude faster and use less memory. The wildcard search is not
> a regular expres
or themselves on a convenience vs
> >>> performance scale: Smaller use-cases can add a single convenient
> >>> MATCH, and larger use-cases can add the multiple optimized PREFIXes.
> >>>
> >>> [1]
> >>>
> https://github.com/apache/kafk
this code, I will complete the documentation
and fix the checkstyle and then open a pull request.
Claude
[1] https://github.com/Claudenw/kafka/pull/new/KIP-1042_Trie_Implementation
On Wed, Jul 3, 2024 at 2:21 PM Claude Warren, Jr
wrote:
> I think that if we put in a trie based system we should
I think that if we put in a trie based system we should be able to halve
the normal searhc times and still be able to locate wild card matches very
quickly. Users should be warned that "head wildcard" matches are slow and
to use them sparingly. I am going to see if I can work out how to do
wildca
I give this a cautious +1 (non binding) as development may yield better
head wildcard results.
I think the adoption criteria for the ACL search needs to be specified in
the KIP. We do not have a good handle on how long the current searches
take. If the wildcard tests can be merged into a trie se
Igor,
Thanks for the well thought out comment. Do you have a suggestion for a
fast way to write to disk? Since the design requires random access perhaps
just a random access file?
Claude
On Thu, May 23, 2024 at 1:17 PM Igor Soarez wrote:
> Hi Claude,
>
> Thanks for writing this KIP. This iss
pshots.
> Currently the snapshot logic is decoupled from the log retention as of
> KIP-360.
>
> Justine
>
> On Mon, May 20, 2024 at 11:20 PM Claude Warren wrote:
>
> > The LRU cache is just that: a cache, so yes things expire from the cache
> > but they are not gone.
ll risk
> expiring
> > > active producers.
> > >
> > > I am willing to discuss some solutions that work with older clients,
> but
> > my
> > > concern is spending too much time on a complicated solution and not
> > > encouraging movement to newer and be
memory usage.
>
> Fixing the client also allows us to fix some of the other issues we have
> with idempotent producers.
>
> Justine
>
> On Fri, May 17, 2024 at 12:46 AM Claude Warren wrote:
>
> > I think that the point here is that the design that assumes that you
> are like PREFIXED, but allow some parameter substitution. For example
> PARAMETERIZED "(nl|de|cz)-accounts-". I'm lifting regex syntax here,
> but this isn't actually a regex, and wouldn't allow arbitrary numbers
> of characters, or the * or + operators.
t; > Also in the rejection alternatives you listed an approved KIP which is a
> > bit confusing can you move this to motivations instead
> >
> > > On 15 May 2024, at 14:35, Claude Warren wrote:
> > >
> > > This is a proposal that should solve the OOM proble
per AuthorizerBenchmark, we can
> drop this kip.
>
> And thank you Claude for the suggestion on the new implementation.
>
> On Tue, May 7, 2024 at 4:37 PM Claude Warren, Jr
> wrote:
>
> > I have updated KIP-1042 with a proposal for how to reduce the time spent
> >
This is a proposal that should solve the OOM problem on the servers without
some of the other proposed KIPs being active.
Full details in
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1044%3A+A+proposal+to+change+idempotent+producer+--+server+implementation
n't actually a regex, and wouldn't allow arbitrary numbers
> of characters, or the * or + operators.
> In the background it could evaluate exactly like the 3 individual
> PREFIX rules, but be easier to evaluate on the backend, and support
> the intersection query I mentioned
ls` to support MATCH-type ACL
> without introducing performance issue?
>
>
> Thanks,
>
> 2024年5月3日(金) 19:51 Claude Warren, Jr :
>
> > As I wrote in [1], the ACL evaluation algorithm needs to be specified
> with
> > respect to the specificity of the pattern so
in reason we
> have avoided that solution is that there is no state stored for idempotent
> producers when grabbing an ID. My concern there is either storing too much
> state to track this or throttling before we need to.
>
> Justine
>
> On Thu, May 2, 2024 at 2
+Support+for+wildcard+when+creating+new+acls>
On Fri, May 3, 2024 at 12:18 PM Claude Warren wrote:
> Took me awhile to find it but the link to the KIP is
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1042%3A+Support+for+wildcard+when+creating+new+acls
>
> On Fri, May
Took me awhile to find it but the link to the KIP is
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1042%3A+Support+for+wildcard+when+creating+new+acls
On Fri, May 3, 2024 at 10:13 AM Murali Basani
wrote:
> Hello,
>
> I'd like to propose a suggestion to our resource patterns in Kafka ACLs
I think that if this is introduced (and perhaps even if it is not) we need
a clear ACL evaluation process.
I know we have both allow and deny, and that deny takes precedence over
allow.
But let's consider two scenarios
1. Unintended access.
Let's assume we start with the 6 topics Murali used in
There is some question about whether or not we need the configuration
options. My take on them is as follows:
producer.id.quota.window.num No opinion. I don't know what this is used
for, but I suspect that there is a good reason to have it. It is not used
within the Bloom filter caching mechan
Quick note: I renamed the example code. It is now at
https://github.com/Claudenw/kafka/blob/KIP-936/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerIDQuotaManagerCache.java
On Thu, May 2, 2024 at 10:47 AM Claude Warren, Jr
wrote:
> Igor, thanks for taking the time
Igor, thanks for taking the time to look and to review the code. I regret
that I have not pushed the latest code, but I will do so and will see what
I can do about answering your Bloom filter related questions here.
How would an operator know or decide to change the configuration
> for the numb
My Confluence ID is "claude"
On Thu, Apr 25, 2024 at 8:40 PM Matthias J. Sax wrote:
> What's your wiki ID? We can grant write access on our side if you have
> already an account.
>
> -Matthias
>
> On 4/25/24 4:06 AM, Claude Warren wrote:
> > I wou
I would like to get edit access to the Kafka confluence so that I can work
on KIP-936. Can someone here do that or do I need to go through Infra?
Claude
at or below Shape.P
- Assuming Shape.N = 11 and Shape.P = 0.06 the Bloom filter at each
layer will consume 35 bytes. https://hur.st/bloomfilter provides a quick
calculator for other values.
Claude
On Tue, Apr 16, 2024 at 8:06 AM Claude Warren wrote:
> Let's put aside the
t a queue entry selector based
> on
> > the modulus of the system by the resolution of the Y' partitions. On each
> > call:
> I didn’t evaluate CPC datasketch or any counter solution as I explained
> above the aim is not to build a counter specially the Kafka Sensor can be
&
he Bloom
filter implementation.
[1] https://datasketches.apache.org/docs/CPC/CPC.html
On Fri, Apr 12, 2024 at 3:10 PM Claude Warren wrote:
> I think there is an issue in the KIP.
>
> Basically the kip says, if the PID is found in either of the Bloom filters
> then no action is taken
> I
Congrats Greg! All the hard work paid off.
On Mon, Apr 15, 2024 at 6:58 AM Ivan Yurchenko wrote:
> Congrats Greg!
>
> On Sun, Apr 14, 2024, at 22:51, Sophie Blee-Goldman wrote:
> > Congrats Greg! Happy to have you
> >
> > On Sun, Apr 14, 2024 at 9:26 AM Jorge Esteban Quilcate Otoya <
> > quilca
l within the 30 minute window we
are trying to track. Or am I missing something? It seems like we need to
add each PID to the last bloom filter
On Fri, Apr 12, 2024 at 2:45 PM Claude Warren wrote:
> Initial code is available at
> https://github.com/Claudenw/kafka/blob/KIP-936/storage/src
Initial code is available at
https://github.com/Claudenw/kafka/blob/KIP-936/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerIDQuotaManager.java
On Tue, Apr 9, 2024 at 2:37 PM Claude Warren wrote:
> I should also note that the probability of false positives does not f
L would need to be provided to write to the DLQ if configured.
>
> Cheers,
> Damien
>
> On Fri, 12 Apr 2024 at 08:59, Claude Warren, Jr
> wrote:
> >
> > I am new to the Kafka codebase so please excuse any ignorance on my part.
> >
> > When a dead letter q
I am new to the Kafka codebase so please excuse any ignorance on my part.
When a dead letter queue is established is there a process to ensure that
it at least is defined with the same ACL as the original queue? Without
such a guarantee at the start it seems that managing dead letter queues
will
9, 2024 at 2:26 PM Claude Warren wrote:
> The overall design for KIP-936 seems sound to me. I would make the
> following changes:
>
> Replace the "TimedBloomFilter" with a "LayeredBloomFilter" from
> commons-collections v4.5
>
> Define the producer.id.quot
The overall design for KIP-936 seems sound to me. I would make the
following changes:
Replace the "TimedBloomFilter" with a "LayeredBloomFilter" from
commons-collections v4.5
Define the producer.id.quota.window.size.seconds to be the length of time
that a Bloom filter of PIDs will exist.
Define
Is there test code, or initial POC code for this KIP somewhere? I would like
to help move this forward but need a few pointers to associated resources. I
have read KIP-853 and it is beginning to sink in, but code would be nice.
Thanks,
Claude
On 2024/03/21 18:41:04 José Armando García Sancio
o match the quota I'll need to replace the bloom for all
> principals however if they are separated like I suggested then replacing
> the LayedBloomFilter of max entries X with another one with max entries Y
> will only impact one user and not everyone. Does this make sense?
>
I misspoke before the LayedBloomFilterTest.testExpiration() uses
milliseconds to expire the data but it layout an example of how to expire
filters in time intervals.
On Fri, Aug 18, 2023 at 4:01 PM Claude Warren wrote:
> Sorry for taking so long to get back to you, somehow I missed your mess
de that point to how often will the eviction function
> runs. Do you mean that the eviction function runs every minute? If so, can
> we control this?
>
> Cheers,
> Omnia
>
> On Wed, Jun 21, 2023 at 11:43 AM Claude Warren wrote:
>
> > I think that the either using a Sta
ayer
This should be very fast and space efficient.
On Wed, Jun 21, 2023 at 11:13 AM Claude Warren wrote:
> I have an implementation of a layered Bloom filter in [1] (note the
> layered branch). This should handle the layering Bloom filter and allow
> for layers that
>
>1. Do
I have an implementation of a layered Bloom filter in [1] (note the layered
branch). This should handle the layering Bloom filter and allow for layers
that
1. Do not become over populated and thus yield too many false positives.
2. Expire and are removed automatically.
The layered Bloom fi
The link I thought I included did not carry over in the last post. The
paper can be found at:
https://webdocs.cs.ualberta.ca/~drafiei/papers/DupDet06Sigmod.pdf
On Thu, Jun 8, 2023 at 9:05 AM Claude Warren wrote:
>
> Have you considered using Stable Bloom Filters [1]. I think they d
Have you considered using Stable Bloom Filters [1]. I think they do what you
want without a lot of the overhead you propose for your solution. In addition,
you may want to look at Commons-Collections v4.5 [2] (currently snapshot) for
efficient Bloom filter code. I have a Stable Bloom filte
Claude Warren created KAFKA-14924:
-
Summary: Kafka DOAP file has an error
Key: KAFKA-14924
URL: https://issues.apache.org/jira/browse/KAFKA-14924
Project: Kafka
Issue Type: Bug
63 matches
Mail list logo
