[jira] [Created] (KAFKA-12605) kafka consumer churns through buffer memory iterating over records

radai rosenblatt created KAFKA-12605: Summary: kafka consumer churns through buffer memory iterating over records Key: KAFKA-12605 URL: https://issues.apache.org/jira/browse/KAFKA-12605 Project

[jira] [Created] (KAFKA-9998) KafkaProducer.close(timeout) still may block indefinitely

radai rosenblatt created KAFKA-9998: --- Summary: KafkaProducer.close(timeout) still may block indefinitely Key: KAFKA-9998 URL: https://issues.apache.org/jira/browse/KAFKA-9998 Project: Kafka

[jira] [Created] (KAFKA-9855) dont waste memory allocating Struct and values objects for Schemas with no fields

radai rosenblatt created KAFKA-9855: --- Summary: dont waste memory allocating Struct and values objects for Schemas with no fields Key: KAFKA-9855 URL: https://issues.apache.org/jira/browse/KAFKA-9855

Re: CompletableFuture?

XXX%3A+Return+CompletableFuture+from+KafkaProducer.send > > > > I was still debating which was the best path forward (i.e. what should be > > in rejected alternatives versus the actual proposal). Feedback is welcome. > > You're also welcome to take over the KIP if you ha

CompletableFuture?

Hi With kip-118 (Drop Support for Java 7) officially done, is there a timeline replacing usage of "plain" Futures with java 8 CompletableFutures? kafka 2.0 was mentioned at some point as a possible target for this ...

Re: [VOTE] KIP-514 Add a bounded flush() API to Kafka Producer

i also have a PR :-) https://github.com/apache/kafka/pull/7569 On Thu, Dec 12, 2019 at 9:50 PM Gwen Shapira wrote: > > You got 3 binding votes (Joel, Harsha, Ismael) - the vote passed on Nov 7. > > Happy hacking! > > On Thu, Dec 12, 2019 at 11:35 AM radai wrote: >

Re: [EXTERNAL] Re: [DISCUSS] KIP-280: Enhanced log compaction

may I suggest that if, under "header" strategy, multiple records are found with identical header values they are ALL kept? this would be useful in cases where users send larger payloads than max record size to kafka and are forced to fragment them - by setting the same header in all fragments it

Re: [VOTE] KIP-514 Add a bounded flush() API to Kafka Producer

so can we call this passed ? On Thu, Nov 7, 2019 at 7:43 AM Satish Duggana wrote: > > +1 (non-binding) > > On Thu, Nov 7, 2019 at 8:58 PM Ismael Juma wrote: > > > > +1 (binding) > > > > On Thu, Oct 24, 2019 at 9:33 PM radai wrote: > > > > > He

Re: Broker Interceptors

text, > > > > RequestObserver requestObserver); > > > > ``` > > > > > > > > where the implementation would look like this: > > > > > > > > ``` > > > > @Override > > > >

Re: Broker Interceptors

to the original kip-388 proposal. On Tue, Dec 3, 2019 at 9:15 PM radai wrote: > > yeah, we tried for this a while back (kip 388 - > https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response) > > its implemented in our kafka repo (

Re: Broker Interceptors

yeah, we tried for this a while back (kip 388 - https://cwiki.apache.org/confluence/display/KAFKA/KIP-388%3A+Add+observer+interface+to+record+request+and+response) its implemented in our kafka repo (linked to above) On Tue, Dec 3, 2019 at 8:59 PM Ignacio Solis wrote: > > At LinkedIn we run a

Re: [VOTE] KIP-514 Add a bounded flush() API to Kafka Producer

+1 (non binding, and another bump) On Fri, Oct 25, 2019 at 11:51 AM Harsha Chintalapani wrote: > > +1 (binding) > -Harsha > > > On Fri, Oct 25 2019 at 11:01 AM, wrote: > > > > > +1 > > > > On Thu, Oct 24, 2019 at 9:33 PM radai wrote: > > >

Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer

vote thread created. On Mon, Oct 21, 2019 at 9:58 AM Colin McCabe wrote: > > Hi Radai, > > It seems reasonable to me. > > best, > Colin > > > On Mon, Oct 21, 2019, at 09:52, radai wrote: > > yet another bump. > > > > can we please have a vote if

[VOTE] KIP-514 Add a bounded flush() API to Kafka Producer

Hello, I'd like to initiate a vote on KIP-514. links: the kip - https://cwiki.apache.org/confluence/display/KAFKA/KIP-514%3A+Add+a+bounded+flush%28%29+API+to+Kafka+Producer the PR - https://github.com/apache/kafka/pull/7569 Thank you

Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer

yet another bump. can we please have a vote if there are no objections ? On Wed, Sep 25, 2019 at 1:28 PM radai wrote: > > bump. > > so if no more concerns, can we move to a vote on this ? > > On Fri, Sep 13, 2019 at 10:05 AM radai wrote: > > > > we have a lot

Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer

bump. so if no more concerns, can we move to a vote on this ? On Fri, Sep 13, 2019 at 10:05 AM radai wrote: > > we have a lot of processes that need a time-bounded checkpoint logic. > > the standard use case is some consume-process-produce logic of the > following form: &

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

while youre at it another metric that we have found to be useful is % time spent in user code vs time spent in poll() (so time between poll calls / time inside poll calls) - the higher the % value the more indicative of user code being the cause of performance bottlenecks. On Fri, Sep 13, 2019 at

Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer

end the whole app into rebalance storms, so we'd really love to be able to put a time bound on it. im also fine with rethrowing InterruptedException On Thu, Sep 12, 2019 at 3:46 PM Colin McCabe wrote: > > Hi Radai, > > Thanks for the KIP. Sounds interesting. I assume that if an > In

Re: [DISCUSS] KIP-514: Add a bounded flush() API to Kafka Producer

bump. if no one has any comments on this can we initiate a vote? On Tue, Sep 3, 2019 at 8:28 AM KUN DU wrote: > > Hi, > > I would like to start discussion on KIP-514 that proposes we add a > bounded flush() API to producer. > > Link to the KIP: >

[jira] [Created] (KAFKA-8366) partitions of topics being deleted show up in the offline partitions metric

radai rosenblatt created KAFKA-8366: --- Summary: partitions of topics being deleted show up in the offline partitions metric Key: KAFKA-8366 URL: https://issues.apache.org/jira/browse/KAFKA-8366

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

> > > > > > > > Jason Gustafson wrote on 27/11/2018 00:09:56: > > > > > > > > > Another wrinkle to consider is KIP-320. If you are planning to > > replicate > > > > > __consumer_offsets directly, then you will have to account

Re: [DISCUSS] KIP-391: Allow Producing with Offsets for Cluster Replication

a few questions: 1. how do you handle possible duplications caused by the "special" producer timing-out/retrying? are you explicitely relying on the "exactly once" sequencing? 2. what about the combination of log compacted topics + replicator downtime? by the time the replicator comes back up

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

clients, central observation of client activity is in > my opinion an essential feature. > > Peter > > On Thu, Nov 8, 2018 at 12:13 PM radai wrote: > > > bump. > > > > I think the proposed API (Observer) is useful for any sort of > > multi-tenant environment

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

bump. I think the proposed API (Observer) is useful for any sort of multi-tenant environment for chargeback and reporting purposes. if no one wants to comment, can we initiate a vote? On Mon, Nov 5, 2018 at 6:31 PM Lincong Li wrote: > > Hi everyone. Here >

[jira] [Resolved] (KAFKA-7475) print the actual cluster bootstrap address on authentication failures

[ https://issues.apache.org/jira/browse/KAFKA-7475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] radai rosenblatt resolved KAFKA-7475. - Resolution: Fixed Reviewer: Rajini Sivaram Fix Version/s: 2.1.0 > pr

[jira] [Created] (KAFKA-7475) print the actual cluster bootstrap address on authentication failures

radai rosenblatt created KAFKA-7475: --- Summary: print the actual cluster bootstrap address on authentication failures Key: KAFKA-7475 URL: https://issues.apache.org/jira/browse/KAFKA-7475 Project

[jira] [Created] (KAFKA-7473) allow configuring kafka client configs to not warn for unknown config peoperties

radai rosenblatt created KAFKA-7473: --- Summary: allow configuring kafka client configs to not warn for unknown config peoperties Key: KAFKA-7473 URL: https://issues.apache.org/jira/browse/KAFKA-7473

[jira] [Created] (KAFKA-6648) Fetcher.getTopicMetadata() only returns "healthy" partitions, not all

radai rosenblatt created KAFKA-6648: --- Summary: Fetcher.getTopicMetadata() only returns "healthy" partitions, not all Key: KAFKA-6648 URL: https://issues.apache.org/jira/browse/KAFKA-6648

[jira] [Created] (KAFKA-6622) GroupMetadataManager.loadGroupsAndOffsets decompresses record batch needlessly

radai rosenblatt created KAFKA-6622: --- Summary: GroupMetadataManager.loadGroupsAndOffsets decompresses record batch needlessly Key: KAFKA-6622 URL: https://issues.apache.org/jira/browse/KAFKA-6622

[jira] [Resolved] (KAFKA-6345) NetworkClient.inFlightRequestCount() is not thread safe, causing ConcurrentModificationExceptions when sensors are read

[ https://issues.apache.org/jira/browse/KAFKA-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] radai rosenblatt resolved KAFKA-6345. - Resolution: Fixed > NetworkClient.inFlightRequestCount() is not thread safe, caus

[jira] [Created] (KAFKA-6345) NetworkClient.inFlightRequestCount() is not thread safe, causing ConcurrentModificationExceptions when sensors are read

radai rosenblatt created KAFKA-6345: --- Summary: NetworkClient.inFlightRequestCount() is not thread safe, causing ConcurrentModificationExceptions when sensors are read Key: KAFKA-6345 URL: https

[jira] [Created] (KAFKA-6216) kafka logs for misconfigured ssl clients are unhelpful

radai rosenblatt created KAFKA-6216: --- Summary: kafka logs for misconfigured ssl clients are unhelpful Key: KAFKA-6216 URL: https://issues.apache.org/jira/browse/KAFKA-6216 Project: Kafka

[GitHub] kafka pull request #4223: when closing a socket in response to an IOExceptio...

GitHub user radai-rosenblatt opened a pull request: https://github.com/apache/kafka/pull/4223 when closing a socket in response to an IOException, also print the root issue if closing fails `Selector.pollSelectionKeys()` attempts to close the channel in response to an Exception

[jira] [Created] (KAFKA-5190) builds with low parallelism exhaust system open files and crash

radai rosenblatt created KAFKA-5190: --- Summary: builds with low parallelism exhaust system open files and crash Key: KAFKA-5190 URL: https://issues.apache.org/jira/browse/KAFKA-5190 Project: Kafka

[jira] [Created] (KAFKA-5189) trunk unstable - DescribeConsumerGroupTest.testDescribeGroupWithNewConsumerWithShortInitializationTimeout fails 90% of times

radai rosenblatt created KAFKA-5189: --- Summary: trunk unstable - DescribeConsumerGroupTest.testDescribeGroupWithNewConsumerWithShortInitializationTimeout fails 90% of times Key: KAFKA-5189 URL: https

[GitHub] kafka pull request #2025: KAFKA-4293 - improve ByteBufferMessageSet.deepIter...

Github user radai-rosenblatt closed the pull request at: https://github.com/apache/kafka/pull/2025 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] kafka pull request #2814: change language level from java 7 to 8

Github user radai-rosenblatt closed the pull request at: https://github.com/apache/kafka/pull/2814 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] kafka pull request #2814: change language level from java 7 to 8

GitHub user radai-rosenblatt opened a pull request: https://github.com/apache/kafka/pull/2814 change language level from java 7 to 8 now that KIP-118 has passed, and there are no major releases coming before 0.11 Signed-off-by: radai-rosenblatt <radai.rose

Re: [VOTE] KIP-112 - Handle disk failure for JBOD

+1, LGTM On Mon, Apr 3, 2017 at 9:49 AM, Dong Lin wrote: > Hi all, > > It seems that there is no further concern with the KIP-112. We would like > to start the voting process. The KIP can be found at > *https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

Re: KafkaProducer may send duplicated message sometimes

code needs to be written to account for this - either "dedup" things you read or make sure read side-effects are idempotent in some other way. longer-term when KIP-98 is implemented you could consider switching to it On Fri, Mar 31, 2017 at 9:28 AM, Yang Cui <y...@freewheel.tv> wrote

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

t; > >> >> >> >> allocated in > > > >> >> >> >> >> the MemoryPool. Smaller messages will be allocated > > directly > > > > on > > > >> the > > > >> >> >> Heap > > &

Re: KafkaProducer may send duplicated message sometimes

kafka (at least out of the box as it is now) is not an exactly-once system. its an at-least-once system, meaning the scenario you described (and similar ones involving socket disconnections, for example) exist by design. there is a KIP for adding exactly once guarantees (among other things) that

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

t; > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks, > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > &g

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

n > > > > > > > > > > > > > > Ismael > > > > > > > > > > On Fri, Mar 3, 2017 at 9:37 PM, Colin McCabe <cmcc...@apache.org> > wrote: > > > > > > > > > > > On Fri, Mar 3, 2017, at 06:41,

Re: [DISCUSS] KIP-82 - Add Record Headers

the common "stack" we envision at linkedin would consist of (at least) the following components that add headers to every outgoing request: 1. auditing/"lineage" - appends a header containing "node" (hostname etc), time (UTC time) and destination (cluster/topic). these accumulate as requests get

Re: [DISCUSS] KIP-82 - Add Record Headers

what i think is the most common case - a for each loop: for (Header stop : headers("lineage")) { //examine stop } On Tue, Mar 7, 2017 at 12:31 PM, radai <radai.rosenbl...@gmail.com> wrote: > ing, as you call it, would probably be implemente

Re: [DISCUSS] KIP-82 - Add Record Headers

where do you see insert-in-the-middle/replace being commonly used? lineage tracing, as you call it, would probably be implemented by way of: 1. every "stop" along the way appending itself (at the end) 2. some replication technologies, instead of just doing #1, may clear out everything when they

[GitHub] kafka pull request #2637: throw NoOffsetForPartitionException from poll once...

GitHub user radai-rosenblatt opened a pull request: https://github.com/apache/kafka/pull/2637 throw NoOffsetForPartitionException from poll once for all TopicPartitions affected Signed-off-by: radai-rosenblatt <radai.rosenbl...@gmail.com> You can merge this pull r

[jira] [Created] (KAFKA-4839) throw NoOffsetForPartitionException once for all assigned partitions from poll

radai rosenblatt created KAFKA-4839: --- Summary: throw NoOffsetForPartitionException once for all assigned partitions from poll Key: KAFKA-4839 URL: https://issues.apache.org/jira/browse/KAFKA-4839

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

quick comment on the request objects: i see "abstract class NewTopic" and "class NewTopicWithReplication" and " NewTopicWithReplicaAssignments" 1. since the result object is called CreateTopicResults should these be called *Request? 2. this seems like a suboptimal approach to me. imagine we add

Re: [DISCUSS] KIP-82 - Add Record Headers

@michael: i used void because im used to java beans. thinking about it, i dont see much use for returning false from adding a header: if the headers are in read-only you should probably thrown an IllegalStateException because lets face it, 99% of users dont check return values. returning "this"

Re: [DISCUSS] KIP-82 - Add Record Headers

I will settle for any API really, but just wanted to point out that as it stands right now the API targets the most "advanced" (hence obscure and rare) use cases, at the expense of the simple and common ones. i'd suggest (as the minimal set): Header header(String key) - returns JUST ONE (the very

Re: [DISCUSS] KIP-82 - Add Record Headers

gt; > We’re trying to make an eco-system for people to be able to use > > headers, I think we want to ensure some least common features are > supported > > and not limited. > > > > > > Some examples we have already. > > > > On consume

Re: [DISCUSS] KIP-124: Request rate quotas

@jun: i wasnt concerned about tying up a request processing thread, but IIUC the code does still read the entire request out, which might add-up to a non-negligible amount of memory. On Thu, Feb 23, 2017 at 11:55 AM, Dong Lin wrote: > Hey Rajini, > > The current KIP says

Re: [DISCUSS] KIP-124: Request rate quotas

i dont think time/cpu% are easy to reason about. most user-facing quota systems i know (especially the commercial ones) focus on things users understand better - iops and bytes. as for quotas and "overhead" requests like heartbeats - on the one hand subjecting them to the quota may cause clients

Re: [DISCUSS] KIP-82 - Add Record Headers

append-only would mean that if (for whatever reason) i want to replace a header or strip it out i'd need to copy the whole record. On Wed, Feb 22, 2017 at 5:09 PM, Michael Pearce wrote: > Im happy to compromise to keep it mutable but move to an append style api. > (as

Re: [DISCUSS] KIP-82 - Add Record Headers

; representation to > slice the headers and initialize them lazily. Either way, it > might be a > good idea to use a separate object to represent the headers in > case we need > to modify it in the future in some way. > > (By the way, doesn'

Re: [DISCUSS] KIP-82 - Add Record Headers

; > >> > >> >> > > > > > >> > >> >> > > > > > >> > >> >> > > > > > >> > >> >> > > > > > >> > &g

Re: Cannot start up Kafka Server within Java

kafka publishes a "test" artifact: http://search.maven.org/#search|ga|1|a%3A%22kafka_2.12%22 if you introduce a test-scoped dep. on it, which in maven would look like: org.apache.kafka kafka_2.12 0.10.1.1 test

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

principal name to authorizer ? > We may need to use PrincipalBuilder for mapping SASL names. > > Related JIRA is here: > https://issues.apache.org/jira/browse/KAFKA-2854 > > > On Wed, Feb 15, 2017 at 7:47 AM, Jun Rao <j...@confluent.io> wrote: > > > Hi, Radai,

Re: [DISCUSS] KIP-82 - Add Record Headers

I've trimmed the inline contents as this mail is getting too big for the apache mailing list software to deliver :-( 1. the important thing for interoperability is for different "interested parties" (plugins, infra layers/wrappers, user-code) to be able to stick pieces of metadata onto msgs

Re: [VOTE] KIP-82 Add Record Headers

+1 from me. also - a more usable link to the discussion thread: http://markmail.org/message/x5wlkieexinovsz3 On Tue, Feb 14, 2017 at 9:42 AM, Michael Pearce wrote: > Hi all, > > We would like to start the voting process for KIP-82 – Add record headers. > The KIP can be

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

1. making the client Closeable/AutoCloseable would allow try (Client = ...) {} without the need to finally close. 2. a "stream processing unit" (producer + consumer) currently holds 2 open sockets to every broker it interacts with, because producer and consumer dont share the network stack. if we

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

if i understand correctly, kafka-acls.sh spins up an instance of (the custom, in our case) Authorizer, and calls things like addAcls(acls: Set[Acl], resource: Resource) on it, which are defined in the interface, hence expected to be "extensible". (side note: if Authorizer and PrincipalBuilder are

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

+1. under ismael's "facet" approach we could always start with AdminClient.topics() and then pile on more of them later. On Tue, Feb 7, 2017 at 8:57 AM, Grant Henke wrote: > +1 I think its important to focus this KIP discussion on the "patterns" we > would like to see in

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

even assuming all consumers use kafka for offset storage, would it be possible to get this information from a single broker without "reaching out" to all brokers in a cluster? On Mon, Feb 6, 2017 at 2:05 PM, Jianbin Wei wrote: > In the specify group information, can we

Re: [VOTE] KIP-111 Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

+1 On Fri, Feb 3, 2017 at 11:25 AM, Mayuresh Gharat wrote: > Hi All, > > It seems that there is no further concern with the KIP-111. At this point > we would like to start the voting process. The KIP can be found at >

Re: [DISCUS] consuming messages is polling - is there a reason? new KIP for poll?

kafka relies on the underlying OS' page cache for serving "popular" data. so "pre-assembling" push batches would move from page cache to heap storage, which is not as appealing. also, for trivial cases a lot of consumers read the same thing, which would make the heap caching even worse. also - i

Re: [DISCUS] consuming messages is polling - is there a reason? new KIP for poll?

minimizing the cost of clients is part of what makes kafka scale. a push model would shift a lot of tracking logic onto the broker. On Tue, Jan 31, 2017 at 2:47 AM, Alexander Binzberger < alexander.binzber...@wingcon.com> wrote: > way it seams like the protocol and the high-level consumer would

Re: [DISCUSS] KIP-112: Handle disk failure for JBOD

a few extra points: 1. broker per disk might also incur more client <--> broker sockets: suppose every producer / consumer "talks" to >1 partition, there's a very good chance that partitions that were co-located on a single 10-disk broker would now be split between several single-disk broker

Re: [DISCUSS] KIP-111 : Kafka should preserve the Principal generated by the PrincipalBuilder while processing the request received on socket channel, on the broker.

LGTM. Kafka currently allows setting both a custom PrincipalBuilder and a custom Authorizer (expected to act on the output of the principal builder) but makes the naive assumption that any and all information about a (custom) principal is solely contained in its name property. this kip addresses

Re: [VOTE] KIP-74: Add FetchResponse size limit in bytes

+1 On Fri, Jan 20, 2017 at 9:51 PM, Apurva Mehta wrote: > +1 > > On Fri, Jan 20, 2017 at 5:19 PM, Jason Gustafson > wrote: > > > +1 > > > > On Fri, Jan 20, 2017 at 4:51 PM, Ismael Juma wrote: > > > > > Good catch, Colin. +1 to

Re: [DISCUSS] KIP-81: Max in-flight fetches

eptions > >> > >> > >> On Wed, Dec 14, 2016 at 12:29 PM, Rajini Sivaram <rsiva...@pivotal.io> > >> wrote: > >> > Edo, > >> > > >> > I wouldn't introduce a new config entry, especially since you don't > need >

Re: [VOTE] KIP-107 Add purgeDataBefore() API in AdminClient

LGTM, +1 On Wed, Jan 11, 2017 at 1:01 PM, Dong Lin wrote: > Hi all, > > It seems that there is no further concern with the KIP-107. At this point > we would like to start the voting process. The KIP can be found at > https://cwiki.apache.org/confluence/display/KAFKA/KIP-107

Re: [DISCUSS] KIP-82 - Add Record Headers

while HTTP-style (string, string) are the most common and most familiar, there is a very significant impact on msg size, especially given that some payloads are literally a few integers (think stock quotes) and would be dwarfed by an http-like header segment. I think we're ok with not allowing

out of the box security?

in light of things like this - https://www.bleepingcomputer.com/news/security/mongodb-apocalypse-is-here-as-ransom-attacks-hit-10-000-servers/ and given that plenty of people/orgs have public facing kafka installations that are wide open - https://www.shodan.io/search?query=kafka (yes, i realize

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

JIRA up - https://issues.apache.org/jira/browse/KAFKA-4602 PR up - https://github.com/apache/kafka/pull/2330 KIP wiki has been updated. On Fri, Jan 6, 2017 at 8:16 AM, radai <radai.rosenbl...@gmail.com> wrote: > Will do (sorry for the delay). > and thank you. > > On Fri, Jan

[GitHub] kafka pull request #2330: KAFKA-4602 - KIP-72 - Allow putting a bound on mem...

GitHub user radai-rosenblatt opened a pull request: https://github.com/apache/kafka/pull/2330 KAFKA-4602 - KIP-72 - Allow putting a bound on memory consumed by Incoming requests this is the initial implementation. You can merge this pull request into a Git repository by running

Re: Different key with the same digest in log compaction

Colin McCabe <cmcc...@apache.org> wrote: > That's a fair point. The calls to Arrays.equals are comparing just the > hashes, not the keys. > > Colin > > On Tue, Jan 3, 2017, at 17:15, radai wrote: > > looking at the code (SkimpyOffsetMap.get/put) they both start with &

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

Will do (sorry for the delay). and thank you. On Fri, Jan 6, 2017 at 7:56 AM, Ismael Juma <ism...@juma.me.uk> wrote: > Radai, you have more than enough votes to declare the vote successful. > Maybe it's time to do so. :) Also, once you have done that, it would be > good t

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

I cant speak for anyone else, but a rolling upgrade is definitely how we (LinkedIn) will do the migration. On Thu, Jan 5, 2017 at 4:28 PM, Gwen Shapira wrote: > it sounds good to have > it, but that's probably not how people will end up migrati >

[jira] [Created] (KAFKA-4602) KIP-72 Allow putting a bound on memory consumed by Incoming requests

radai rosenblatt created KAFKA-4602: --- Summary: KIP-72 Allow putting a bound on memory consumed by Incoming requests Key: KAFKA-4602 URL: https://issues.apache.org/jira/browse/KAFKA-4602 Project

Re: [DISCUSS] KIP-109: Old Consumer Deprecation

im all for (working towards) getting rid of old code, but there's still no solid migration path - you'll be "stranding" users on deprecated, no longer maintained code with no "safe" way out that does not involve downtime (specifically old and new consumers cannot correctly divide up partitions

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

l eternity, unless its cleaned-up itself. ... complexity :-) On Wed, Jan 4, 2017 at 8:04 AM, radai <radai.rosenbl...@gmail.com> wrote: > in summary - i'm not opposed to the idea of a per-topic clean up config > that tracks some set of consumer groups' offsets (which would pr

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

4, 2017 at 7:54 AM, radai <radai.rosenbl...@gmail.com> wrote: > a major motivation for this KIP is cost savings. > > lots of internal systems at LI use kafka as an intermediate pipe, and set > the topic retention period to a "safe enough" amount of time to be ab

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

stava <e...@confluent.io> > wrote: > > > On Tue, Jan 3, 2017 at 6:14 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > Hey Ewen, > > > > > > Thanks for the review. As Radai explained, it would be complex in terms > > of > > > use

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

@jun - good proposal. i was willing to concede that read-uncommitted was impossible under my proposal but if LSO/NSO is introduced is becomes possible. On Tue, Jan 3, 2017 at 7:50 AM, Jun Rao wrote: > Just to follow up on Radai's idea of pushing the buffering logic to the >

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

also 4. some apps may do their own offset bookkeeping On Tue, Jan 3, 2017 at 5:29 PM, radai <radai.rosenbl...@gmail.com> wrote: > the issue with tracking committed offsets is whos offsets do you track? > > 1. some topics have multiple groups > 2. some "groups" are rea

Re: [DISCUSS] KIP-107: Add purgeDataBefore() API in AdminClient

the issue with tracking committed offsets is whos offsets do you track? 1. some topics have multiple groups 2. some "groups" are really one-offs like developers spinning up console consumer "just to see if there's data" 3. there are use cases where you want to deliberately "wipe" data EVEN IF its

Re: Different key with the same digest in log compaction

looking at the code (SkimpyOffsetMap.get/put) they both start with hashInto(key, hash1) and then ignore key from that point on - so we're not using the key unless im missing something? as for the probability of collision - it depends on the hash algo and the number of keys. if you configure it to

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

I've just re-validated the functionality works - broker throttles under stress instead of OOMs. at this point my branch ( https://github.com/radai-rosenblatt/kafka/tree/broker-memory-pool-with-muting) is "code complete" and somewhat tested and im waiting on the voting proce

Re: [DISCUSS] KIP-105: Addition of Record Level for Sensors

link leads to 104. i think this is the correct one - https://cwiki.apache.org/confluence/display/KAFKA/KIP-105%3A+Addition+of+Record+Level+for+Sensors ? On Fri, Dec 30, 2016 at 8:31 PM, Aarti Gupta wrote: > Hi all, > > I would like to start the discussion on KIP-105:

Re: custom offsets in ProduceRequest

or even better - if topic creation is done dynamically by the replicator, setting the initial offsets for partitions could be made part of topic creation ? even less API changes this way On Thu, Dec 29, 2016 at 10:49 PM, radai <radai.rosenbl...@gmail.com> wrote: > ah, I didnt r

Re: custom offsets in ProduceRequest

e made part of the replicator component (mirror maker, or whatever else you want to use) - if topic X does not exist in destination, create it, reset initial offsets to match source, start replication On Thu, Dec 29, 2016 at 12:41 PM, Andrey L. Neporada < anepor...@yandex-team.ru> wrote: &g

Re: custom offsets in ProduceRequest

ion. none of these options look good to me. On Thu, Dec 29, 2016 at 3:22 AM, Andrey L. Neporada < anepor...@yandex-team.ru> wrote: > Hi! > > > On 27 Dec 2016, at 19:35, radai <radai.rosenbl...@gmail.com> wrote: > > > > IIUC if you replicate from a single source cluster

Re: custom offsets in ProduceRequest

IIUC if you replicate from a single source cluster to a single target cluster, the topic has the same number of partitions on both, and no one writes directly to the target cluster (so master --> slave) the offsets would be preserved. but in the general case - how would you handle the case where

Re: Different key with the same digest in log compaction

i think the code assumes that with a "good enough" hash function (and maybe few enough keys) the chance of such a collision is acceptably small to justify the savings of not keeping the keys in memory. On Wed, Dec 21, 2016 at 11:50 PM, Renkai wrote: > Hi, all: > > I am

Re: [VOTE] KIP-92 - Add per partition lag metrics to KafkaConsumer

+1 On Wed, Dec 21, 2016 at 9:51 AM, Dong Lin wrote: > +1 (non-binding) > > On Thu, Dec 15, 2016 at 5:32 PM, Becket Qin wrote: > > > Hi, > > > > I want to start a voting thread on KIP-92 which proposes to add per > > partition lag metrics to

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

when the leader decides to commit a TX (of X msgs, known at this point), it writes an "intent to append X msgs" msg (control?) followed by the X msgs (at this point it is the leader and therefor point of sync, so this can be done with no "foreign" msgs in between). if there's a crash/change of

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

much of that is > obvious from the proposal. > > -Jay > > > > > > On Tue, Dec 20, 2016 at 9:11 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > Just got some time to go through most of this thread and KIP - great to > see > > this materialize and

  1   2   >