Re: [DISCUSS] KIP-939: Support Participation in 2PC

with a single producer. > > -Artem > > On Wed, Aug 23, 2023 at 4:50 PM Roger Hoover > wrote: > > > Thanks. I like that you're moving Kafka toward supporting this > dual-write > > pattern. Each use case needs to consider the tradeoffs. You already > > summarized

Re: [DISCUSS] KIP-939: Support Participation in 2PC

threaded per > application, but a single producer per thread model (i.e. an application > could have a pool of threads + producers to increase concurrency). > > -Artem > > On Tue, Aug 22, 2023 at 7:22 PM Roger Hoover > wrote: > > > Artem, > > > > Thanks for

Re: [DISCUSS] KIP-939: Support Participation in 2PC

l. > Or > > are we saying that when used in this 2PC context, we should configure > these > > transaction timeouts to very large durations? > > > > Thanks in advance! > > > > Best, > > Arjun > > > > > > On Mon, Aug 21, 2023 at 1:06 P

Re: [DISCUSS] KIP-939: Support Participation in 2PC

Hi Artem, Thanks for writing this KIP. Can you clarify the requirements a bit more for managing transaction state? It looks like the application must have stable transactional ids over time? What is the granularity of those ids and producers? Say the application is a multi-threaded Java web

Re: [VOTE] KIP-602: Change default value for client.dns.lookup

+1 (non-binding) Thanks, Badai. On Fri, May 22, 2020 at 10:05 AM Ismael Juma wrote: > Thanks for the KIP, +1 (binding). > > Ismael > > On Fri, May 22, 2020 at 1:40 AM Badai Aqrandista > wrote: > > > Hi All, > > > > I would like to start the vote on KIP-602: Change default value for > >

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

subset of time metrics that we feel would be super useful for debugging? > > On Tue, Sep 12, 2017 at 6:08 PM, Roger Hoover <roger.hoo...@gmail.com> > wrote: > > > Thanks, Ewen. > > > > I agree with you on the overhead of measuring time for SMTs and > > co

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

r and those actually provide a bit more detail > in some ways (e.g. join & sync vs the entire rebalance). Not sure if we > want to effectively duplicate some info so it can all be located under > Connect names or rely on the existing metrics for some of these. > > -Ewen &g

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

m > probably > > > > also > > > > > reduces the risk of them being misinterpretted. > > > > > > > > > > > > > > >> * You have status for connectors but not for tasks. Any reason > why? > > > > Seem

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

ight not be as useful for connectors that have > > relatively > > > static tasks, but it would be more interesting/useful for connectors > that > > > create tasks dynamically and periodically request task > reconfigurations. > > > > > > > > &

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Randall, Thank you for the KIP. This should improve visibility greatly. I had a few questions/ideas for more metrics. 1. What's the relationship between the worker state and the connector status? Does the 'paused' status at the Connector level include the time that worker is

Re: [DISCUSS] KIP-170: Enhanced TopicCreatePolicy and introduction of TopicDeletePolicy

Edoardo, thanks for the KIP. I think it's a good idea overall. +1 especially for including Session/Principal in the API. (#2 mentioned by Ismael) Also, the AlterPolicy should get the same info as create + delete (#4). Cheers, Roger On Thu, Sep 7, 2017 at 8:43 AM, Ismael Juma

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

smaller impact > on reducing replication latency. But it would definitely be interesting to > get those numbers and see whether the theory bears out in practice. > > Regards, > Apurva > > On Thu, Aug 31, 2017 at 8:56 AM, Roger Hoover <roger.hoo...@gmail.com>

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

PM, Roger Hoover <roger.hoo...@gmail.com> wrote: > Sorry if this is a bit out of left field but can't help wondering... > > One way to improve producer performance while still having good guarantees > would be to allow a setting between acks=1 and acks=all. We could > i

Re: [DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

Sorry if this is a bit out of left field but can't help wondering... One way to improve producer performance while still having good guarantees would be to allow a setting between acks=1 and acks=all. We could introduce "acks=minIsr". This is already the guarantee you get when the ISR set

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

eady voted, please let me know if you have any > concerns about the new field. > > Thank you... > > Regards, > > Rajini > > > > On Tue, Aug 29, 2017 at 8:44 PM, Roger Hoover <roger.hoo...@gmail.com> > wrote: > > > Hi Rajini, > > &g

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

L Impersonation not allowed", or "You account has been locked, please contact cluster admin". Thanks, Roger On Tue, Aug 29, 2017 at 12:41 PM, Roger Hoover <roger.hoo...@gmail.com> wrote: > Hi Rajini, > > The metrics in KIP-188 will provide counts across all users but

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

ul for monitoring, do we really need info-level logging for > authentication? At the moment, there don't seem to be any per-connection > informational messages at info-level, but if you think it is useful, we > could do this in a separate JIRA. Let me know what you think. > > On Tue

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

Just re-read the KIP and was wondering if you think INFO would be ok for logging successful authentications? They should be relatively infrequent. On Tue, Aug 29, 2017 at 9:54 AM, Roger Hoover <roger.hoo...@gmail.com> wrote: > +1 (non-binding). Thanks, Rajini > > On Tue, Aug 2

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

t; > 30*30, probably less than 100. If we were using Kafka > > Metrics > > > > for > > > > > > > this, > > > > > > > > > we > > > > > > > > > > could easily add a long expiration period to be > > conservative,

Re: [VOTE] KIP-152 - Improve diagnostics for SASL authentication failures

+1 (non-binding). Thanks, Rajini On Tue, Aug 29, 2017 at 2:10 AM, Ismael Juma wrote: > Thanks for the KIP, +1 (binding) from me. > > Ismael > > On Thu, Aug 24, 2017 at 6:29 PM, Rajini Sivaram > wrote: > > > Hi all, > > > > I would like to start vote

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

ajini > > On Thu, Aug 17, 2017 at 12:05 AM, Roger Hoover <roger.hoo...@gmail.com> > wrote: > > > I think it would useful to make clear somewhere for each metric, the > level > > at which it's counted. I don't know all the details of the Kafka > protocol > &

Re: [DISCUSS] KIP-187 - Add cumulative count metric for all Kafka rate metrics

Rajini, Thank you. This is very useful. Grouping by metric by prefixing the name instead of making them MBeans is not quite as nice but seems like an good compromise for backward compatibility. Cheers, Roger On Wed, Aug 16, 2017 at 5:35 AM, Rajini Sivaram wrote: >

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

? Cheers, Roger On Wed, Aug 16, 2017 at 4:02 PM, Roger Hoover <roger.hoo...@gmail.com> wrote: > Rajini, > > Thank you for the KIP. These are very helpful additions. One question on > the error code metrics: > > Will the total error counting happen at the the level of topic

Re: [DISCUSS] KIP-188 - Add new metrics to support health checks

Rajini, Thank you for the KIP. These are very helpful additions. One question on the error code metrics: Will the total error counting happen at the the level of topic partition? For example, if a single ProduceRequest contains messages to append to 3 partitions and say all 3 appends are

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

Well deserved, indeed! Congrats, Ismael. On Wed, Jul 5, 2017 at 3:24 PM, Damian Guy wrote: > Congratulations Ismael! Very well deserved. > Cheers, > Damian > On Wed, 5 Jul 2017 at 22:54, Dong Lin wrote: > > > Congratulations Ismael! > > > > On Wed,

Re: [VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

+1 Sent from my iPhone > On May 8, 2017, at 5:00 AM, Edoardo Comar wrote: > > +1 > Many thanks Jun > -- > Edoardo Comar > IBM MessageHub > eco...@uk.ibm.com > IBM UK Ltd, Hursley Park, SO21 2JN > > IBM United Kingdom Limited

Re: [DISCUSS] KIP-153 : Include only client traffic in BytesOutPerSec metric

Very helpful. Thank you, Jun. On Fri, May 5, 2017 at 4:42 PM, Guozhang Wang wrote: > Jun, > > Thanks for the KIP, LGTM. > > Guozhang > > On Fri, May 5, 2017 at 3:38 PM, Ismael Juma wrote: > > > Thanks Jun, looks good to me. > > > > Ismael > > > > On Fri,

[jira] [Commented] (KAFKA-3795) Transient system test failure upgrade_test.TestUpgrade

[ https://issues.apache.org/jira/browse/KAFKA-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15953759#comment-15953759 ] Roger Hoover commented on KAFKA-3795: - Happened again: http://confluent-systest.s3-website-us-west-2

Re: [DISCUSS] KIP-136: Add Listener name and Security Protocol name to SelectorMetrics tags

Edo, Thanks for the proposal. This looks great to me. Cheers, Roger On Thu, Mar 30, 2017 at 8:51 AM, Edoardo Comar wrote: > Hi all, > > We created KIP-136: Add Listener name and Security Protocol name to > SelectorMetrics tags > >

[jira] [Updated] (KAFKA-4755) SimpleBenchmark consume test fails for streams

[ https://issues.apache.org/jira/browse/KAFKA-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-4755: Priority: Blocker (was: Major) > SimpleBenchmark consume test fails for stre

[jira] [Reopened] (KAFKA-4755) SimpleBenchmark consume test fails for streams

[ https://issues.apache.org/jira/browse/KAFKA-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover reopened KAFKA-4755: - Happened again. Logs here: http://confluent-kafka-0-10-2-system-test-results.s3-us-west-2

[jira] [Commented] (KAFKA-4689) OffsetValidationTest fails validation with "Current position greater than the total number of consumed records"

[ https://issues.apache.org/jira/browse/KAFKA-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945543#comment-15945543 ] Roger Hoover commented on KAFKA-4689: - Happened again. Logs here http://confluent-kafka-0-10-2-system

Re: [VOTE] KIP-124: Request rate quotas

Rajini, This is great. Thank you. +1 (non-binding) Roger On Tue, Mar 21, 2017 at 8:55 AM, Ismael Juma wrote: > Rajini, > > Thanks for the proposal and for addressing the (sometimes contradictory) > feedback. :) +1 (binding) from me. > > Ismael > > On Mon, Mar 20, 2017 at

Re: [DISCUSS] KIP-124: Request rate quotas

Great to see this KIP and the excellent discussion. To me, Jun's suggestion makes sense. If my application is allocated 1 request handler unit, then it's as if I have a Kafka broker with a single request handler thread dedicated to me. That's the most I can use, at least. That allocation

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

naming of the > mechanisms would be a bit misleading if support for bcrypt was added > (SCRAM-PKBDF2-SHA512, SCRAM-BCRYPT*, etc. would be better). > > Does that make sense? > > Ismael > > On Tue, Jan 24, 2017 at 7:26 PM, Roger Hoover <roger.hoo...@gmail.com> > wrote:

Re: [VOTE] KIP-48 Support for delegation tokens as an authentication mechanism

> > Pl refer discuss mail thread for impersonation related discussion. > > Thanks, > Manikumar > > On Wed, Feb 8, 2017 at 8:37 AM, Roger Hoover <roger.hoo...@gmail.com> > wrote: > > > Hi Jun, > > > > How does it allow impersonation at the connection level? Lookin

Re: [VOTE] KIP-48 Support for delegation tokens as an authentication mechanism

at the > connection level. Are you talking about impersonation at the request level? > > Thanks, > > Jun > > On Tue, Feb 7, 2017 at 5:53 PM, Roger Hoover <roger.hoo...@gmail.com> > wrote: > > > Just wondering...how difficult would be it be to later add impersonatio

Re: [VOTE] KIP-48 Support for delegation tokens as an authentication mechanism

Just wondering...how difficult would be it be to later add impersonation ( https://issues.apache.org/jira/browse/KAFKA-3712)? One use case would be a Kafka admin UI that would take action on the cluster on behalf different users.I suppose we could later add an "effectiveUserId" (in Unix

Re: [DISCUSS] KIP-118: Drop Support for Java 7 in Kafka 0.11

This is great. Thanks, Ismael. On Fri, Feb 3, 2017 at 7:35 AM, Grant Henke wrote: > Looks good to me. Thanks for handling the KIP. > > On Fri, Feb 3, 2017 at 8:49 AM, Damian Guy wrote: > > > Thanks Ismael. Makes sense to me. > > > > On Fri, 3 Feb

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

A-256 and SCRAM-SHA-512 (not > SCRAM-SHA-1). > > Ismael > > On Mon, Jan 23, 2017 at 10:49 PM, Roger Hoover <roger.hoo...@gmail.com> > wrote: > > > Sorry for the late question but is there a reason to choose SHA-1 and > > SHA-256 instead of bcrypt? > > >

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

Sorry for the late question but is there a reason to choose SHA-1 and SHA-256 instead of bcrypt? https://codahale.com/how-to-safely-store-a-password/ On Fri, Nov 11, 2016 at 5:30 AM, Rajini Sivaram < rajinisiva...@googlemail.com> wrote: > I think all the comments and suggestions on this thread

[jira] [Closed] (KAFKA-4670) Kafka Consumer should validate FetchResponse

[ https://issues.apache.org/jira/browse/KAFKA-4670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover closed KAFKA-4670. --- > Kafka Consumer should validate FetchRespo

[jira] [Commented] (KAFKA-4670) Kafka Consumer should validate FetchResponse

[ https://issues.apache.org/jira/browse/KAFKA-4670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830670#comment-15830670 ] Roger Hoover commented on KAFKA-4670: - Ah, yes, thanks, [~ijuma]. > Kafka Consumer should valid

[jira] [Created] (KAFKA-4670) Kafka Consumer should validate FetchResponse

Roger Hoover created KAFKA-4670: --- Summary: Kafka Consumer should validate FetchResponse Key: KAFKA-4670 URL: https://issues.apache.org/jira/browse/KAFKA-4670 Project: Kafka Issue Type: Bug

Re: [VOTE] KIP-108: Create Topic Policy

+1 (non-binding) On Mon, Jan 9, 2017 at 2:15 AM, Edoardo Comar wrote: > Ismael, > thanks for the KIP I see it as quite useful in a managed cloud > environment. > > +1 (non-binding) > -- > Edoardo Comar > IBM MessageHub >

Re: [DISCUSS] KIP-108: Create Topic Policy

Policy`. I'll mention this explicitly in the KIP. > > Ismael > > On Mon, Jan 9, 2017 at 6:04 PM, Roger Hoover <roger.hoo...@gmail.com> > wrote: > > > This is great. Thanks, Ismael. > > > > One question. When TopicDetails are passed to the policy implementation, >

Re: [DISCUSS] KIP-108: Create Topic Policy

This is great. Thanks, Ismael. One question. When TopicDetails are passed to the policy implementation, would the server defaults already have been merged? If not, I think the policy also needs access to the server defaults. Cheers, Roger On Fri, Jan 6, 2017 at 9:26 AM, Ismael Juma

Re: KIP-103: Separation of Internal and External traffic

use that. > > Ismael > >> On Fri, Jan 6, 2017 at 9:42 PM, Roger Hoover <roger.hoo...@gmail.com> wrote: >> >> Ismael, >> >> Listener id would also convey uniqueness but I'm ok with listener key as >> well since it fits with the use of the term &qu

Re: KIP-103: Separation of Internal and External traffic

to be unique per > listener, so I prefer it a little over `listener_name`. Since the existing > config is called `listeners` instead of `protocol.listeners`, maybe we > don't need the protocol prefix? > > Ismael > >> On Fri, Jan 6, 2017 at 7:48 PM, Roger Hoover <roger.ho

Re: KIP-103: Separation of Internal and External traffic

Maybe it's clearer to to say protocol_listener_name? The proposed config allows you to name each listener and refer to their names in various places. On Wed, Jan 4, 2017 at 4:34 AM, Ismael Juma wrote: > Hi Colin, > > Thanks for the feedback. It's a good question regarding

Re: [VOTE] KIP-103: Separation of Internal and External traffic

+1 (non-binding) On Fri, Jan 6, 2017 at 11:16 AM, Tom Crayford wrote: > +1 (non-binding) > > On Fri, Jan 6, 2017 at 6:58 PM, Colin McCabe wrote: > > > Looks good. +1 (non-binding). > > > > What do you think about changing "protocol label" to "listener

[jira] [Commented] (KAFKA-4527) Transient failure of ConnectDistributedTest.test_pause_and_resume_sink where paused connector produces messages

[ https://issues.apache.org/jira/browse/KAFKA-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759269#comment-15759269 ] Roger Hoover commented on KAFKA-4527: - Happened again: http://confluent-systest.s3-website-us-west-2

[jira] [Commented] (KAFKA-3808) Transient failure in ReplicaVerificationToolTest

[ https://issues.apache.org/jira/browse/KAFKA-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759266#comment-15759266 ] Roger Hoover commented on KAFKA-3808: - Happened again: http://confluent-systest.s3-website-us-west-2

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759264#comment-15759264 ] Roger Hoover commented on KAFKA-4166: - Similar failure: http://confluent-systest.s3-website-us-west-2

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759260#comment-15759260 ] Roger Hoover commented on KAFKA-4526: - Thanks, [~apurva] > Transient fail

Re: [DISCUSS] KIP-82 - Add Record Headers

Matthias, Thanks for your input. I'm +1 on control messages as they seem to be the simplest way to implement watermarks ( https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102), a feature that would add a lot of value to Kafka Streams IMHO. Your argument that the control-message

[jira] [Commented] (KAFKA-3808) Transient failure in ReplicaVerificationToolTest

[ https://issues.apache.org/jira/browse/KAFKA-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15758169#comment-15758169 ] Roger Hoover commented on KAFKA-3808: - Happened again: {code

[jira] [Resolved] (KAFKA-4554) ReplicaVerificationToolTest.test_replica_lags system test failure

[ https://issues.apache.org/jira/browse/KAFKA-4554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover resolved KAFKA-4554. - Resolution: Duplicate Duplicate of https://issues.apache.org/jira/browse/KAFKA-3808

[jira] [Created] (KAFKA-4554) ReplicaVerificationToolTest.test_replica_lags system test failure

Roger Hoover created KAFKA-4554: --- Summary: ReplicaVerificationToolTest.test_replica_lags system test failure Key: KAFKA-4554 URL: https://issues.apache.org/jira/browse/KAFKA-4554 Project: Kafka

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15758158#comment-15758158 ] Roger Hoover commented on KAFKA-4166: - Failed again: http://confluent-systest.s3-website-us-west-2

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15758155#comment-15758155 ] Roger Hoover commented on KAFKA-4526: - Failed again {code

[jira] [Created] (KAFKA-4551) StreamsSmokeTest.test_streams intermittent failure

Roger Hoover created KAFKA-4551: --- Summary: StreamsSmokeTest.test_streams intermittent failure Key: KAFKA-4551 URL: https://issues.apache.org/jira/browse/KAFKA-4551 Project: Kafka Issue Type

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754984#comment-15754984 ] Roger Hoover commented on KAFKA-4166: - Happened again here: http://confluent-systest.s3-website-us

[jira] [Commented] (KAFKA-4527) Transient failure of ConnectDistributedTest.test_pause_and_resume_sink where paused connector produces messages

[ https://issues.apache.org/jira/browse/KAFKA-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754980#comment-15754980 ] Roger Hoover commented on KAFKA-4527: - Happened again: http://confluent-systest.s3-website-us-west-2

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754982#comment-15754982 ] Roger Hoover commented on KAFKA-4526: - Happened again here: http://confluent-systest.s3-website-us

[jira] [Comment Edited] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747454#comment-15747454 ] Roger Hoover edited comment on KAFKA-4166 at 12/14/16 6:37 AM: --- It happened

[jira] [Comment Edited] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747454#comment-15747454 ] Roger Hoover edited comment on KAFKA-4166 at 12/14/16 6:36 AM: --- It happened

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

[ https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747454#comment-15747454 ] Roger Hoover commented on KAFKA-4166: - It happened twice more on these test: {code} "M

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

[ https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15747308#comment-15747308 ] Roger Hoover commented on KAFKA-4526: - This happened again on the Dec 13 nightly run. > Transi

Re: [DISCUSS] KIP-82 - Add Record Headers

basically the same thing - > youre just namespacing an int64 and giving people while 2^32 ranges at a > time. the part i like about this is letting people have a large swath of > numbers with one registration so they dont have to come back for every > single plugin/header they want t

Re: [DISCUSS] KIP-82 - Add Record Headers

as headers > we > > > >> wouldn't > > > >> >> >>>>>> expect > > > >> >> >>>>>>>> the number of headers being used concurrently being that > > > high. > >

Re: [DISCUSS] KIP-82 - Add Record Headers

rk the same way > in > > > the > > > > ecosystem, having to have this also customizable makes it a bit > harder. > > > > Think about making the whole message record custom serializable, this > > > would > > > > make it fairly tricky (though

Re: [DISCUSS] KIP-82 - Add Record Headers

mat client could be completely separate from Apache Kafka > or it could be part of Apache Kafka. The people that would like to use > headers can use that client, and the people that think it's an overhead can > use the one without. > > > Nacho > > > On Mon, Nov 7, 2016 a

Re: [DISCUSS] KIP-82 - Add Record Headers

and > values of the form "1/20" someone who uses (contrived example) Map<Byte[], > Double> wouldnt be able to re-use my code. > > not the end of a the world within an organization, but problematic if you > want to enable an ecosystem > > On Thu, Nov 3, 2016 at

Re: [DISCUSS] KIP-82 - Add Record Headers

more > > intimate > > knowledge of the code that produced specific headers, but keys should > > be > > universally readable). > > > > it would also make it hard to write really portable plugins - say i > > wrote a > > large message s

Re: [DISCUSS] KIP-82 - Add Record Headers

to write really portable plugins - say i > wrote a > large message splitter/combiner - if i rely on key "largeMessage" and > values of the form "1/20" someone who uses (contrived example) > Map<Byte[], > Double> wouldnt be able to re-use my code. &g

Re: [DISCUSS] KIP-82 - Add Record Headers

As others have laid out, I see strong reasons for a common message metadata structure for the Kafka ecosystem. In particular, I've seen that even within a single organization, infrastructure teams often own the message metadata while application teams own the application-level data format.

[jira] [Updated] (KAFKA-4361) Streams does not respect user configs for "default" params

[ https://issues.apache.org/jira/browse/KAFKA-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-4361: Description: For the config params in CONSUMER_DEFAULT_OVERRIDES (https://github.com/apache/kafka

[jira] [Updated] (KAFKA-4361) Streams does not respect user configs for "default" params

[ https://issues.apache.org/jira/browse/KAFKA-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-4361: Description: For the config params in CONSUMER_DEFAULT_OVERRIDES (https://github.com/apache/kafka

[jira] [Updated] (KAFKA-4361) Streams does not respect user configs for "default" params

[ https://issues.apache.org/jira/browse/KAFKA-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-4361: Description: For the config params in CONSUMER_DEFAULT_OVERRIDES (https://github.com/apache/kafka

[jira] [Created] (KAFKA-4361) Streams does not respect user configs for "default" params

Roger Hoover created KAFKA-4361: --- Summary: Streams does not respect user configs for "default" params Key: KAFKA-4361 URL: https://issues.apache.org/jira/browse/KAFKA-4361 Project: Kafka

Re: [VOTE] Add REST Server to Apache Kafka

-1 I think a good set of HTTP APIs can simplify client implementations and allow for easier integration with Kafka as well as simpler ops/admin tools. Elasticsearch is a great example. However, I think it's too early to put this into the core. The existing Kafka REST proxies (that I know of)

[jira] [Created] (KAFKA-4331) Kafka Streams resetter is slow because it joins the same group for each topic

Roger Hoover created KAFKA-4331: --- Summary: Kafka Streams resetter is slow because it joins the same group for each topic Key: KAFKA-4331 URL: https://issues.apache.org/jira/browse/KAFKA-4331 Project

[jira] [Commented] (KAFKA-3993) Console producer drops data

[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450054#comment-15450054 ] Roger Hoover commented on KAFKA-3993: - Thanks, [~cotedm]. I tried to set acks=all but apparently

[jira] [Closed] (KAFKA-4063) Add support for infinite endpoints for range queries in Kafka Streams KV stores

[ https://issues.apache.org/jira/browse/KAFKA-4063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover closed KAFKA-4063. --- The JIRA UI was unresponsive so I accidentally submitted the form twice. > Add support for infin

[jira] [Created] (KAFKA-4063) Add support for infinite endpoints for range queries in Kafka Streams KV stores

Roger Hoover created KAFKA-4063: --- Summary: Add support for infinite endpoints for range queries in Kafka Streams KV stores Key: KAFKA-4063 URL: https://issues.apache.org/jira/browse/KAFKA-4063 Project

[jira] [Created] (KAFKA-4064) Add support for infinite endpoints for range queries in Kafka Streams KV stores

Roger Hoover created KAFKA-4064: --- Summary: Add support for infinite endpoints for range queries in Kafka Streams KV stores Key: KAFKA-4064 URL: https://issues.apache.org/jira/browse/KAFKA-4064 Project

[jira] [Commented] (KAFKA-3993) Console producer drops data

[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415811#comment-15415811 ] Roger Hoover commented on KAFKA-3993: - Thanks, [~vahid]. Yeah, it looks the same. > Console produ

[jira] [Updated] (KAFKA-3993) Console producer drops data

[ https://issues.apache.org/jira/browse/KAFKA-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-3993: Description: The console producer drops data when if the process exits too quickly. I suspect

[jira] [Commented] (KAFKA-3752) Provide a way for KStreams to recover from unclean shutdown

[ https://issues.apache.org/jira/browse/KAFKA-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404439#comment-15404439 ] Roger Hoover commented on KAFKA-3752: - [~guozhang] You're assessment seems correct. It happened again

[jira] [Created] (KAFKA-3993) Console producer drops data

Roger Hoover created KAFKA-3993: --- Summary: Console producer drops data Key: KAFKA-3993 URL: https://issues.apache.org/jira/browse/KAFKA-3993 Project: Kafka Issue Type: Bug Components

Re: [DISCUSS] Client Side Auto Topic Creation

My comments go a bit beyond just topic creation but I'd like to see Kafka make it easier for application developers to specify their requirements declaratively in a single place. Today, for example, if your application requires strong guarantees against data loss, you must set a mix of

[DISCUSS] KAFKA-3761: Controller has RunningAsBroker instead of RunningAsController state

Hi all, Does anyone have an issue with removing the broker state called "RunningAsController"? The reasons to remove it are: 1. It's currently broken. The purpose of the JIRA was to report that the RunningAsController state gets overwritten

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

Thanks, Eno. On Tue, Jun 21, 2016 at 2:22 AM, Eno Thereska <eno.there...@gmail.com> wrote: > Hi Roger, > > I realised I never got back to you on this one, sorry. Some answers inline: > > > On 3 Jun 2016, at 22:48, Roger Hoover <roger.hoo...@gmail.com> wrote

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[ https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334812#comment-15334812 ] Roger Hoover commented on KAFKA-3740: - [~h...@pinterest.com], any update on this? I'm wondering

[jira] [Created] (KAFKA-3858) Add functions to print stream topologies

Roger Hoover created KAFKA-3858: --- Summary: Add functions to print stream topologies Key: KAFKA-3858 URL: https://issues.apache.org/jira/browse/KAFKA-3858 Project: Kafka Issue Type: New Feature

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

Hi Eno, Does this mean that Kafka Streams will disable the RocksDB write buffer? Is it currently safe to recover a Kafka Streams application after SIGKILL on the same machine? If not, will this make it safe to do so? If RocksDB is not flushed before offsets are commited in Kafka and is killed

[jira] [Work started] (KAFKA-3761) Controller has RunningAsBroker instead of RunningAsController state

[ https://issues.apache.org/jira/browse/KAFKA-3761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3761 started by Roger Hoover. --- > Controller has RunningAsBroker instead of RunningAsController st

[jira] [Issue Comment Deleted] (KAFKA-3760) Set broker state as running after publishing to ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Hoover updated KAFKA-3760: Comment: was deleted (was: Create PR: https://github.com/apache/kafka/pull/1436) > Set bro

[jira] [Commented] (KAFKA-3760) Set broker state as running after publishing to ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15303141#comment-15303141 ] Roger Hoover commented on KAFKA-3760: - Create PR: https://github.com/apache/kafka/pull/1436 >

  1   2   >