Re: [DISCUSS] KIP-228 Negative record timestamp support

On Tue, Jan 2, 2018 at 8:04 PM, Matthias J. Sax wrote: > I was thinking about a broker/topic config. > > However, I am not sure if we only need to worry about data written in > the future (this would only be true, if there would be no records with > -1 timestamp already).

Re: [DISCUSS] KIP-242: Mask password fields in Kafka Connect REST response

Vincent, Thanks for the KIP. This is definitely an issue we know is a problem for some users. I think the major problem with the KIP as-is is that it makes it impossible to get the original value back out of the API. This KIP probably ties in significantly with ideas for securing the REST API

Re: [VOTE] KIP-237: More Controller Health Metrics

Dong, Seems like a reasonable addition. +1 (binding) from me. There were some final naming issues in the DISCUSS thread that didn't get follow up. I'm fine with either version of the naming as these are details I think mostly advanced users will be monitoring and both naming arguments are

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

Builders have historically seen some resistance in the project (by some notable original authors...) but they've been increasingly making their way in -- SchemaBuilder in Connect, I believe some small streams APIs, AdminClient stuff. The general trend seems to be towards more fluent APIs.

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

After investigation, I have found that the InternalStreamsBuilder#globalTable method is the only instance where the constructor for GlobalKTableImpl is called. The KTableValueGetterSupplier parameter used in this particular constructor is an instance of KTableSourceValueGetterSupplier. Hence, your

Re: [VOTE] KIP-233: Simplify StreamsBuilder#addGlobalStore

It must be two different names, as we add two processor to the topology: a source processor that only read the data from a topic, and the actual processor that maintains the global table. -Matthias On 1/2/18 7:14 PM, Ewen Cheslack-Postava wrote: > +1 binding, seems like a nice simplification. >

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hi, Dong, Thanks for the reply. To solve the topic recreation issue, we could use either a global metadata version or a partition level epoch. But either one will be a new concept, right? To me, the latter seems more natural. It also makes it easier to detect if a consumer's offset is still

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Tue, Jan 2, 2018, at 04:46, Becket Qin wrote: > Hi Colin, > > Good point about KIP-226. Maybe a separate broker epoch is needed although > it is a little awkward to let the consumer set this. So was there a > solution to the frequent pause and resume scenario? Did I miss something? > >

Build failed in Jenkins: kafka-trunk-jdk7 #3072

See Changes: [wangguoz] Revert "KAFKA-6383: complete shutdown for CREATED StreamThreads (#4343)" -- [...truncated 220.28 KB...] kafka.api.SslEndToEndAuthorizationTest >

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

+1 binding The idea seems reasonable. Looking at it implementation-wise, seems there is a bit of awkwardness because GlobalKTableImpl uses a KTableValueGetterSupplier which seems to possibly have multiple stores, but maybe using the more specific KTableSourceValueGetterSupplier implementation

Re: [VOTE] KIP-233: Simplify StreamsBuilder#addGlobalStore

+1 binding, seems like a nice simplification. Regarding the source and processor name, do they actually need to be unique or could they use the same value? Since these use incrementing integers, it could be nice for debuggability/understanding to have them use the same name if possible instead of

Re: [DISCUSS] KIP-228 Negative record timestamp support

For `allow.negative.timestamps`, do you mean this as a broker config? I'm not entirely clear on what the proposal would entail. I think taking into account whether we're talking about compatibility with existing data in Kafka vs enabling use of negative timestamps is important here. If they're

Re: [DISCUSS] KIP-228 Negative record timestamp support

I was thinking about a broker/topic config. However, I am not sure if we only need to worry about data written in the future (this would only be true, if there would be no records with -1 timestamp already). Assume that we you an existing topic that contains data with -1 = UNKNOWN records -- for

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

+1 (binding) Thanks for the KIP Vahid, nice improvement! -Ewen On Tue, Dec 19, 2017 at 11:30 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > I believe the concerns on this KIP have been addressed so far. > Therefore, I'd like to start a vote. > >

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

Late to the game here, but I'm +1 on this. Some of the ConsumerGroup permissions are weird, but this KIP brings the describe ACLs into better alignment with everything else and makes things more functional for clients with more locked down permissions. -Ewen On Fri, Dec 15, 2017 at 12:57 PM,

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

Oh, the KIP passes w/ the required votes. My comment was just on implementation details. I will leave comments about that up to the subsequent PR and to the Kafka Streams folks that are much better suited than me to comment on them :) -Ewen On Tue, Jan 2, 2018 at 9:28 PM, Richard Yu

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

A subsequent PR has already been created: https://github.com/apache/kafka/pull/4340/ It should be seen on the JIRA. On Tue, Jan 2, 2018 at 9:51 PM, Ewen Cheslack-Postava wrote: > Oh, the KIP passes w/ the required votes. My comment was just on > implementation details. I

[jira] [Created] (KAFKA-6419) Menu order in Streams docs incorrect

Matthias J. Sax created KAFKA-6419: -- Summary: Menu order in Streams docs incorrect Key: KAFKA-6419 URL: https://issues.apache.org/jira/browse/KAFKA-6419 Project: Kafka Issue Type: Bug

Re: [VOTE] KIP-239 Add queryableStoreName() to GlobalKTable

@Richard: you can close this vote thread with a summary as usual and update the KIP wiki page accordingly. -Matthias On 1/2/18 9:57 PM, Richard Yu wrote: > A subsequent PR has already been created: > https://github.com/apache/kafka/pull/4340/ > It should be seen on the JIRA. > > > > On Tue,

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

Thanks for the feedback. I updated the KIP and PR accordingly. -Matthias On 1/2/18 9:18 PM, Ewen Cheslack-Postava wrote: > Builders have historically seen some resistance in the project (by some > notable original authors...) but they've been increasingly making their way > in --

Re: [DISCUSS] KIP-222 - Add "describe consumer group" to KafkaAdminClient

listGroups and listGroupOffsets will make it a snap to transition the existing ConsumerGroups CLI to depend on client libraries only. Thanks for adding them :) On Sun, Dec 31, 2017 at 1:39 PM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Thanks all for your feedback, and

[jira] [Created] (KAFKA-6416) Create an official Kafka Helm chart for running a Kafka cluster

Matthew T. Adams created KAFKA-6416: --- Summary: Create an official Kafka Helm chart for running a Kafka cluster Key: KAFKA-6416 URL: https://issues.apache.org/jira/browse/KAFKA-6416 Project: Kafka

Re: [VOTE] KIP-233: Simplify StreamsBuilder#addGlobalStore

On a side note, could you update the "compatibility and upgrade" section, that when users start to make code changes to leverage the new API, what kind of upgrade executions they'd need to do? I feel they need to rename topics / etc. On Tue, Jan 2, 2018 at 9:10 AM, Guozhang Wang

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

There's been a bit of discussion on the PR about the choice of the default header converter. The proposal currently uses the new `SimpleHeaderConverter` so that by default connector devs and users get meaningful header values by default without much work. An alternative is to default to

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

I got the impression that use of Strings in headers is really common, so the SimpleHeaderConverter makes a lot of sense to me. Agree that this introduces overhead, but perhaps simply documenting an easy "optimization" will be enough to help those who are concerned about it? Since the

Re: [VOTE] KIP-233: Simplify StreamsBuilder#addGlobalStore

+1, thanks! On Wed, Dec 27, 2017 at 6:01 PM, Ted Yu wrote: > +1 > > On Wed, Dec 27, 2017 at 12:15 PM, Bill Bejeck wrote: > > > +1 > > > > On Wed, Dec 27, 2017 at 3:07 PM, Matthias J. Sax > > wrote: > > > > > +1 > > > > > > On

[jira] [Created] (KAFKA-6415) KafkaLog4jAppender deadlocks when logging from producer network thread

Rajini Sivaram created KAFKA-6415: - Summary: KafkaLog4jAppender deadlocks when logging from producer network thread Key: KAFKA-6415 URL: https://issues.apache.org/jira/browse/KAFKA-6415 Project:

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Colin, Good point about KIP-226. Maybe a separate broker epoch is needed although it is a little awkward to let the consumer set this. So was there a solution to the frequent pause and resume scenario? Did I miss something? Thanks, Jiangjie (Becket) Qin On Wed, Dec 27, 2017 at 1:40 PM,

[jira] [Created] (KAFKA-6418) adminclient throws timeoutexception when there is a SchemaException

dan norwood created KAFKA-6418: -- Summary: adminclient throws timeoutexception when there is a SchemaException Key: KAFKA-6418 URL: https://issues.apache.org/jira/browse/KAFKA-6418 Project: Kafka

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Thanks for the reply, Colin. My concern for the reinitialization is potential churn rather than efficiency. The current KIP proposal uses the time and priority based protection to avoid thrashing, but it is not clear to me if that is sufficient. For example, consider topic creation/deletion. In

Build failed in Jenkins: kafka-trunk-jdk8 #2306

See Changes: [wangguoz] KAFKA-6391 ensure topics are created with correct partitions BEFORE [wangguoz] KAFKA-6318: StreamsResetter should return non-zero return code on error [wangguoz] MINOR: refactored code

Build failed in Jenkins: kafka-trunk-jdk8 #2305

See Changes: [wangguoz] KAFKA-6383: complete shutdown for CREATED StreamThreads (#4343) [wangguoz] KAFKA-5368: Add test for skipped-records metric (#4365) [wangguoz] KAFKA-6269: KTable restore fails after

Re: Kafka Consumer manual partition assignment

Hi, Any help here would be highly appreciated :) Sagar. On Wed, 27 Dec 2017 at 07:53, Sagar wrote: > We have a use case where in we want to assign partitions manually for a > set of topics to allow fine grained control of the records we are fetching. > > Basically

Re: Vote for KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor

Boyang, Thanks for the proposed change, the wiki page lgtm. One minor comment otherwise I'm +1: For the new API, we now also have a constructor that accepts both a clientSupplier and a Time, so we should consider having four overloads in total: // New API (using Properties) public

Jenkins build is back to normal : kafka-trunk-jdk7 #3069

See

Pull Request emails are not being sent to the mailing list

Since Dec 21st, it seems that email notifications of Github pull requests are not being sent to the dev mailing list. Is this a bug, or have PR emails moved to somewhere else? If it’s a bug, can we look into this? The last email PR that was sent was this one:

Re: Pull Request emails are not being sent to the mailing list

pull request emails, at least the initial one, are recorded on the underlying JIRA. For the time being, you can get to pull request(s) of interest thru JIRA. FYI On Tue, Jan 2, 2018 at 1:30 PM, James Cheng wrote: > Since Dec 21st, it seems that email notifications of

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

A few thoughts, mostly just details: * Is the SchemaAndValue return type from Headers necessary? We needed to use that in Converters, but otherwise I believe no public API has to use that type. If possible I think it is better to avoid making Connector developers aware of that type. * For

[jira] [Created] (KAFKA-6417) plugin.path pointing at a plugin directory causes ClassNotFoundException

Dustin Cote created KAFKA-6417: -- Summary: plugin.path pointing at a plugin directory causes ClassNotFoundException Key: KAFKA-6417 URL: https://issues.apache.org/jira/browse/KAFKA-6417 Project: Kafka

[jira] [Resolved] (KAFKA-6269) KTable state restore fails after rebalance

[ https://issues.apache.org/jira/browse/KAFKA-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-6269. -- Resolution: Fixed > KTable state restore fails after rebalance >

[jira] [Resolved] (KAFKA-6383) StreamThread.shutdown doesn't clean up completely when called before StreamThread.start

[ https://issues.apache.org/jira/browse/KAFKA-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-6383. -- Resolution: Fixed Fix Version/s: 1.1.0 > StreamThread.shutdown doesn't clean up

Re: [VOTE] KIP-226 - Dynamic Broker Configuration

Hi, Rajini, Thank for the KIP. +1. Just a couple of minor comments below. 50. config.secret.*: Could you document how the encryption/decryption of passwd work? In particular, how do we support changing config.secret? 51. At the topic level, we also have leader.replication.throttled.replicas

[jira] [Reopened] (KAFKA-6383) StreamThread.shutdown doesn't clean up completely when called before StreamThread.start

[ https://issues.apache.org/jira/browse/KAFKA-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohan Desai reopened KAFKA-6383: I missed a race condition in my fix. If we start and then shutdown a streams thread without the