Re: Why standby tasks read from the StandbyTasks::checkpointedOffsets in assignStandbyPartitions()

2019-11-10 Thread Guozhang Wang
gt; Navinder > > On Sunday, 10 November, 2019, 12:50:39 am IST, Guozhang Wang < > wangg...@gmail.com> wrote: > > > Hello Navinder, > > Sorry for the late reply and thanks for bringing this up. I think this is > indeed a bug that needs to be fixed. > > The r

Re: Why standby tasks read from the StandbyTasks::checkpointedOffsets in assignStandbyPartitions()

2019-11-09 Thread Guozhang Wang
Hello Navinder, Sorry for the late reply and thanks for bringing this up. I think this is indeed a bug that needs to be fixed. The rationale behind was the following: for restoring active tasks and processing standby tasks, we are using the same consumer client within the thread (the

[jira] [Resolved] (KAFKA-8520) TimeoutException in client side doesn't have stack trace

2019-11-08 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8520. -- Resolution: Duplicate > TimeoutException in client side doesn't have stack tr

Re: [VOTE] KIP-544: Make metrics exposed via JMX configurable

2019-11-08 Thread Guozhang Wang
+1 (binding), thanks! Guozhang On Thu, Nov 7, 2019 at 11:17 AM Xavier Léauté wrote: > Hi everyone, > > I'd like to open up the vote for KIP-544 on making exposed JMX metrics more > configurable. > > >

Re: [DISCUSS] KIP-544: Make metrics exposed via JMX configurable

2019-11-08 Thread Guozhang Wang
Sounds good, thanks. Guozhang On Fri, Nov 8, 2019 at 9:26 AM Xavier Léauté wrote: > > > > 1. I do feel there're similar needs for clients make JMX configurable. > Some > > context: in modules like Connect and Streams we have added / refactored a > > large number of metrics so far [0, 1], and

Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

2019-11-07 Thread Guozhang Wang
Hello Brian, Could you elaborate a bit more on this sentence: "This logic can be made more intelligent by managing the expiry from when the topic was last used, enabling the expiry duration to be reduced to improve cases where a large number of topics are touched intermittently." Not sure I fully

Re: [DISCUSS] KIP-544: Make metrics exposed via JMX configurable

2019-11-07 Thread Guozhang Wang
Thanks Xavier for the KIP, I think it is a great idea to add to AK. A couple of questions / comments: 1. I do feel there're similar needs for clients make JMX configurable. Some context: in modules like Connect and Streams we have added / refactored a large number of metrics so far [0, 1], and

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

2019-11-07 Thread Guozhang Wang
be > > >>>>>>>>>>> started on this. On Saturday, 26 October, 2019, 05:41:44 > > >>> pm > > >>>>> IST, > > >>>>>>>> Navinder > > >>>>>>>>>>> Brar wrote: > > >&

Re: [VOTE] KIP-150 - Kafka-Streams Cogroup

2019-11-06 Thread Guozhang Wang
+1 (binding), Thanks Walker! Guozhang On Wed, Nov 6, 2019 at 8:41 AM Bill Bejeck wrote: > Thanks for the KIP. > +1 (binding) > > -Bill > > > On Wed, Nov 6, 2019 at 1:22 AM Matthias J. Sax > wrote: > > > +1 (binding) > > > > > > On 10/31/19 10:52 AM, Walker Carlson wrote: > > > Hello all, > >

Re: [DISCUSS] KIP-280: Enhanced log compaction

2019-11-04 Thread Guozhang Wang
ey. If there are multiple values for a key, it > seems more intuitive that the newest value is the one that should be used > for compaction. > > Thanks, > Eric > > On Mon, Nov 4, 2019 at 11:00 AM Guozhang Wang wrote: > > > Hello Senthilnathan, > > > > Than

Re: [VOTE] KIP-280: Enhanced log compaction

2019-11-04 Thread Guozhang Wang
I only have one minor comment on the DISCUSS thread, otherwise I'm +1 (binding). On Mon, Nov 4, 2019 at 9:53 AM Senthilnathan Muthusamy wrote: > Hi all, > > I would like to start the vote on the updated KIP-280: Enhanced log > compaction. Thanks to Guozhang, Matthias & Tom for the valuable

Re: [DISCUSS] KIP-280: Enhanced log compaction

2019-11-04 Thread Guozhang Wang
equired to avoid duplicates. > > > > And the timestamp strategy is from the original KIP author and we are > keeping it as is. > > > > Finally on the sequence order guarantee by the producer, it is not > feasible on waiting for ack in async / multi-threads/processes sce

Re: [VOTE] KIP-523 Add KStream#toTable to the Streams DSL

2019-11-04 Thread Guozhang Wang
+1 (binding), thanks Aishwarya! On Sun, Nov 3, 2019 at 11:46 AM aishwarya kumar wrote: > This thread has been open for more than 72 hours. So far there are 2 > binding and 1 non-binding votes, looking to conclude this quickly!! > > Best, > Aishwarya > > On Mon, Oct 28, 2019 at 5:00 PM John

[jira] [Resolved] (KAFKA-8972) KafkaConsumer.unsubscribe could leave inconsistent user rebalance callback state

2019-11-01 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8972. -- Resolution: Fixed > KafkaConsumer.unsubscribe could leave inconsistent user rebalance callb

[jira] [Resolved] (KAFKA-9073) Kafka Streams State stuck in rebalancing after one of the StreamThread encounters java.lang.IllegalStateException: No current assignment for partition

2019-10-31 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9073. -- Resolution: Fixed > Kafka Streams State stuck in rebalancing after one of the StreamThr

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

2019-10-31 Thread Guozhang Wang
le streams into one KTable can be done more efficiently than making > > multiple KTables and then joining them together. We may be able to get > > around this in the case of a stitching join but I am not sure how we > could > > do it safely otherwise. > > > > Walker

Re: [Jira contributor list] Adding request

2019-10-29 Thread Guozhang Wang
Hello Ahmed, I've added you to the contributors list, thank you! Guozhang On Mon, Oct 28, 2019 at 11:37 AM Ahmed Oubalas wrote: > Hi all, > > I'm new to apache kafka project and would love to contribute to it by > fixing bugs/documentation. > > As per

Re: contribution

2019-10-29 Thread Guozhang Wang
Hi Jianhai, I've added you to the contributor list, you should be able to assign tickets to yourself now. For the newbie tasks, you can take a look at those un-assigned or assigned-but-no-longer actively worked on tasks (you can contact the current assignee on the ticket itself by leaving a

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

2019-10-28 Thread Guozhang Wang
contend with the fact that it looks like the > > KGroupedStream API, but behaves differently. > > > > What do you think about all this? > > > > Thanks again for the KIP and the discussion! > > -John > > > > On Mon, Oct 28, 2019 at 3:32 PM Walker Carlson > >

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

2019-10-28 Thread Guozhang Wang
t; operation. > > 3. There is only one processor made. We are actually having the naming > conversation right now in the above thread > > 4, 5. fair points > > Walker > > On Fri, Oct 25, 2019 at 11:58 AM Guozhang Wang wrote: > > > Hi Walker, thanks for the KIP! I made

[jira] [Resolved] (KAFKA-8800) Flaky Test SaslScramSslEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

2019-10-28 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8800. -- Fix Version/s: (was: 2.5.0) Resolution: Duplicate > Flaky T

[jira] [Resolved] (KAFKA-8937) Flaky Test SaslPlainSslEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

2019-10-28 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8937. -- Resolution: Duplicate > Flaky Test > SaslPlainSslEndToEndAuthorizati

[jira] [Resolved] (KAFKA-9045) Flaky Test DelegationTokenEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

2019-10-28 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9045. -- Resolution: Duplicate > Flaky Test > DelegationTokenEndToEndAuthorizati

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

2019-10-25 Thread Guozhang Wang
Hi Walker, thanks for the KIP! I made a pass on the writeup and have some comments below: Meta: 1. Syntax-wise, I'm wondering if we have compared our current proposal with Spark's co-group syntax (I know they are targeting for different use cases, but wondering if their syntax is closer to the

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

2019-10-24 Thread Guozhang Wang
, it > > will > > > > >> happen in an async manner. Hence, I am wondering if we should > > address > > > > >> this race condition (I think we should). The idea would be to > check > > if a > > > > >> standby/active(restoring) task is actuall

Re: [ANNOUNCE] Apache Kafka 2.3.1

2019-10-24 Thread Guozhang Wang
thank you for the following 41 contributors to this release! > > A. Sophie Blee-Goldman, Arjun Satish, Bill Bejeck, Bob Barrett, Boyang > Chen, Bruno Cadonna, Cheng Pan, Chia-Ping Tsai, Chris Egerton, Chris > Stromberger, Colin P. Mccabe, Colin Patrick McCabe, cpettitt-confluent, >

Re: [VOTE] KIP-530: Consider renaming 'UsePreviousTimeOnInvalidTimeStamp' class to 'UsePartitionTimeOnInvalidTimeStamp'

2019-10-24 Thread Guozhang Wang
+1 (binding). Guozhang On Thu, Oct 24, 2019 at 8:22 AM Bill Bejeck wrote: > Hi Rabi, > > Thanks for the KIP. > Just to be clear, we're going to deprecate > `UsePreviousTimeOnInvalidTimeStamp` > and add `UsePartitionOnInvalidTimeStamp` class correct? > > Otherwise, it's a +1 (binding) from me.

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

2019-10-21 Thread Guozhang Wang
Hi Navinder, Thanks for the KIP, I have a high level question about the proposed API regarding: "StreamsMetadataState::allMetadataForKey(boolean enableReplicaServing...)" I'm wondering if it's more general to refactor it to "allMetadataForKey(long tolerableDataStaleness, ...)", and when it's

[jira] [Resolved] (KAFKA-8940) Flaky Test SmokeTestDriverIntegrationTest.shouldWorkWithRebalance

2019-10-21 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8940. -- Fix Version/s: 2.5.0 Resolution: Fixed > Flaky T

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

2019-10-20 Thread Guozhang Wang
Thanks for the catch John, the JIRA description looks good! On Sat, Oct 19, 2019 at 11:43 PM Matthias J. Sax wrote: > Thanks John! > > Good catch! > > On 10/17/19 12:11 PM, Adam Bellemare wrote: > > Awesome. Thanks John for fixing this! > > > > On Thu, Oct 17, 2019 at 3:07 PM John Roesler

Re: [VOTE] KIP-534: Retain tombstones for approximately delete.retention.ms milliseconds

2019-10-18 Thread Guozhang Wang
We have run into scenarios in the past as well where > > recompression has actually increased message size. We may also want to be > > able to upconvert messages to the new format in the future in the > cleaner. > > > > -Jason > > > > > > &g

Re: [VOTE] KIP-534: Retain tombstones for approximately delete.retention.ms milliseconds

2019-10-17 Thread Guozhang Wang
recompression has actually increased message size. We may also want to be > able to upconvert messages to the new format in the future in the cleaner. > > -Jason > > > > On Thu, Oct 17, 2019 at 9:08 AM Guozhang Wang wrote: > > > Here's my understanding: when log com

[jira] [Reopened] (KAFKA-8700) Flaky Test QueryableStateIntegrationTest#queryOnRebalance

2019-10-17 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang reopened KAFKA-8700: -- Re-opening to further investigation cc [~cpettitt-confluent] > Flaky T

Re: [VOTE] KIP-534: Retain tombstones for approximately delete.retention.ms milliseconds

2019-10-17 Thread Guozhang Wang
clear on how small the delta > (or big) is. > To be fair, would the the delta size pose a big problem if it takes up more > bytes to encode? > > Cheers, > Richard > > On Wed, Oct 16, 2019 at 7:36 PM Guozhang Wang wrote: > > > Hello Richard, > > > > Thanks fo

[jira] [Resolved] (KAFKA-8891) invalid assignment protocol java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null

2019-10-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8891. -- Fix Version/s: 2.4.0 Resolution: Fixed > invalid assignment proto

[jira] [Resolved] (KAFKA-7263) Container exception java.lang.IllegalStateException: Coordinator selected invalid assignment protocol: null

2019-10-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-7263. -- Fix Version/s: 2.4.0 Resolution: Fixed > Container except

[jira] [Resolved] (KAFKA-9000) Flaky Test KTableKTableForeignKeyJoinIntegrationTest.doLeftJoinFromRightThenDeleteRightEntity

2019-10-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9000. -- Fix Version/s: 2.4.0 Resolution: Fixed > Flaky T

Re: [VOTE] KIP-534: Retain tombstones for approximately delete.retention.ms milliseconds

2019-10-16 Thread Guozhang Wang
Hello Richard, Thanks for the KIP, I just have one clarification regarding "So the idea is to set the base timestamp to the delete horizon and adjust the deltas accordingly." My understanding is that during compaction, for each compacted new segment, we would set its base offset of each batch as

[jira] [Resolved] (KAFKA-9039) Optimize replica fetching CPU utilization with large number of partitions

2019-10-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9039. -- Fix Version/s: 2.5.0 Assignee: Lucas Bradstreet Resolution: Fixed > Optim

[jira] [Resolved] (KAFKA-9032) Serializer is invoked with null when foreignValue=null

2019-10-16 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-9032. -- Resolution: Fixed > Serializer is invoked with null when foreignValue=n

[jira] [Resolved] (KAFKA-5566) Flaky Test QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied

2019-10-15 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5566. -- Fix Version/s: (was: 1.0.0) 2.4.0 Resolution: Fixed > Flaky T

[jira] [Resolved] (KAFKA-8700) Flaky Test QueryableStateIntegrationTest#queryOnRebalance

2019-10-15 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8700. -- Assignee: Guozhang Wang Resolution: Fixed This is a duplicate of KAFKA-4222. > Fl

[jira] [Resolved] (KAFKA-4222) Flaky Test QueryableStateIntegrationTest.queryOnRebalance

2019-10-15 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-4222. -- Fix Version/s: (was: 0.11.0.0) 2.4.0 Assignee: Guozhang Wang

[jira] [Resolved] (KAFKA-8942) Document RocksDB metrics

2019-10-15 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8942. -- Fix Version/s: 2.4.0 Resolution: Fixed Issue resolved by pull request 7490 [https

Re: KAFKA-8104: Help with the review

2019-10-14 Thread Guozhang Wang
Hello Nikolay, I'm still on your PR, but was swamped with some other issues as the release code freeze date's approaching, will try to make another pass on it asap. Guozhang On Mon, Oct 14, 2019 at 12:46 PM Nikolay Izhikov wrote: > Hello. > > I got very helpfull advices from guozhang. > And

[jira] [Created] (KAFKA-9039) Optimize replica fetching CPU utilization with large number of partitions

2019-10-14 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-9039: Summary: Optimize replica fetching CPU utilization with large number of partitions Key: KAFKA-9039 URL: https://issues.apache.org/jira/browse/KAFKA-9039 Project

Re: Please add wangfenjin to the contributiors list

2019-10-12 Thread Guozhang Wang
Hi Fenjin, Done. Thanks! Guozhang On Sat, Oct 12, 2019 at 2:41 AM Fenjin Wang wrote: > Hi, > > I'm interested to contribute to Kafka, please add me into the list so I > can assign > JIRA issues to myself. Thanks. > > Apache ID: wangfenjin > > -- > Fenjin Wang > -- -- Guozhang

[jira] [Created] (KAFKA-9033) Change default client-id in consumer / producer to be more meaningful

2019-10-11 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-9033: Summary: Change default client-id in consumer / producer to be more meaningful Key: KAFKA-9033 URL: https://issues.apache.org/jira/browse/KAFKA-9033 Project: Kafka

Re: [VOTE] KIP-511: Collect and Expose Client's Name and Version in the Brokers

2019-10-11 Thread Guozhang Wang
Hi David, Thanks for the KIP. I know I'm late on voting here but I'd be +1 on this too! Just a quick clarification on the tag "name=Connections" of the third metric, what would be the format of "Connections" here? Would that be the connection listener name? Guozhang On Fri, Oct 11, 2019 at

[jira] [Created] (KAFKA-9020) Streams sub-topologies should be sorted by sink -> source relationship

2019-10-10 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-9020: Summary: Streams sub-topologies should be sorted by sink -> source relationship Key: KAFKA-9020 URL: https://issues.apache.org/jira/browse/KAFKA-9020 Project: Ka

Re: [VOTE] KIP-527: Add VoidSerde to Serdes

2019-10-08 Thread Guozhang Wang
Thanks, I'm +1 (binding). On Tue, Oct 8, 2019 at 2:33 AM Nikolay Izhikov wrote: > Hello, Guozhang. > > Following added to the KIP: > > > If not null parameters passed then an java.lang.IllegalArgumentException > will be thrown. > > В Пн, 07/10/2019 в 15:51 -0700, Guo

[jira] [Created] (KAFKA-8995) Add new metric on broker to illustrate produce request compression percentage

2019-10-07 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-8995: Summary: Add new metric on broker to illustrate produce request compression percentage Key: KAFKA-8995 URL: https://issues.apache.org/jira/browse/KAFKA-8995 Project

Re: [VOTE] KIP-527: Add VoidSerde to Serdes

2019-10-07 Thread Guozhang Wang
Hello Nikolay, Could you clarify in the wiki doc what would happen if the passed in `byte[] bytes` or `Void data` parameters are not null? From the example PR we would throw exception here. I think it worth documenting it as part of the contract in the wiki page. Otherwise, I'm +1. Guozhang

Re: [DISCUSS] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

2019-10-04 Thread Guozhang Wang
Hello Brian, Thanks for the KIP. I think using asynchronous metadata update to address 1) metadata update blocking send, but for other issues, currently at producer we do have a configurable `METADATA_MAX_AGE_CONFIG` similar to consumer, by default is 5min. So maybe we do not need to introduce

Re: [VOTE] KIP-429: Kafka Consumer Incremental Rebalance Protocol

2019-09-30 Thread Guozhang Wang
+Rebalance+Protocol#KIP-429:KafkaConsumerIncrementalRebalanceProtocol-ConsumerMetrics Guozhang On Mon, Aug 5, 2019 at 2:36 PM Guozhang Wang wrote: > Hello folks, > > I've also updated the wiki page by making the augmented > `ConsumerPartitionAssignor` out as

[jira] [Resolved] (KAFKA-8609) Add consumer metrics for rebalances (part 9)

2019-09-30 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8609. -- Resolution: Fixed > Add consumer metrics for rebalances (par

[jira] [Resolved] (KAFKA-8427) Error while cleanup under windows for EmbeddedKafkaCluster

2019-09-27 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8427. -- Fix Version/s: 2.4.0 Assignee: Guozhang Wang Resolution: Fixed Should have

[jira] [Resolved] (KAFKA-8319) Flaky Test KafkaStreamsTest.statefulTopologyShouldCreateStateDirectory

2019-09-27 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8319. -- Fix Version/s: 2.4.0 Assignee: Guozhang Wang (was: Bill Bejeck) Resolution

Re: KIP Permissions

2019-09-26 Thread Guozhang Wang
Hello Michal, I can add you to the wiki space so you can edit the KIP. I searched the users list and cannot find a profile that matches your full name, but there's an id named "atais". Is that you? Guozhang On Thu, Sep 26, 2019 at 5:40 AM Michał Siatkowski wrote: > In article > > >

Kafka Meetup hosted by Confluent at San Francisco, Tuesday 6:30pm, October 1st, 2019

2019-09-25 Thread Guozhang Wang
Hello folks, This is a kind reminder of the Bay Area Kafka® meetup next Tuesday 6:30pm, at Confluent's San Francisco office. *RSVP and Register* (if you intend to attend in person): https://www.meetup.com/KafkaBayArea/events/264562779/ *Date* 6:30pm, Tuesday, October 1st, 2019 *Location*

[jira] [Resolved] (KAFKA-8880) Augment Consumer.committed(partition) to allow multiple partitions

2019-09-24 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8880. -- Fix Version/s: 2.4.0 Assignee: Guozhang Wang Resolution: Fixed > Augm

[jira] [Created] (KAFKA-8940) Flaky SmokeTestDriverIntegrationTest.shouldWorkWithRebalance

2019-09-24 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-8940: Summary: Flaky SmokeTestDriverIntegrationTest.shouldWorkWithRebalance Key: KAFKA-8940 URL: https://issues.apache.org/jira/browse/KAFKA-8940 Project: Kafka

[jira] [Resolved] (KAFKA-8086) Flaky Test GroupAuthorizerIntegrationTest#testPatternSubscriptionWithTopicAndGroupRead

2019-09-23 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8086. -- Resolution: Fixed > Flaky Test > GroupAuthorizerIntegrati

[jira] [Resolved] (KAFKA-7990) Flaky Test KafkaStreamsTest#shouldCleanupOldStateDirs

2019-09-20 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-7990. -- Fix Version/s: 2.3.0 2.2.1 Resolution: Fixed > Flaky T

[jira] [Reopened] (KAFKA-8319) Flaky Test KafkaStreamsTest.statefulTopologyShouldCreateStateDirectory

2019-09-20 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang reopened KAFKA-8319: -- > Flaky Test KafkaStreamsTest.statefulTopologyShouldCreateStateDirect

Re: [VOTE] KIP-470: TopologyTestDriver test input and output usability improvements

2019-09-20 Thread Guozhang Wang
+1 (binding), Thanks! On Fri, Sep 20, 2019 at 9:56 AM Matthias J. Sax wrote: > Thanks for the KIP Jukka! > > +1 (binding) > > -Matthias > > On 9/20/19 9:44 AM, Paul Whalen wrote: > > +1 (non-binding). I haven’t contributed to the discussion but I’ve been > following - it’ll definitely make my

Re: [VOTE] KIP-528: Deprecate PartitionGrouper configuration and interface

2019-09-19 Thread Guozhang Wang
+1, we should do that long ago :) Guozhang On Thu, Sep 19, 2019 at 4:28 PM Matthias J. Sax wrote: > Hi, > > I would like to propose a small KIP to deprecate some public APIs that > are considered "dangerous" to use, with the goal to remove them in the > next major release. > > Because the KIP

Re: [VOTE] KIP-479 Add Materialized to Join

2019-09-19 Thread Guozhang Wang
> > >> As for the name, I originally had "StreamJoined" and updated it after > > some > > >> comments on the KIP. > > >> I do feel that the name "StreamJoin" is better in this case since it > is > > >> used to represen

[jira] [Created] (KAFKA-8925) Flaky Test kafka.api.AuthorizerIntegrationTest.testIdempotentProducerNoIdempotentWriteAclInProduce

2019-09-19 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-8925: Summary: Flaky Test kafka.api.AuthorizerIntegrationTest.testIdempotentProducerNoIdempotentWriteAclInProduce Key: KAFKA-8925 URL: https://issues.apache.org/jira/browse/KAFKA-8925

Re: [VOTE] KIP-479 Add Materialized to Join

2019-09-18 Thread Guozhang Wang
the following votes: > > > > binding +1s: 3 (Guozhang, Matthias, and Bill) > > -1 votes: none > > > > Thanks to everyone who voted and participated in the discussion for this > > KIP! > > > > -Bill > > > > On Mon, Jul 29, 2019 at 6:03 P

[jira] [Resolved] (KAFKA-8919) Flaky Test kafka.api.AuthorizerIntegrationTest.testSimpleConsumeWithOffsetLookupAndNoGroupAccess

2019-09-17 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8919. -- Resolution: Fixed Realize it is fixed as part of polling (1L) in https://github.com/apache

[jira] [Created] (KAFKA-8919) Flaky Test kafka.api.AuthorizerIntegrationTest.testSimpleConsumeWithOffsetLookupAndNoGroupAccess

2019-09-17 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-8919: Summary: Flaky Test kafka.api.AuthorizerIntegrationTest.testSimpleConsumeWithOffsetLookupAndNoGroupAccess Key: KAFKA-8919 URL: https://issues.apache.org/jira/browse/KAFKA-8919

[jira] [Created] (KAFKA-8918) Flaky Test org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskCancellation

2019-09-17 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-8918: Summary: Flaky Test org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskCancellation Key: KAFKA-8918 URL: https://issues.apache.org/jira/browse/KAFKA-8918

Re: [VOTE] KIP-481: SerDe Improvements for Connect Decimal type in JSON

2019-09-17 Thread Guozhang Wang
+1 (binding). Thanks Almog! Guozhang On Tue, Sep 17, 2019 at 11:56 AM Bill Bejeck wrote: > +1 (binding) > > On Tue, Sep 17, 2019 at 11:32 AM Matthias J. Sax > wrote: > > > +1 (binding) > > > > -Matthias > > > > On 9/17/19 12:20 AM, Arjun Satish wrote: > > > Thanks for the KIP, Almog. > > > >

Re: [DISCUSS] KIP-479: Add Materialized to Join

2019-09-17 Thread Guozhang Wang
we are using. (More thoughts later.) > >> > > > > > >> > > > > 1b) I don't see an issue with allowing users to query all > stores? > >> > What > >> > > > > is the rational behind it? What do we gain by not allowing it? > >> &

[jira] [Resolved] (KAFKA-8755) Stand-by Task of an Optimized Source Table Does Not Write Anything to its State Store

2019-09-13 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8755. -- Resolution: Fixed > Stand-by Task of an Optimized Source Table Does Not Write Anything to

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

2019-09-13 Thread Guozhang Wang
> > I don't have a strong preference. So I am also fine to deprecate > > the > > > > > > existing methods. Let's see what Jason thinks. > > > > > > > > > > > > Can you update the KIP to reflect the semantics of the return > `Map` > > > >

[jira] [Resolved] (KAFKA-8355) Add static membership to Range assignor

2019-09-12 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8355. -- Fix Version/s: 2.4.0 Resolution: Fixed > Add static membership to Range assig

[jira] [Resolved] (KAFKA-8817) Flaky Test KafkaProducerTest.testCloseIsForcedOnPendingAddOffsetRequest

2019-09-10 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8817. -- Fix Version/s: 2.4.0 Assignee: Guozhang Wang Resolution: Fixed > Flaky T

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

2019-09-10 Thread Guozhang Wang
that may have been better, but we already have exposed the singular APIs > > and users are depending on them. Not sure it's worth breaking > compatibility > > just for aesthetics. > > > > -Jason > > > > On Tue, Sep 10, 2019 at 9:41 AM Guozhang Wang > wro

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

2019-09-10 Thread Guozhang Wang
at 6:09 AM Mickael Maison > wrote: > > > +1 (non-binding), thanks Guozhang > > > > On Tue, Sep 10, 2019 at 1:14 AM Boyang Chen > > wrote: > > > > > > Hey Guozhang, > > > > > > LGTM, +1 (non-binding) > > > > > >

[jira] [Resolved] (KAFKA-8889) Root cause is lost for FetchSessionHandler.handleError

2019-09-09 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8889. -- Resolution: Fixed > Root cause is lost for FetchSessionHandler.handleEr

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

2019-09-09 Thread Guozhang Wang
ourse, we need to record which panes are non-empty). Assuming we can have a better way of representing a secondary index out of the local state storage engine, I feel such implementation should be sufficiently better than our current implementation. > > > -Matthias > > > > > On 9/9/19

[VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

2019-09-09 Thread Guozhang Wang
Hello folks, I've created a new KIP allowing consumer.committed to take a set of partitions instead of just one partition to allow batching effects of such requests (the protocol already allows us to send multiple partitions in one round-trip):

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

2019-09-09 Thread Guozhang Wang
(initializer, aggregator, > merger/combinator/whatever) lets you supply lambdas for all the args, > and also makes it clear that you're getting different (more efficient) > execution behavior. > > WDYT? > > Thanks again, > -John > > > On Wed, Sep 4, 2019 at 7:53 PM

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2019-09-09 Thread Guozhang Wang
mentation can have some safety guard like this. Guozhang > Boyang > > On Thu, Sep 5, 2019 at 8:43 PM Guozhang Wang wrote: > > > Hello Boyang, > > > > Just realized one thing about timeout configurations that we should > > consider including in this KIP as well:

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2019-09-05 Thread Guozhang Wang
re it is indeed the case, we should override the default values to obey the above rules. Guozhang On Thu, Sep 5, 2019 at 5:47 PM Guozhang Wang wrote: > Thanks Boyang, I'm +1 on the KIP. > > Could you also update the detailed design doc > https:/

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

2019-09-05 Thread Guozhang Wang
Thanks Boyang, I'm +1 on the KIP. Could you also update the detailed design doc https://docs.google.com/document/d/1LhzHGeX7_Lay4xvrEXxfciuDWATjpUXQhrEIkph9qRE/edit which seems a bit out-dated with the latest proposal? Guozhang On Wed, Sep 4, 2019 at 10:45 AM Boyang Chen wrote: > Hey all, >

[jira] [Created] (KAFKA-8880) Augment Consumer.committed(partition) to allow multiple partitions

2019-09-05 Thread Guozhang Wang (Jira)
Guozhang Wang created KAFKA-8880: Summary: Augment Consumer.committed(partition) to allow multiple partitions Key: KAFKA-8880 URL: https://issues.apache.org/jira/browse/KAFKA-8880 Project: Kafka

[jira] [Resolved] (KAFKA-8793) StickyTaskAssignor throws java.lang.ArithmeticException

2019-09-05 Thread Guozhang Wang (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8793. -- Resolution: Not A Problem > StickyTaskAssignor throws java.lang.ArithmeticExcept

Re: [VOTE] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-05 Thread Guozhang Wang
+1 (binding). On Thu, Sep 5, 2019 at 2:47 PM John Roesler wrote: > Hello, all, > > After a great discussion, I'd like to open voting on KIP-441, > to avoid long restore times in Streams after rebalancing. > Please cast your votes! > > >

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

2019-09-05 Thread Guozhang Wang
for each single test. > > Best, > Bruno > > On Thu, Sep 5, 2019 at 8:46 PM Guozhang Wang wrote: > > > > Hi Bruno, > > > > Thanks for raising this point. I think the main motivation behind this > > proposal is, like Matthias said, to ease the un

Re: [VOTE] KIP-444: Refactor and Augment Metrics for Kafka Streams

2019-09-05 Thread Guozhang Wang
gt; > Hi Guozhang, > > > > > > +1 (non-binding) > > > > > > Thank you for driving this! > > > Bruno > > > > > > On Tue, Aug 20, 2019 at 8:29 PM Guozhang Wang > > wrote: > > > > > > > > Hello

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

2019-09-05 Thread Guozhang Wang
ics. > >> 2. added a task-level INFO metric "dropped-records" that covers all > >> scenarios and merges in the existing "late-records-drop", > >> "skipped-records", and "expired-window-records-drop". > >> 3. renamed the util functions

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-05 Thread Guozhang Wang
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-IterativeBalancingAssignments > . > > Thanks! > -John > > On Wed, Sep 4, 2019 at 5:58 PM Guozhang Wang wrote: > > > Hi John,

Re: [DISCUSS] KIP-450: Sliding Window Aggregations in the DSL

2019-09-04 Thread Guozhang Wang
d why using this extractor would not > meet the requirements for processing-time sliding windows? > > > -Matthias > > > On 4/16/19 10:16 AM, Guozhang Wang wrote: > > Regarding 4): yes I agree with you that invertibility is not a common > > property for agg-functions. Jus

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-04 Thread Guozhang Wang
s stateless tasks when it doesn't change the > balance at all to do so. It seems like this should accomplish the same > spiritual goal with no special cases. > > WDYT? > -John > > On Wed, Sep 4, 2019 at 2:26 PM Guozhang Wang wrote: > > > Hello John, > > > >

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

2019-09-04 Thread Guozhang Wang
a "should" clause to the assignment spec: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP-441:SmoothScalingOutforKafkaStreams-AssignmentAlgorithm > > > > > > For example, we would sort

Re: [VOTE] KIP-401: TransformerSupplier/ProcessorSupplier StateStore connecting

2019-09-03 Thread Guozhang Wang
he KIP for clarity. > > > On Sep 3, 2019, at 5:04 PM, Matthias J. Sax > wrote: > > > > I am strongly in favor of "must be the same reference". > > > > > > -Matthias > > > >> On 9/3/19 2:09 PM, Guozhang Wang wrote: > >> Hi P

Re: [VOTE] KIP-401: TransformerSupplier/ProcessorSupplier StateStore connecting

2019-09-03 Thread Guozhang Wang
Hi Paul, Thanks for the KIP! +1 (binding). One minor comment about the following: "In order to solve the problem of addStateStore potentially being called twice for the same store (because more than one Supplier specifies it), the check for duplicate stores in addStateStores will be relaxed to

<    4   5   6   7   8   9   10   11   12   13   >