Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

Hi Dong, I took a look at KIP-287. Producers writing using the new scheme prior to processor catch up and cut-over makes sense. Thanks. On Sat, Apr 14, 2018 at 7:09 PM, Dong Lin wrote: > Hey Jeff, > > Thanks for the review. The scheme for expanding processors of the

Build failed in Jenkins: kafka-trunk-jdk10 #29

See Changes: [github] HOTFIX: use the new prop object (#4888) -- [...truncated 1.48 MB...] kafka.zk.KafkaZkClientTest > testIsrChangeNotificationsDeletion STARTED

[jira] [Resolved] (KAFKA-6281) Kafka JavaAPI Producer failed with NotLeaderForPartitionException

[ https://issues.apache.org/jira/browse/KAFKA-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6281. -- Resolution: Not A Problem Closing inactive issue. Please reopen if you think the issue still exists >

Build failed in Jenkins: kafka-trunk-jdk7 #3350

See Changes: [github] HOTFIX: use the new prop object (#4888) -- [...truncated 414.13 KB...] kafka.zookeeper.ZooKeeperClientTest > testPipelinedGetData PASSED

Jenkins build is back to normal : kafka-trunk-jdk10 #28

See

Jenkins build is back to normal : kafka-trunk-jdk8 #2560

See

Re: [VOTE] KIP-281: ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

+1 (binding) On Tue, Apr 17, 2018 at 9:04 AM, Moshe Blumberg wrote: > +1 > > > > From: Ted Yu > Sent: 16 April 2018 22:43 > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-281: ConsumerPerformance: Increase Polling

Re: [DISCUSS] KIP-283: Efficient Memory Usage for Down-Conversion

Hi Jun, Yes, that is true. Ideally, we should be able to down-convert only the first message batch in the request handling thread and delay everything else till the network thread. I have not thought through all the details of how we could do this but at first glance this seems tricky to

Re: [DISCUSS] KIP-275 - Indicate "isClosing" in the SinkTaskContext

Hello all, I've updated a KIP again to add a few sentences about the general problem we were facing in the motivation section. Please let me know if there is any further feedback. On Tue, Apr 3, 2018 at 1:46 PM, Matt Farmer wrote: > Hey Randall, > > Devil's advocate sparring is

Re: [VOTE] KIP-279: Fix log divergence between leader and follower after fast leader fail over

The vote for KIP-279 has passed with 5 binding and 2 non-binding +1s (and no objections). Thanks everyone for your reviews and feedback, Anna On Tue, Apr 17, 2018 at 1:49 PM, Anna Povzner wrote: > Guozhang, thanks for catching this, I fixed the description (the example >

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Hi John, Do you have a preference for fixing the poll() method (e.g. using asyncPoll or just sticking with the current method but with an extra timeout parameter) ? I think your current proposition for KIP-288 is better than what I have on my side. If you think there is something that you want to

Build failed in Jenkins: kafka-trunk-jdk10 #27

See Changes: [jason] KAFKA-6770; Add New Protocol Versions to 1.1.0 documentation (#4847) [wangguoz] KAFKA-6376: preliminary cleanup (#4872) -- [...truncated 1.53 MB...]

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Ok, I'll close the discussion on KIP-288 and mark it discarded. We can solidify the design for poll in KIP-266, and once it's approved, I'll coordinate with Qiang Zhao on the PR for the poll part of the work. Once that is merged, you'll have a clean slate for the rest of the work. On Tue, Apr

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Hi all, I'm closing this discussion and marking KIP-288 as discarded. Please pick up the discussion on the KIP-266 thread. Thanks, -John On Tue, Apr 17, 2018 at 2:07 PM, John Roesler wrote: > Ah, I see that this KIP is a sub-kip of KIP-266. Sorry about that; I've > sent a

Re: [VOTE] KIP-279: Fix log divergence between leader and follower after fast leader fail over

Guozhang, thanks for catching this, I fixed the description (the example assumed response with 21, '11' was a typo). On Tue, Apr 17, 2018 at 1:25 PM, Anna Povzner wrote: > Hi Colin, > > Yes, the impact of "losing" entries in the LeaderEpoch file is more > round-trips for

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Hi John, I think that you could finish your PR that corresponds with KIP-288 and merge it. I can finish my side of the work afterwards. On another note, adding an asynchronized version of poll() would make sense, particularily since the current version of Kafka does not support it. Thanks

[jira] [Resolved] (KAFKA-6770) 1.1.0 New Protocol Versions section in doc is empty

[ https://issues.apache.org/jira/browse/KAFKA-6770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6770. Resolution: Fixed > 1.1.0 New Protocol Versions section in doc is empty >

Re: [VOTE] KIP-279: Fix log divergence between leader and follower after fast leader fail over

Hi Colin, Yes, the impact of "losing" entries in the LeaderEpoch file is more round-trips for OffsetForLeaderEpoch. There is JIRA for the log cleaner: https://issues.apache.org/jira/browse/KAFKA-6780, and to investigate if there is actual possibility of losing committed records due to cleaning

Jenkins build is back to normal : kafka-trunk-jdk7 #3347

See

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Cross-pollinating from some discussion we've had on KIP-288, I think there's a good reason that poll() takes a timeout when none of the other methods do, and it's relevant to this discussion. The timeout in poll() is effectively implementing a long-poll API (on the client side, so it's not really

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Ah, I see that this KIP is a sub-kip of KIP-266. Sorry about that; I've sent a message to the discussion thread of KIP-266 to work this out. Thanks, -John On Tue, Apr 17, 2018 at 1:29 PM, John Roesler wrote: > Thanks, Richard, I'll check it out. > > On Tue, Apr 17, 2018 at

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Hey Richard, As you noticed, the newly introduced KIP-288 overlaps with this one. Sorry for stepping on your toes... How would you like to proceed? I'm happy to "close" KIP-288 in deference to this KIP. With respect to poll(), reading this discussion gave me a new idea for providing a

Re: [VOTE] KIP-279: Fix log divergence between leader and follower after fast leader fail over

Thanks, Anna, this looks great. >From the KIP: > Impact of topic compaction > > The proposed solution requires that we preserve history in > LeaderEpochSequence file. Note that this is also required in the current > implementation if we want to guarantee no log divergence. The only

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Hi Ted, Thanks for the review. During phase #2, the consumer continuously polls until it either gets some responses or spends the whole timeout. If it reaches the timeout, it doesn't mean that the operation "timed out" in an exceptional way, it just means we determined that there are no records

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Thanks, Richard, I'll check it out. On Tue, Apr 17, 2018 at 1:19 PM, Richard Yu wrote: > Hi John, > > bq. #1 (wait for metadata) is infinite. > > Some of what you stated in this KIP has already been previously discussed > in a older KIP. (KIP-266) > Just for your

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

bq. This method will continue to return an empty collection if it doesn't complete phase #2 within the timeout. If phase #1 times out, poll will throw a TimeoutException. It seems response from the new poll method depends on which phase it is in prior to the timeout. Wouldn't it be cleaner if

Build failed in Jenkins: kafka-trunk-jdk8 #2559

See Changes: [wangguoz] KAFKA-6742: TopologyTestDriver error when dealing with stores from -- [...truncated 416.30 KB...] kafka.admin.ResetConsumerGroupOffsetTest >

Re: [DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Hi John, bq. #1 (wait for metadata) is infinite. Some of what you stated in this KIP has already been previously discussed in a older KIP. (KIP-266) Just for your reference. Thanks, Richard On Tue, Apr 17, 2018 at 11:08 AM, John Roesler wrote: > Hello all, > > I am

Build failed in Jenkins: kafka-trunk-jdk10 #26

See Changes: [wangguoz] KAFKA-6742: TopologyTestDriver error when dealing with stores from -- [...truncated 1.48 MB...] kafka.server.epoch.LeaderEpochFileCacheTest >

[DISCUSS] KIP-288: Consumer poll timeout change and new waitForAssignment method

Hello all, I am proposing KIP-288 to improve the semantics of Consumer.poll() w.r.t. the timeout parameter and to add a new method for blocking for assignment only. Please find the details here:

Re: [DISCUSS] KIP-277 - Fine Grained ACL for CreateTopics API

Hi Edo, Thanks for addressing that concern in the KIP. And I agree that in the long run the create cluster permission should be deprecated. --Vahid From: Edoardo Comar To: dev@kafka.apache.org Date: 04/17/2018 03:52 AM Subject:Re: [DISCUSS] KIP-277 - Fine

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Hi all, If possible, would a committer please review? Thanks On Sun, Apr 1, 2018 at 7:24 PM, Richard Yu wrote: > Hi Guozhang, > > I have clarified the KIP a bit to account for Becket's suggestion on > ClientTimeoutException. > About adding an extra config, you were

Re: [DISCUSS] KIP-277 - Fine Grained ACL for CreateTopics API

Thanks for the KIP. The updated wiki LGTM to me now. I'd also want to have wildcard support for create / delete topic ACLs for Streams usage :) But I'd agree the feature itself may deserve a separate KIP. Guozhang On Tue, Apr 17, 2018 at 4:40 AM, Edoardo Comar wrote: > We

Re: [VOTE] KIP-281: ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

+1 From: Ted Yu Sent: 16 April 2018 22:43 To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-281: ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User +1 On Mon, Apr 16, 2018 at 2:25 PM, Alex Dunayevsky

Jenkins build is back to normal : kafka-trunk-jdk10 #25

See

Build failed in Jenkins: kafka-trunk-jdk7 #3346

See Changes: [github] KAFKA-6765: Handle exception while reading throttle metric value in test -- [...truncated 412.20 KB...] kafka.admin.ResetConsumerGroupOffsetTest >

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

On Mon, Apr 16, 2018, at 08:15, Ron Dagostino wrote: > Hi Rajini. I think a good, illustrative OAuth example is the situation > where a Kafka client (whether non-broker or broker, where the latter occurs > when OAUTHBEARER is the inter-broker SASL mechanism) needs to authenticate > to the token

Build failed in Jenkins: kafka-trunk-jdk8 #2558

See Changes: [github] KAFKA-6765: Handle exception while reading throttle metric value in test -- [...truncated 414.82 KB...] kafka.server.epoch.LeaderEpochFileCacheTest

[jira] [Resolved] (KAFKA-6765) Intermittent test failure in CustomQuotaCallbackTest

[ https://issues.apache.org/jira/browse/KAFKA-6765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-6765. --- Resolution: Fixed Reviewer: Jun Rao > Intermittent test failure in

[jira] [Created] (KAFKA-6799) Consumer livelock during consumer group rebalance

Pierre-Henri Dezanneau created KAFKA-6799: - Summary: Consumer livelock during consumer group rebalance Key: KAFKA-6799 URL: https://issues.apache.org/jira/browse/KAFKA-6799 Project: Kafka

Re: RE: [DISCUSS] KIP-280: Enhanced log compaction

Oops, missed that email... bq. It is because when we compare the bytes we do not treat them as longs atall, so we just compare them based on bytes; I admit that if users's headertypes have some semantic meanings (e.g. it is encoded from a long) they weare forcing them to choose the encoder

[jira] [Created] (KAFKA-6798) Kafka leader rebalance failures

Riley Zimmerman created KAFKA-6798: -- Summary: Kafka leader rebalance failures Key: KAFKA-6798 URL: https://issues.apache.org/jira/browse/KAFKA-6798 Project: Kafka Issue Type: Bug

Re: RE: [DISCUSS] KIP-280: Enhanced log compaction

Can you respond to: http://search-hadoop.com/m/Kafka/uyzND1OlYaSzZ3SM1?subj=Re+RE+DISCUSS+KIP+280+Enhanced+log+compaction Original message From: Luís Cabral Date: 4/17/18 2:41 AM (GMT-08:00) To: dev@kafka.apache.org Subject: Re: RE: [DISCUSS]

Re: [DISCUSS] KIP-277 - Fine Grained ACL for CreateTopics API

We updated the KIP addressing the backward compatibilty issue, proposing to change the current ACL check for creating a topic T, from CREATE on Cluster to CREATE on Cluster OR CREATE on Topic(T). leaving the enhancement of wildcard support to a future KIP. please let us know of any

Re: [DISCUSS] KIP-277 - Fine Grained ACL for CreateTopics API

Thanks Colin, > I think the reason why CreateTopics requires CREATE CLUSTER is that creating topics was considered something that only an administrator would want to do. I think this may no longer be the case ... Streams/KSQL application create intermediate topics where only the start of the

Re: [DISCUSS] KIP-277 - Fine Grained ACL for CreateTopics API

Thanks Vahid, as described in the rejected section, we wanted to get feedback on the point : > An alternative that we want to discuss with the community is to favour compatibility rather than simplicity, > and consider existing "Create Cluster" permission as equivalent to "Create Any Topics",

Re: [DISCUSS] KIP-257 - Configurable Quota Management

Hi Ismael, Sorry, I had missed your note earlier. Jun had suggested changing the quota callback methods so that there is only one method to obtain quota limits. We used to have two methods, one which returned (*quotaLimit+metricTags)* and another which returned *quotaLimit* given the

Re: RE: [DISCUSS] KIP-280: Enhanced log compaction

Hi all, There aren't that many discussions on this KIP, does that mean it should now move to voting? I'm not sure on the process here... Cheers

[jira] [Created] (KAFKA-6797) Connect oracle database to kafka and stream data

Ujjwal Kumar created KAFKA-6797: --- Summary: Connect oracle database to kafka and stream data Key: KAFKA-6797 URL: https://issues.apache.org/jira/browse/KAFKA-6797 Project: Kafka Issue Type: