Re: [VOTE] KIP-249: Add Delegation Token Operations to Kafka Admin Client

Hi All, The vote has passed with 3 binding votes (Jun, Gwen, Rajini) and 2 non-binding votes (Satish, Viktor). Thanks everyone for the votes. Thanks, Manikumar On Tue, Apr 3, 2018 at 2:46 PM, Rajini Sivaram wrote: > +1 (binding) > > Thanks for the KIP, Manikumar! > >

Jenkins build is back to normal : kafka-trunk-jdk9 #529

See

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

Sounds great! The cryptic topic names can be an issue -- however, people can `describe()` their topology to map the name to the corresponding sub-topology/tasks to narrow the error down to the corresponding operators. I think, this should be "sufficient for now" for debugging. Renaming those

Build failed in Jenkins: kafka-trunk-jdk8 #2523

See Changes: [jason] KAFKA-6739; Ignore headers when down-converting from V2 to V0/V1 (#4813) -- [...truncated 422.24 KB...] kafka.controller.ReplicaStateMachineTest >

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

Thanks John, your proposal looks fine to me. I'll go ahead and look into the PR for more details myself. Guozhang On Tue, Apr 3, 2018 at 1:35 PM, Bill Bejeck wrote: > Hi John, > > Thanks for making the updates. > > I agree with the information you've included in the logs

Re: Gradle strategy for exposing and using public test-utils modules

Hello again all, It turns out that the implementation I provided was not correct after all. The issue was gradle tracked the compiled source files included via > compile project(':streams').sourceSets.main.output and included them in the release tarball. Ewen found the issue and corrected the

Build failed in Jenkins: kafka-trunk-jdk8 #2522

See Changes: [me] MINOR: Make kafka-streams-test-utils dependencies work with releases [me] KAFKA-6728: Corrected the worker’s instantiation of the HeaderConverter [wangguoz] MINOR: Refactor return value

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Boyang, Thanks for making changes to the KIP, I'm +1 on the updated version. -Bill On Tue, Apr 3, 2018 at 1:14 AM, Boyang Chen wrote: > Hey friends, > > > both KIP

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

Hi John, Thanks for making the updates. I agree with the information you've included in the logs as described above, as log statements without enough context/information can be frustrating. -Bill On Tue, Apr 3, 2018 at 3:29 PM, John Roesler wrote: > Allrighty, how about

[jira] [Created] (KAFKA-6745) kafka consumer rebalancing takes long time (from 3 secs to 5 minutes)

Ramkumar created KAFKA-6745: --- Summary: kafka consumer rebalancing takes long time (from 3 secs to 5 minutes) Key: KAFKA-6745 URL: https://issues.apache.org/jira/browse/KAFKA-6745 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk9 #528

See Changes: [jason] KAFKA-6739; Ignore headers when down-converting from V2 to V0/V1 (#4813) -- [...truncated 1.48 MB...] kafka.server.epoch.LeaderEpochFileCacheTest >

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

Allrighty, how about this, then... I'll move the metric back to the StreamThread and maintain the existing tag (client-id=...(per-thread client-id)). It won't be present in the TopologyTestDriver's metrics. As a side note, I'm not sure that the location of the log messages has visibility into

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

Hi Dong, Thanks for the response. I think I'm broadly in agreement with your statement that with KIP-253 implemented as written, KStreams would be able to use partition expansion on its various internal topics and processor threads. As you say, the reason we can do this is that for internal

Build failed in Jenkins: kafka-1.0-jdk7 #183

See Changes: [jason] KAFKA-6739; Ignore headers when down-converting from V2 to V0/V1 (#4813) -- [...truncated 373.00 KB...] kafka.server.epoch.LeaderEpochFileCacheTest >

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

1. If we can indeed gather all the context information from the log4j entries I'd suggest we change to thread-level (I'm not sure if that is doable, so if John have already some WIP PR that can help us decide). 2. We can consider adding the API in TopologyTestDriver for general testing purposes;

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

I am fine adding `TopologyTestDriver#metrics()`. It should be helpful to test custom metrics users can implement. -Matthias On 4/3/18 11:35 AM, John Roesler wrote: > Oh, sorry, I missed the point. > > Yeah, we can totally do that. The reason to move it to the task level was > mainly to make it

Build failed in Jenkins: kafka-trunk-jdk9 #527

See Changes: [me] MINOR: Make kafka-streams-test-utils dependencies work with releases [me] KAFKA-6728: Corrected the worker’s instantiation of the HeaderConverter [wangguoz] MINOR: Refactor return value

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

Thanks Guozhang, that was my intent. @John: yes, we should not nail down the exact log message. It's just to point out the trade-off. If we can get the required information in the logs, we might not need task level metrics. -Matthias On 4/3/18 11:26 AM, Guozhang Wang wrote: > I think Matthias'

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

Oh, sorry, I missed the point. Yeah, we can totally do that. The reason to move it to the task level was mainly to make it available for the metrics in TopologyTestDriver as well. But if we decide that's a non-goal, then there's no motivation to change it. And actually that reminds me that we do

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

I think Matthias' comment is that, we can still record the metrics on the thread-level, while having the WARN log entry to include sufficient context information so that users can still easily narrow down the investigation scope. Guozhang On Tue, Apr 3, 2018 at 11:22 AM, John Roesler

Re: [DISCUSS] KIP-274: Kafka Streams Skipped Records Metrics

I agree we should add as much information as is reasonable to the log. For example, see this WIP PR I started for this KIP: https://github.com/apache/kafka/pull/4812/files#diff-88d129f048bc842c7db5b2566a45fce8R80 and

[jira] [Created] (KAFKA-6744) MockProducer with transaction enabled doesn't fail on commit if a record was failed

Pascal Gélinas created KAFKA-6744: - Summary: MockProducer with transaction enabled doesn't fail on commit if a record was failed Key: KAFKA-6744 URL: https://issues.apache.org/jira/browse/KAFKA-6744

[jira] [Resolved] (KAFKA-6739) Broker receives error when handling request with java.lang.IllegalArgumentException: Magic v0 does not support record headers

[ https://issues.apache.org/jira/browse/KAFKA-6739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6739. Resolution: Fixed Fix Version/s: 1.0.2 > Broker receives error when handling request

Re: [DISCUSS] KIP-275 - Indicate "isClosing" in the SinkTaskContext

Hey Randall, Devil's advocate sparring is always a fun game so I'm down. =) Rebalance caused by connectivity failure is the case that caused us to notice the issue. But the issue is really more about giving connectors the tools to facilitate rebalances or a Kafka connect shutdown cleanly.

Re: Kafka Authorizer interface review

Hi Mickael, Thanks for detailed description on these authorization issues. I agree they need to be reviewed and fixed in the areas you specified, or even at a higher that simplifies their maintenance as the matrix is expanded or needs to be modified. FYI, KIP-231 also attempts at addressing the

Re: Kafka Authorizer interface review

bq. you too are concerned about the current delete record/topic limitation Yes. I think this is a security hole. On Tue, Apr 3, 2018 at 9:37 AM, Mickael Maison wrote: > Yes this is indeed a typo! > > And yes we're considering filing another KIP but I thought

Build failed in Jenkins: kafka-trunk-jdk8 #2521

See Changes: [rajinisivaram] MINOR: Add Timed wait to -- [...truncated 422.76 KB...] kafka.zookeeper.ZooKeeperClientTest > testGetChildrenExistingZNodeWithChildren

Re: Kafka Authorizer interface review

Yes this is indeed a typo! And yes we're considering filing another KIP but I thought collecting all our feedback and providing a full summary might be beneficial for others. I see you too are concerned about the current delete record/topic limitation. On Tue, Apr 3, 2018 at 5:26 PM, Ted Yu

Re: [DISCUSS] KIP-275 - Indicate "isClosing" in the SinkTaskContext

Matt, Let me play devil's advocate. Do we need this additional complexity? The motivation section talked about needing to deal with task failures due to connectivity problems. Generally speaking, isn't it possible that if one task has connectivity problems with either the broker or the external

Re: Kafka Authorizer interface review

bq. There is now way to distinguish between topic and record deletion. I guess you meant 'no way' above. I think deleting a topic has higher impact than deleting records. Have you considered filing KIP to distinguish the two operations ? Cheers On Tue, Apr 3, 2018 at 9:22 AM, Mickael Maison

Kafka Authorizer interface review

Hi all, Over the past few months the IBM Message Hub team has "played quite a bit" with the pluggable Authorizer interface and I'll try to give a summary of our findings. First when implementing a custom Authorizer, we found it hard having a global view of all the Resource/Operation required for

[jira] [Resolved] (KAFKA-6728) Kafka Connect Header Null Pointer Exception

[ https://issues.apache.org/jira/browse/KAFKA-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-6728. -- Resolution: Fixed Fix Version/s: 1.1.1 1.2.0 Issue

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

Hey John, Thanks much for your comments!! I have yet to go through the emails of John/Jun/Guozhang in detail. But let me present my idea for how to minimize the delay for state loading for stream use-case. For ease of understanding, let's assume that the initial partition number of input topics

[jira] [Created] (KAFKA-6743) ConsumerPerformance fails to consume all messages on topics with large number of partitions

Alex Dunayevsky created KAFKA-6743: -- Summary: ConsumerPerformance fails to consume all messages on topics with large number of partitions Key: KAFKA-6743 URL: https://issues.apache.org/jira/browse/KAFKA-6743

Re: [DISCUSS] KIP-275 - Indicate "isClosing" in the SinkTaskContext

I have made the requested updates to the KIP! :) On Mon, Apr 2, 2018 at 11:02 AM, Matt Farmer wrote: > Ugh > > * I can update > > I need more caffeine... > > On Mon, Apr 2, 2018 at 11:01 AM, Matt Farmer wrote: > >> I'm can update the rejected alternatives section as

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

Thanks for working on it Viktor. It looks good to me, but I have some questions: - I see a new type DOUBLE is used for quota_value , and it is not listed among the primitive types on the Kafka protocol guide. Can you add some more details? - I am not sure that using an environment (i.e.

[jira] [Created] (KAFKA-6742) TopologyTestDriver error when dealing with stores from GlobalKTable

Valentino Proietti created KAFKA-6742: - Summary: TopologyTestDriver error when dealing with stores from GlobalKTable Key: KAFKA-6742 URL: https://issues.apache.org/jira/browse/KAFKA-6742 Project:

[jira] [Created] (KAFKA-6741) Transient test failure: SslTransportLayerTest.testNetworkThreadTimeRecorded

Manikumar created KAFKA-6741: Summary: Transient test failure: SslTransportLayerTest.testNetworkThreadTimeRecorded Key: KAFKA-6741 URL: https://issues.apache.org/jira/browse/KAFKA-6741 Project: Kafka

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

+1 (binding) Thanks for the KIP, Viktor! On Tue, Apr 3, 2018 at 8:25 AM, Peter Toth wrote: > +1 (non binding) > > Thanks Viktor. > Please remove "Unknown" AlterOperation type from Wire Format Types section > as you did from the AdminClient APIs section. > > > On Wed, Mar

Re: [VOTE] KIP-249: Add Delegation Token Operations to Kafka Admin Client

+1 (binding) Thanks for the KIP, Manikumar! On Thu, Mar 29, 2018 at 5:34 PM, Gwen Shapira wrote: > +1 > > Thank you and sorry for missing it the first time around. > > On Thu, Mar 29, 2018 at 3:05 AM, Manikumar > wrote: > > > I'm bumping this up

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

+1 (non binding) Thanks Viktor. Please remove "Unknown" AlterOperation type from Wire Format Types section as you did from the AdminClient APIs section. On Wed, Mar 21, 2018 at 5:41 PM, Viktor Somogyi wrote: > Hi Everyone, > > I've started a vote on KIP-248 >

[jira] [Resolved] (KAFKA-6716) discardChannel should be released in MockSelector#completeSend

[ https://issues.apache.org/jira/browse/KAFKA-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6716. -- Resolution: Fixed Fix Version/s: 1.2.0 > discardChannel should be released in

[jira] [Resolved] (KAFKA-6630) Speed up the processing of TopicDeletionStopReplicaResponseReceived events on the controller

[ https://issues.apache.org/jira/browse/KAFKA-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6630. -- Resolution: Fixed Fix Version/s: 1.2.0 > Speed up the processing of

[jira] [Resolved] (KAFKA-6702) Wrong className in LoggerFactory.getLogger method

[ https://issues.apache.org/jira/browse/KAFKA-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6702. -- Resolution: Fixed > Wrong className in LoggerFactory.getLogger method >

[jira] [Resolved] (KAFKA-6707) The default value for config of Type.LONG should be *L

[ https://issues.apache.org/jira/browse/KAFKA-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6707. -- Resolution: Fixed > The default value for config of Type.LONG should be *L >