[jira] [Created] (KAFKA-6689) Kafka not release .deleted file.

A created KAFKA-6689: Summary: Kafka not release .deleted file. Key: KAFKA-6689 URL: https://issues.apache.org/jira/browse/KAFKA-6689 Project: Kafka Issue Type: Bug Components: config,

Jenkins build is back to normal : kafka-trunk-jdk9 #495

See

[jira] [Created] (KAFKA-6688) The Trogdor coordinator should track task statuses

Colin P. McCabe created KAFKA-6688: -- Summary: The Trogdor coordinator should track task statuses Key: KAFKA-6688 URL: https://issues.apache.org/jira/browse/KAFKA-6688 Project: Kafka Issue

Re: [DISCUSS] KIP-242: Mask password fields in Kafka Connect REST response

SSL authentication was added in KIP-208, which will be included in Kafka 1.1.0: https://cwiki.apache.org/confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface Connect isn't much different from the core Kafka/client configs currently where in some security setups you

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

Clarificarion, it's a by name parameter so nothing happens unless the first parameter evaluates to false. Ismael On Mon, 19 Mar 2018, 19:21 Jay Kreps, wrote: > Optimizing startup seems really valuable but I'm a little confused by this. > > There are two different things: >

Jenkins build is back to normal : kafka-trunk-jdk7 #3274

See

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

Optimizing startup seems really valuable but I'm a little confused by this. There are two different things: 1. Recovery 2. Sanity check The terminology we're using is a bit mixed here. Recovery means checksumming the log segments and rebuilding the index on a hard crash. This only happens on

Re: Seeking Feedback on Kafka Connect Issues

Responses inline. On Mon, Mar 19, 2018 at 3:02 PM, Matt Farmer wrote: > Hi everyone, > > We’ve been experimenting recently with some limited use of Kafka Connect > and are hoping to expand to wider use cases soon. However, we had some > internal issues that gave us a well-timed

Build failed in Jenkins: kafka-trunk-jdk8 #2492

See Changes: [jason] MINOR: KafkaFutureImpl#addWaiter should be protected (#4734) [github] MINOR: Pass a streams config to replace the single state dir (#4714) [rajinisivaram] KAFKA-6680: Fix issues related

Build failed in Jenkins: kafka-1.1-jdk7 #92

See Changes: [jason] MINOR: KafkaFutureImpl#addWaiter should be protected (#4734) [rajinisivaram] KAFKA-6680: Fix issues related to Dynamic Broker configs (#4731) --

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Hi Ismael, You have a great point. Since most of the methods in this KIP have similar callbacks (position() and committed() both use fetchCommittedOffsets(), and commitSync() is similar to position(), except just updating offsets), the amount of time they block should be also about equal.

Re: [DISCUSS] KIP-250 Add Support for Quorum-based Producer Acknowledgment

Hi, Jason, Thanks for the comment. I didn't realize that the KIP was only focusing on the latency of the producer. From an application's perspective, if an application is latency sensitive, it probably wants to optimize the end-to-end latency from the producer to the consumer. Litao, would you

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Hi, An option that is not currently covered in the KIP is to have a separate config max.block.ms, which is similar to the producer config with the same name. This came up during the KAFKA-2391 discussion. I think it's clear that we can't rely on request.timeout.ms, so the decision is between

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

Guozhang, thanks for your comments. 2: I think my main concern is, that 1.2 would be "special" release that everybody need to use to upgrade. As an alternative, we could say that we add the config in 1.2 and keep it for 2 additional releases (1.3 and 1.4) but remove it in 1.5. This gives users

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Hi Guozhang, I made some clarifications to KIP-266, namely: 1. Stated more specifically that commitSync will accept user input. 2. fetchCommittedOffsets(): Made its role in blocking more clear to the reader. 3. Sketched what would happen when time limit is exceeded. These changes should make the

Re: Subject: [VOTE] 1.1.0 RC3

Vahid, The Java 9 Connect issue is similar to the one being fixed for Trogdor in the following PR: https://github.com/apache/kafka/pull/4725 We need to do something similar for Connect. Ismael On Fri, Mar 16, 2018 at 3:10 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Damian,

[jira] [Resolved] (KAFKA-6661) Sink connectors that explicitly 'resume' topic partitions can resume a paused task

[ https://issues.apache.org/jira/browse/KAFKA-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch resolved KAFKA-6661. -- Resolution: Fixed > Sink connectors that explicitly 'resume' topic partitions can resume a

Re: [DISCUSS] KIP-272: Add API version tag to broker's RequestsPerSec metric

bq. *additional hash lookup is needed when updating the metric to locate the metric * *Do you have estimate how much memory is needed for maintaining the hash map ?* *Thanks* On Mon, Mar 19, 2018 at 3:19 PM, Allen Wang wrote: > Hi all, > > I have created KIP-272: Add API

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

Matthias, You have to choose between the package and class names when it comes to adding a suffix (or prefix). I think the package name is the lesser of two evils, but it would be interesting to know what others think. I agree that we should include the information in the KIP (whatever we

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

About the package name. Would it be better/cleaner to omit the `scala` sub-package? Or is this required to avoid naming conflicts with the Java classes? If yes, please point it out in the KIP. -Matthias On 3/19/18 11:24 AM, Debasish Ghosh wrote: > Cool .. so the changes that u suggested e.g

Build failed in Jenkins: kafka-trunk-jdk7 #3273

See Changes: [rajinisivaram] MINOR: Fix some compiler warnings (#4726) -- [...truncated 1.54 MB...] at

[DISCUSS] KIP-272: Add API version tag to broker's RequestsPerSec metric

Hi all, I have created KIP-272: Add API version tag to broker's RequestsPerSec metric. Here is the link to the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-272%3A+Add+API+version+tag+to+broker%27s+RequestsPerSec+metric Looking forward to the discussion. Thanks, Allen

Seeking Feedback on Kafka Connect Issues

Hi everyone, We’ve been experimenting recently with some limited use of Kafka Connect and are hoping to expand to wider use cases soon. However, we had some internal issues that gave us a well-timed preview of error handling behavior in Kafka Connect. I think the fixes for this will require at

Re: [DISCUSS] KIP-242: Mask password fields in Kafka Connect REST response

What’s the status of this? This is a pretty hard blocker for us to meet requirements internally to deploy connect in a distributed fashion. @Ewen - Regarding the concern of accessing information securely - has there been any consideration of adding authentication to the connect api? > On Jan

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

Cool .. so the changes that u suggested e.g the WordCount example, package name changes etc. will be in the PR only and not in the KIP document .. Or we keep the KIP document updated as well ? regards. On Mon, 19 Mar 2018 at 11:46 PM, Guozhang Wang wrote: > Hi Debasish, >

Jenkins build is back to normal : kafka-trunk-jdk7 #3272

See

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

Hi Debasish, You can work on the PR in parallel to the KIP discussion so that people can start reviewing it. The only restriction is that the PR cannot be merged until the KIP is accepted. Guozhang On Mon, Mar 19, 2018 at 11:09 AM, Debasish Ghosh < debasish.gh...@lightbend.com> wrote: >

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

Based on this discussion I have a question .. For the actual implementation we need to have a PR on https://github.com/apache/kafka and the new library will be in the package structure org.apache.kafka.streams.scala. My question is, is this PR part of the KIP ? Or we work on the PR once the KIP

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

Thanks Guozhang for the comments .. we will start working on them .. regards. On Mon, Mar 19, 2018 at 11:02 PM, Guozhang Wang wrote: > And one more comment about the type safety: > > 7. I'm in favor of the approach of enforcing type safety at compile time, > since with

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

Yup :) On Mon, Mar 19, 2018 at 10:01 AM, Ted Yu wrote: > bq. some snippet like ProduceRequest / ProduceRequest > > Did you mean ProduceRequest / Response ? > > Cheers > > On Mon, Mar 19, 2018 at 9:51 AM, Guozhang Wang wrote: > > > Hi Matthias, > > > >

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

And one more comment about the type safety: 7. I'm in favor of the approach of enforcing type safety at compile time, since with Scala's strong typing system I think it makes more sense to get rid of config-based serde specifications that can cause runtime error in the Scala wrapper. Guozhang

Build failed in Jenkins: kafka-trunk-jdk8 #2490

See Changes: [rajinisivaram] KAFKA-6676: Ensure Kafka chroot exists in system tests and use chroot on [rajinisivaram] MINOR: Some logging improvements for debugging delayed produce status

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

Hello Debasish, Thanks for the KIP. Here are some comments: 1. About the naming: I'd also vote for option 2 and enforce users to rename when import since this scenario seems rare to me. 2. About the dependency: since this KIP can only be merged as early as 1.2, it means that for users who wants

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Hi Jason, Thanks for your feedback and suggestion. I updated the name "empty_state_timestamp" to "current_state_timestamp" to expand its usage to all state changes. If you can think of a better name let me know. And, I fixed the statement on Dead group state to include the coordinator change

Jenkins build is back to normal : kafka-1.0-jdk7 #174

See

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

bq. some snippet like ProduceRequest / ProduceRequest Did you mean ProduceRequest / Response ? Cheers On Mon, Mar 19, 2018 at 9:51 AM, Guozhang Wang wrote: > Hi Matthias, > > About 2: yeah I guess this is a subjective preference. My main concern > about keeping the config

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

Hi Matthias, About 2: yeah I guess this is a subjective preference. My main concern about keeping the config / handling code beyond 1.2 release is that it will become a non-cleanable tech debt forever, as fewer and fewer users would need to upgrade from 0.10.x and 1.1.x, and eventually we will

Support kafka

Dear, Please your valuable support as I need to configure on a server (Docker) Server1: confluentinc / cp-schema-registry confluentinc / cp-kafka-rest landoop / kafka-topics-ui For a named kafka cluster (3 independent nodes): zookeeper1: 2181, zookeeper2: 2181, zookeeper3: 2181

Re: [DISCUSSION] KIP-266: Add TimeoutException to KafkaConsumer#position()

Hi Richard, I made a pass over the KIP again, some more clarifications / comments: 1. seek() call itself is not blocking, only the following poll() call may be blocking as the actually metadata rq will happen. 2. I saw you did not include Consumer.partitionFor(), Consumer.OffsetAndTimestamp()

[jira] [Created] (KAFKA-6687) Allow to read a topic multiple times

Matthias J. Sax created KAFKA-6687: -- Summary: Allow to read a topic multiple times Key: KAFKA-6687 URL: https://issues.apache.org/jira/browse/KAFKA-6687 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-6686) Allow reading topic as KStream and GlobalKTable

Matthias J. Sax created KAFKA-6686: -- Summary: Allow reading topic as KStream and GlobalKTable Key: KAFKA-6686 URL: https://issues.apache.org/jira/browse/KAFKA-6686 Project: Kafka Issue

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

I agree with Sean on name unification. +1 to option 2. On Mon, Mar 19, 2018 at 7:23 AM, Sean Glover wrote: > Type names: I vote for option 2. The user must explicitly add a dependency > to this library and the wrapper types are in a different package. It seems >

[jira] [Created] (KAFKA-6685) Connect deserialization log message should distinguish key from value

Yeva Byzek created KAFKA-6685: - Summary: Connect deserialization log message should distinguish key from value Key: KAFKA-6685 URL: https://issues.apache.org/jira/browse/KAFKA-6685 Project: Kafka

[jira] [Created] (KAFKA-6684) Support casting values with bytes schema to string

Amit Sela created KAFKA-6684: Summary: Support casting values with bytes schema to string Key: KAFKA-6684 URL: https://issues.apache.org/jira/browse/KAFKA-6684 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-6683) ReplicaFetcher crashes with "Attempted to complete a transaction which was not started"

Chema Sanchez created KAFKA-6683: Summary: ReplicaFetcher crashes with "Attempted to complete a transaction which was not started" Key: KAFKA-6683 URL: https://issues.apache.org/jira/browse/KAFKA-6683

Re: [DISCUSS] KIP-270 A Scala wrapper library for Kafka Streams

Type names: I vote for option 2. The user must explicitly add a dependency to this library and the wrapper types are in a different package. It seems reasonable to expect them to do an import rename if there's a need to drop down to the Java API. Test Utils: The test utils in

Build failed in Jenkins: kafka-1.1-jdk7 #90

See Changes: [rajinisivaram] KAFKA-6676: Ensure Kafka chroot exists in system tests and use chroot on -- [...truncated 1.68 MB...]

Build failed in Jenkins: kafka-trunk-jdk7 #3271

See Changes: [rajinisivaram] KAFKA-6676: Ensure Kafka chroot exists in system tests and use chroot on -- [...truncated 1.53 MB...]

[jira] [Created] (KAFKA-6682) Kafka reconnection after broker restart

Tomasz Gąska created KAFKA-6682: --- Summary: Kafka reconnection after broker restart Key: KAFKA-6682 URL: https://issues.apache.org/jira/browse/KAFKA-6682 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Last week I was able to spend a bit of time working on KIP-236 again and, based on the discussion about that with Jun back in December, I refactored the controller to store the reassignment state in /brokers/topics/${topic} instead of introducing new ZK nodes. This morning I was wondering what to

Re: [DISCUSS] KIP-248 Create New ConfigCommand That Uses The New AdminClient

Hi, Since there were no additional on this KIP, I'd like to restart the vote tomorrow. If anyone has comments, please do address them. Thanks, Viktor On Mon, Feb 19, 2018 at 9:44 AM, Viktor Somogyi wrote: > Hi Rajini, > > Thanks for the feedback, I've applied your

[jira] [Created] (KAFKA-6681) Two instances of kafka consumer reading the same partition within a consumer group

Narayan Periwal created KAFKA-6681: -- Summary: Two instances of kafka consumer reading the same partition within a consumer group Key: KAFKA-6681 URL: https://issues.apache.org/jira/browse/KAFKA-6681

[jira] [Created] (KAFKA-6680) Fix config initialization in DynamicBrokerConfig

Manikumar created KAFKA-6680: Summary: Fix config initialization in DynamicBrokerConfig Key: KAFKA-6680 URL: https://issues.apache.org/jira/browse/KAFKA-6680 Project: Kafka Issue Type: Bug

Build failed in Jenkins: kafka-trunk-jdk8 #2489

See Changes: [mjsax] KAFKA-6486: Implemented LinkedHashMap in TimeWindows (#4628) -- [...truncated 3.55 MB...] org.apache.kafka.streams.StreamsConfigTest >

Jenkins build is back to normal : kafka-trunk-jdk9 #491

See

Re: [DISCUSS] KIP-233: Simplify StreamsBuilder#addGlobalStore

Hi Matthias, I am not sure weather I understand the impact correctly. With the old api user have to specify name for state store and the name is used for auto generated topics for fault tolerance, so when replace the old api usage with the new one we don’t have any kind of auto renaming so the

Build failed in Jenkins: kafka-trunk-jdk7 #3270

See Changes: [mjsax] KAFKA-6486: Implemented LinkedHashMap in TimeWindows (#4628) -- [...truncated 1.53 MB...]