Jenkins build is back to normal : kafka-2.1-jdk8 #215

See

Build failed in Jenkins: kafka-2.3-jdk8 #64

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H34 (ubuntu xenial) in workspace

Re: [VOTE] KIP-484: Expose metrics for group and transaction metadata loading duration

+1 (binding). Thanks for the KIP, Anastasia. best, Colin On Fri, Jul 12, 2019, at 10:09, Jason Gustafson wrote: > +1 Thanks for the KIP! > > -Jason > > On Fri, Jul 12, 2019 at 9:59 AM Gwen Shapira wrote: > > > +1 (binding) > > > > Looks great to me. Thank you for the KIP. > > > > On Mon,

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

On Sat, Jul 13, 2019, at 17:54, George Li wrote: > I just take a look at the updated KIP-455 again. I noticed > this "targetReplicas" is removed and instead, put "addingReplicas" & > "removingReplicas".  So where does the new reassigned "targetReplicas" > is stored? When all replicas in ISR,

Build failed in Jenkins: kafka-2.3-jdk8 #63

See -- Started by an SCM change Started by an SCM change Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H34 (ubuntu xenial) in workspace

Build failed in Jenkins: kafka-2.1-jdk8 #214

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H34 (ubuntu xenial) in workspace

Jenkins build is back to normal : kafka-2.3-jdk8 #62

See

[jira] [Created] (KAFKA-8672) RebalanceSourceConnectorsIntegrationTest#testReconfigConnector

Matthias J. Sax created KAFKA-8672: -- Summary: RebalanceSourceConnectorsIntegrationTest#testReconfigConnector Key: KAFKA-8672 URL: https://issues.apache.org/jira/browse/KAFKA-8672 Project: Kafka

Build failed in Jenkins: kafka-2.1-jdk8 #213

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H34 (ubuntu xenial) in workspace

Build failed in Jenkins: kafka-trunk-jdk11 #696

See Changes: [bbejeck] KAFKA-8602: Fix bug in stand-by task creation (#7008) -- [...truncated 2.56 MB...]

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

Hi Boyang, Thanks for the updates. A few comments below: 1. The KIP mentions that `transaction.timeout.ms` should be reduced to 10s. I think this makes sense for Kafka Streams which is tied to the consumer group semantics and uses a default 10s session timeout. However, it seems a bit dangerous

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

On Mon, Jul 15, 2019, at 11:51, Jason Gustafson wrote: > Hi Colin, > > A few more questions below: > > 1. The KIP says that the --zookeeper option will be removed from > kafka-reassign.sh. Do you mean that it will be deprecated and eventually > removed? Keeping the --zookeeper flag here would

Build failed in Jenkins: kafka-2.1-jdk8 #212

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H34 (ubuntu xenial) in workspace

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

On Mon, Jul 15, 2019, at 14:31, Stanislav Kozlovski wrote: > Hey George, > > > Different replica threads for throttling > The reason we can't support throttling for reassigning partitions right now > is because we have no good way of telling whether a replica is part of an > ongoing reassignment

[jira] [Created] (KAFKA-8671) NullPointerException occurs if topic associated with GlobalKTable changes

Alex Leung created KAFKA-8671: - Summary: NullPointerException occurs if topic associated with GlobalKTable changes Key: KAFKA-8671 URL: https://issues.apache.org/jira/browse/KAFKA-8671 Project: Kafka

[jira] [Resolved] (KAFKA-8637) WriteBatch objects leak off-heap memory

[ https://issues.apache.org/jira/browse/KAFKA-8637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sophie Blee-Goldman resolved KAFKA-8637. Resolution: Fixed > WriteBatch objects leak off-heap memory >

Build failed in Jenkins: kafka-2.1-jdk8 #211

See -- Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on H34 (ubuntu xenial) in workspace

Re: [DISCUSS] KIP-478 Strongly Typed Processor API

Hey all, It sounds like there's general agreement now on this KIP, so I updated the KIP to fit in with Guozhang's overall proposed package structure. Specifically, the proposed name for the new Processor interface is "org.apache.kafka.streams.processor.api.Processor". If there are no objections,

Bay area Kafka Meetup hosted by Confluent at Palo Alto, Tuesday 6:30pm, July 16, 2019

Hello folks, This is just a kind reminder of the Bay Area Kafka® meetup tomorrow, Tuesday 6:30pm, at Confluent's Palo Alto HQ. *RSVP and Register* (if you intend to attend in person): https://www.meetup.com/KafkaBayArea/events/262715611/ please read instructions within the link to register for

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

Hey George, > Different replica threads for throttling The reason we can't support throttling for reassigning partitions right now is because we have no good way of telling whether a replica is part of an ongoing reassignment or not. KIP-455 now gives us this knowledge with the `addingReplicas`

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

On Sat, Jul 13, 2019, at 17:54, George Li wrote: > Hi Stanislav, > > sorry for the late reply. comments below:  > > > Thanks for the reminder. A lot of your suggestions are outlined in the > > "Future Work" section of KIP-455. The pointer towards different > > ReplicaFetcher thread pools is

[jira] [Resolved] (KAFKA-8602) StreamThread Dies Because Restore Consumer is not Subscribed to Any Topic

[ https://issues.apache.org/jira/browse/KAFKA-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck resolved KAFKA-8602. Resolution: Fixed > StreamThread Dies Because Restore Consumer is not Subscribed to Any Topic >

Re: [DISCUSS] KIP-455 Create an Admin API for Replica Reassignments

Hi Colin, A few more questions below: 1. The KIP says that the --zookeeper option will be removed from kafka-reassign.sh. Do you mean that it will be deprecated and eventually removed? 2. The KIP mentions that AR is shrunk iteratively. Does this have a benefit? The downside is that we lose

[DISCUSS] KIP-492 Add java security providers in Kafka Security config

Hello all, I'd like to start a discussion thread for KIP-492. This KIP plans on introducing a new security config parameter for a custom security providers. Please take a look and let me know what do you think. More information can be found here:

[jira] [Created] (KAFKA-8670) kafka-topics.sh shows IllegalArgumentException when describing all topics if no topics exist

Tirtha Chatterjee created KAFKA-8670: Summary: kafka-topics.sh shows IllegalArgumentException when describing all topics if no topics exist Key: KAFKA-8670 URL:

[jira] [Created] (KAFKA-8669) Add java security providers in Kafka Security config

Sai Sandeep created KAFKA-8669: -- Summary: Add java security providers in Kafka Security config Key: KAFKA-8669 URL: https://issues.apache.org/jira/browse/KAFKA-8669 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

Hi John Translated it wrong, I meant ‚might‘ instead of ‚may‘. If I find the time I‘ll take a look at the code how they could be added as metrics. Thanks for your input. Regards Patrik > Am 15.07.2019 um 18:03 schrieb John Roesler : > > Hey Patrik, > > Since KIP-471 is already accepted, and

[jira] [Created] (KAFKA-8668) Improve broker shutdown time

Zhanxiang (Patrick) Huang created KAFKA-8668: Summary: Improve broker shutdown time Key: KAFKA-8668 URL: https://issues.apache.org/jira/browse/KAFKA-8668 Project: Kafka Issue

[jira] [Created] (KAFKA-8667) Improve leadership transition time

Zhanxiang (Patrick) Huang created KAFKA-8667: Summary: Improve leadership transition time Key: KAFKA-8667 URL: https://issues.apache.org/jira/browse/KAFKA-8667 Project: Kafka

Re: JIRA and KIP contributor permissions

I added you as a contributor on JIRA and granted write access to the Kafka Wiki. You don't need any special access on Github. Anyone can open PRs. -Matthias On 7/12/19 11:46 AM, Tirtha Chatterjee wrote: > Hi developers, > > I would like to be a contributor to the Apache Kafka project. I would

Re: Request for permission to create a KIP

Done. On 7/12/19 2:17 PM, Sandeep Mopuri wrote: > Hi, I'm planning to create a KIP titled "Add keymanager and trustmanager > security providers in Kafka Security config". My wiki id is mprsai. Please > grant me permissions to create one. > signature.asc Description: OpenPGP digital signature

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hi John, I knew I was missing something. Yes, that makes sense now, I removed all `listSerde()` methods, and left empty constructors instead. As per `CommonClientConfigs` I looked at the class, it doesn’t have any properties related to serdes, and that bothers me a little. All properties like

[jira] [Created] (KAFKA-8666) Improve Documentation on usage of Materialized config object

Bill Bejeck created KAFKA-8666: -- Summary: Improve Documentation on usage of Materialized config object Key: KAFKA-8666 URL: https://issues.apache.org/jira/browse/KAFKA-8666 Project: Kafka

[jira] [Created] (KAFKA-8665) WorkerSourceTask race condition when rebalance occurs before task has started

Chris Cranford created KAFKA-8665: - Summary: WorkerSourceTask race condition when rebalance occurs before task has started Key: KAFKA-8665 URL: https://issues.apache.org/jira/browse/KAFKA-8665

Re: Behaviour of KStreamKTableJoinProcessor

Hi again, Ties, I think I spoke too soon and also misread your email. By any chance, are you doing a join of a KStream and a GlobalKTable? In this case, it would make perfect sense to do what you're doing, but unfortunately the current implementation doesn't support it. Your workaround would

Re: [DISCUSS] KIP-488: Clean up Sum,Count,Total Metrics

Hey Stanislav, Thanks for the feedback! I do think we should try to keep the scope of this KIP small. The idea you proposed is orthogonal, though, so if you wanted to write up a quick KIP concurrently, I'd be happy to review it. Regarding deprecation: yes, the metrics are in the public

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

Hey Patrik, Since KIP-471 is already accepted, and since this idea is not a trivial extension of the KIP, I think we'd need to do a new KIP. Some points to consider: these additions could not be made to the KeyValueStore interface, since they'd be only applicable to RocksDB-backed stores, but

Re: Behaviour of KStreamKTableJoinProcessor

Hi Ties, You're on the right track. You need to use `KTable.map` ahead of the join to select the new key. This will allow Streams to make sure the data is correctly partitioned to perform the join. Thanks, -John On Mon, Jul 15, 2019 at 10:07 AM Ven, Ties Jens van de wrote: > > I recently

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hi all, Regarding the placement, you might as well move the constants to `org.apache.kafka.clients.CommonClientConfigs`, so that the constants and the configs and the code are in the same module. Regarding the constructor... What Matthias said is correct: The serde, serializer, and deserializer

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

One problem though. Since WindowedSerde (Windowed(De)Serializer) are so similar, I’m trying to mimic the implementation of my ListSerde accordingly. I created couple constants under StreamsConfig: And trying to do similar construct: final String propertyName = isKey ?

Behaviour of KStreamKTableJoinProcessor

I recently started working with kafka streams and I noticed some odd behavior. I was using a KTable left join with a null key, and ofcourse this will not work, since it will join based on keys. But I also supplied a KeyValueMapper, which takes a property from the value and returns this as key,

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hi Matthias, Thank you for your input. I updated the KIP, made it a little more readable. I think the configuration parameters strategy is finalized then. Do you have any other questions/concerns regarding this KIP? Meanwhile I’ll start doing appropriate code changes, and commit them under my

Re: [DISCUSS] KIP-489 Kafka Consumer Record Latency Metric

To hopefully spark some discussion I've copied the motivation section from the KIP: Consumer lag is a useful metric to monitor how many records are queued to be processed. We can look at individual lag per partition or we may aggregate metrics. For example, we may want to monitor what the

Any documents on this

Hi, I got few questions. Can anyone please give me a feedback i)To upgrade to latest version of kafka from cluster to cloud. What should I adopt. How will it be helpful ii)How will you move the data from cluster into the cloud ?We are currently using oracle DB. Any help will be appreciated.

Re: [DISCUSS] KIP-471: Expose RocksDB Metrics in Kafka Streams

Hi Adding this here because I asked and may have found the answer: The memory consumption may not be exposed as RocksDB metrics, but they should be available as properties of the RocksDB instance itself. RocksDBStore could easily make this available by allowing access to db.getProperty. Available

[jira] [Created] (KAFKA-8664) non-JSON format messages when streaming data from Kafka to Mongo

Vu Le created KAFKA-8664: Summary: non-JSON format messages when streaming data from Kafka to Mongo Key: KAFKA-8664 URL: https://issues.apache.org/jira/browse/KAFKA-8664 Project: Kafka Issue Type: