Jenkins build is back to normal : kafka-trunk-jdk14 #67

See

[jira] [Created] (KAFKA-9973) __consumer_offsets record is invalid lead to log clean failed and __consumer_offsets grows too big

leibo created KAFKA-9973: Summary: __consumer_offsets record is invalid lead to log clean failed and __consumer_offsets grows too big Key: KAFKA-9973 URL: https://issues.apache.org/jira/browse/KAFKA-9973 Proj

[jira] [Resolved] (KAFKA-9290) Update IQ related JavaDocs

[ https://issues.apache.org/jira/browse/KAFKA-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-9290. Fix Version/s: 2.6.0 Resolution: Fixed > Update IQ related JavaDocs > --

[jira] [Created] (KAFKA-9972) Corrupted standby task could be committed

Boyang Chen created KAFKA-9972: -- Summary: Corrupted standby task could be committed Key: KAFKA-9972 URL: https://issues.apache.org/jira/browse/KAFKA-9972 Project: Kafka Issue Type: Bug

Jenkins build is back to normal : kafka-2.5-jdk8 #112

See

Build failed in Jenkins: kafka-trunk-jdk11 #1436

See Changes: [github] KAFKA-6145: Set HighAvailabilityTaskAssignor as default in -- [...truncated 3.08 MB...] org.apache.kafka.streams.TestTopicsTest > testTimestamp ST

Re: [Discuss] KIP-582 Add a "continue" option for Kafka Connect error handling

Hi Zihan, In my experience, direct analysis of a record isn't often necessary when diagnosing issues with conversion or transformations. I raised the idea of log messages because in most cases they've been completely sufficient, paired with the source code for the converter or the transformation,

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

Hi Aakash, Thanks for the KIP! Given the myriad of different error-handling mechanisms available in sink connectors today, I think it'd be a great improvement to add this kind of support to the framework and not only take some of the development burden off of connector writers but also some of the

Re: Please give me the permission to create KIPs

Done. On 5/6/20 3:06 AM, Eddi Em wrote: > My id: eddiejamsession > > Eddie > signature.asc Description: OpenPGP digital signature

Re: default auto.offset.reset value is reset to none

There is a ticket about it: https://issues.apache.org/jira/browse/KAFKA-7480 The current solution is to delete the local copy of the global state store manually to be able to restart. -Matthias On 5/6/20 6:14 AM, Alexander Sibiryakov wrote: > Hello, > > I'm facing an issue in one of our Kafka

Re: [DISCUSS] KIP-606: Add Metadata Context to MetricsReporter

Thanks, Xavier. This does seem very useful. A minor request would be to mention the new configs for Connect, Streams and clients, specifically that because they are optional they will not hinder upgrades, and because they are namespaced are unlikely to clash or conflict with other configs from ext

Jenkins build is back to normal : kafka-trunk-jdk8 #4512

See

Build failed in Jenkins: kafka-trunk-jdk14 #66

See Changes: [github] KAFKA-9667: Connect JSON serde strip trailing zeros (#8230) -- [...truncated 6.16 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldFee

Jenkins build is back to normal : kafka-trunk-jdk11 #1435

See

Fwd: Is committing offset required for Consumer

Add dev mailing list as well. Thanks for your help! -- Forwarded message - From: Boyuan Zhang Date: Thu, May 7, 2020 at 3:51 PM Subject: Is committing offset required for Consumer To: Hi team, I'm building an application which uses Kafka Consumer APIs to read messages from top

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Just to adds a bit more FYI here related to the last question from David: in KIP-595 while implementing the new requests we are also adding a "KafkaNetworkChannel" which is used for brokers to send vote / fetch records, so besides the discussion on listeners I think implementation wise we can also

[jira] [Created] (KAFKA-9971) Error Reporting in Sink Connectors

Aakash Shah created KAFKA-9971: -- Summary: Error Reporting in Sink Connectors Key: KAFKA-9971 URL: https://issues.apache.org/jira/browse/KAFKA-9971 Project: Kafka Issue Type: New Feature

Re: [DISCUSS] Kafka 3.0

Hey folks, Sorry for stating that the bridge release would not break any compatibility before, which is incorrect and confused many people. I think one way to think about the versioning is that: 0) In a 2.x version moving ahead we would deprecate the ZK-dependent tools such as --zookeeper flags

Re: [REVIEW REQUEST] KAFKA-3184: Add Checkpoint for In-memory State Store

Thanks Nikolay, We will take a look at the PR asap. Guozhang On Thu, May 7, 2020 at 4:15 AM Nikolay Izhikov wrote: > Hello, Kafka Team. > > I prepared a PR [1] for the KAFKA-3184 [2] > Can someone, please, do the review. > > > [1] https://github.com/apache/kafka/pull/8592 > [2] https://issues

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

Hello folks, I've also updated the KIP wiki page adding a section of the proposed metrics for this Raft protocol: https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum#KIP-595:ARaftProtocolfortheMetadataQuorum-Metrics Please let us know if you have

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

Thanks Jun. Further replies are in-lined. On Mon, May 4, 2020 at 11:58 AM Jun Rao wrote: > Hi, Guozhang, > > Thanks for the reply. A few more replies inlined below. > > On Sun, May 3, 2020 at 6:33 PM Guozhang Wang wrote: > > > Hello Jun, > > > > Thanks for your comments! I'm replying inline bel

Re: [DISCUSS] Kafka 3.0

On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > > In fact, we know that the bridge release will involve at least one > > incompatible change. We will need to drop support for the --zookeeper > > flags in the command-line tools. > > If the bridge release(s) and the subsequent post-ZK release

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

Cheng, Thanks for the KIP and the detailed proposal section. LGTM! On Thu, May 7, 2020 at 3:38 PM Cheng Tan wrote: > > I think more about the potential wider use cases, I modified the proposal to > target all the connection. Thanks. > > - Best, - Cheng Tan > > > On May 7, 2020, at 1:41 AM, Chen

Re: [DISCUSS] KIP-606: Add Metadata Context to MetricsReporter

This would be useful, thank you :) On Tue, May 5, 2020 at 4:58 PM Xavier Léauté wrote: > Hi Everyone, > > I've published a KIP to address some shortcoming of our current metrics > reporter interface. Would appreciate feedback. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-606%3A+Ad

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

I think more about the potential wider use cases, I modified the proposal to target all the connection. Thanks. - Best, - Cheng Tan > On May 7, 2020, at 1:41 AM, Cheng Tan wrote: > > Hi Colin, > > Sorry for the confusion. I’m proposing to implement timeout in the > NetworkClient.leastLoadedN

Build failed in Jenkins: kafka-trunk-jdk14 #65

See Changes: [github] MINOR: Log4j Improvements on Fetcher (#8629) -- [...truncated 3.08 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldGetSinkTopicNames[E

Build failed in Jenkins: kafka-trunk-jdk11 #1434

See Changes: [github] MINOR: Log4j Improvements on Fetcher (#8629) -- [...truncated 3.08 MB...] org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldNotAllo

Re: [VOTE] KIP-605 - Expand Connect Worker Internal Topic Settings

LGTM, +1 (non-binding) Thanks Randall! On Thu, May 7, 2020 at 11:01 AM Randall Hauch wrote: > I'd like to open the vote for KIP-605: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-605%3A+Expand+Connect+Worker+Internal+Topic+Settings > > This is relatively straightforward, and the dis

Re: [DISCUSS] Kafka 3.0

On Wed, May 6, 2020, at 21:40, Ryanne Dolan wrote: > > This will allow us to get an "alpha" version of the KIP-500 mode out > > early for people to experiment with > > I think this is a non-sequitur. It's not a requirement that KIP-500 be > merged to master and released in order for people to expe

Build failed in Jenkins: kafka-trunk-jdk8 #4511

See Changes: [github] MINOR: Log4j Improvements on Fetcher (#8629) -- [...truncated 3.06 MB...] org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnCorrectInM

[jira] [Created] (KAFKA-9970) Kafka 2.4.0 crashes under Windows when client is restarted

SledgeHammer created KAFKA-9970: --- Summary: Kafka 2.4.0 crashes under Windows when client is restarted Key: KAFKA-9970 URL: https://issues.apache.org/jira/browse/KAFKA-9970 Project: Kafka Issue

[jira] [Created] (KAFKA-9969) ConnectorClientConfigRequest is loaded in isolation and throws LinkageError

Greg Harris created KAFKA-9969: -- Summary: ConnectorClientConfigRequest is loaded in isolation and throws LinkageError Key: KAFKA-9969 URL: https://issues.apache.org/jira/browse/KAFKA-9969 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk14 #64

See Changes: [github] KAFKA-9865: Expose output topic names from TopologyTestDriver (#8483) -- [...truncated 6.16 MB...] org.apache.kafka.streams.internals.WindowStoreFaca

Build failed in Jenkins: kafka-trunk-jdk11 #1433

See Changes: [github] KAFKA-9865: Expose output topic names from TopologyTestDriver (#8483) -- [...truncated 6.16 MB...] org.apache.kafka.streams.TopologyTestDriverTest

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

Hi, Thanks for the KIP. I wonder whether this idea would be better implemented using a new method on the SinkTaskContext. public void putFailed(Collection records) Then the rest of KIP-298 could apply. Failed records could be put to the DLQ or logged, as appropriate. I think there's real valu

[DISCUSS] KIP-610: Error Reporting in Sink Connectors

Hello all, I've created a KIP to handle error reporting for records in sink connectors, specifically within the context of put(...): https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors I would appreciate any kind of feedback. Thanks, Aakash

[VOTE] KIP-605 - Expand Connect Worker Internal Topic Settings

I'd like to open the vote for KIP-605: https://cwiki.apache.org/confluence/display/KAFKA/KIP-605%3A+Expand+Connect+Worker+Internal+Topic+Settings This is relatively straightforward, and the discussion thread had just a few suggestions that have already been incorporated into the KIP. Best regards

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

Tom, good point, I've done exactly that -- hashing record keys -- but it's unclear to me what should happen when the hash key must be rotated. In my case the (external) solution involved rainbow tables, versioned keys, and custom materializers that were aware of older keys for each record. In part

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

Hi again, Of course I was overlooking at least one thing. Anyone who could guess the record keys could hash them and compare. To make it work the producer and consumer would need a shared secret to include in the hash computation. But the key management service could furnish them with this in addi

[jira] [Created] (KAFKA-9968) Newly subscribed topic not present in metadata request

Man Hin created KAFKA-9968: -- Summary: Newly subscribed topic not present in metadata request Key: KAFKA-9968 URL: https://issues.apache.org/jira/browse/KAFKA-9968 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

Hi Rayanne, You raise some good points there. Similarly, if the whole record is encrypted, it becomes impossible to do > joins, group bys etc, which just need the record key and maybe don't have > access to the encryption key. Maybe only record _values_ should be > encrypted, and maybe Kafka Stre

Build failed in Jenkins: kafka-trunk-jdk8 #4510

See Changes: [github] KAFKA-9865: Expose output topic names from TopologyTestDriver (#8483) -- [...truncated 3.06 MB...] org.apache.kafka.streams.TopologyTestDriverTest

Re: [DISCUSS] KIP-595: A Raft Protocol for the Metadata Quorum

Hi Jun, Thanks for the comments. Here are a few quick responses. > My question was on the auto generated broker id. Currently, the broker can choose to have its broker Id auto generated. The generation is done through ZK to guarantee uniqueness. Without ZK, it's not clear how the broker id is aut

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

Thanks Sönke, this is an area in which Kafka is really, really far behind. I've built secure systems around Kafka as laid out in the KIP. One issue that is not addressed in the KIP is re-encryption of records after a key rotation. When a key is compromised, it's important that any data encrypted u

LogManager.nextLogDir should consider total partition size not count

When running Kafka with multiple log directories kafka.log.LogManager.getOrCreateLog selects the first available log directory with the smallest number of topic partitions. Topic partitions can have different sizes and this policy easily leads to data imbalances between log directories (or dis

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

Hi  I have just spotted this. I would be a little -1 encrypting headers these are NOT safe to encrypt. The whole original reason for headers was for non-sensitive but transport or other meta information details, very akin to tcp headers, e.g. those also are not encrypted. These should remai

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

Small typo correction i meant headers at the end of this paragraph not keys (sorry long week already) corrected: " Second i would suggest we do not add additional section (again i would be a little -1 here) into the record specifically for this the whole point of headers being added, is add

Kafka Connector

Hi, This is Monisha. I am a Netsuite Developer. I want to connect Netsuite and Apache Kafka. It would be great if you provide me information about how to connect Kafka and Netsuite. Looking forward to your reply. -- thank you. Monisha Kodi Phone: +49 15124770514

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

Hi Sönke, Replies inline 1. The functionality in this first phase could indeed be achieved with > custom serializers, that would then need to wrap the actual serializer that > is to be used. However, looking forward I intend to add functionality that > allows configuration to be configured broker

Build failed in Jenkins: kafka-2.5-jdk8 #111

See Changes: [konstantine] KAFKA-9768: Fix handling of rest.advertised.listener config (#8360) -- [...truncated 5.90 MB...] org.apache.kafka.streams.TopologyTestDriverTest

Build failed in Jenkins: kafka-2.4-jdk8 #197

See Changes: [konstantine] KAFKA-9419: Fix possible integer overflow in CircularIterator (#7950) [konstantine] KAFKA-9768: Fix handling of rest.advertised.listener config (#8360) ---

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

Hi Colin, Could you clarify how this fits with KIP-506 which seems to deal with the same? Thanks & Regards Jakub On Fri, May 1, 2020 at 8:18 AM Colin McCabe wrote: > Hi all, > > I posted a KIP about adding a new SCRAM configuration API on the broker. > Check it out here if you get a chance: >

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

Hi Colin, Thanks for the KIP. A couple of comments below: 1) SCRAM password is never sent over the wire today, not by clients, not by tools. A salted-hashed version of it stored in ZooKeeper is sent over the wire to ZK and read by brokers from ZK. Another randomly-salted-hashed version is sent by

[REVIEW REQUEST] KAFKA-3184: Add Checkpoint for In-memory State Store

Hello, Kafka Team. I prepared a PR [1] for the KAFKA-3184 [2] Can someone, please, do the review. [1] https://github.com/apache/kafka/pull/8592 [2] https://issues.apache.org/jira/browse/KAFKA-3184

Re: [DISCUSS] KIP-601: Configurable socket connection timeout

Hi Colin, Sorry for the confusion. I’m proposing to implement timeout in the NetworkClient.leastLoadedNode() when iterating all the cached node. The alternative I can think is to implement the timeout in NetworkClient.poll() I’d prefer to implement in the leastLoadedNode(). Here’re the reasons

Build failed in Jenkins: kafka-trunk-jdk11 #1432

See Changes: [github] MINOR - Increase the number of Trogdor Histogram buckets to 1 -- [...truncated 3.08 MB...] org.apache.kafka.streams.TopologyTestDriverTest > s

Build failed in Jenkins: kafka-trunk-jdk14 #63

See Changes: [github] MINOR - Increase the number of Trogdor Histogram buckets to 1 -- [...truncated 3.08 MB...] org.apache.kafka.streams.TopologyTestDriverTest > sho