Re: [DISCUSS] KIP-82 - Add Record Headers

+1 on this slimmer version of our proposal I def think the Id space we can reduce from the proposed int32(4bytes) down to int16(2bytes) it saves on space and as headers we wouldn't expect the number of headers being used concurrently being that high. I would wonder if we should make the value

Re: [DISCUSS] KIP-82 - Add Record Headers

For me 5c and 5a are almost identical. The idea in the kip(5a) is that the core message just has a header length and then the header bytes, which are then in a pre agreed sub wire protocol as described. 5c instead of having a pre agreed wire format allows custom serialisation of a map of

Re: [DISCUSS] KIP-89: Allow sink connectors to decouple flush and offset commit

I don't have much to say here since I reviewed a draft of this already. But I did want to point out that although it expands the API for Connectors (which I think we should be conservative about), it has the really nice impact that it allows connectors that really care about high throughput to

Re: Use Android App as a “Producing client” for Kafka?

Artur, It is possible to do this, but the second approach you mentioned is much more common. Normally people don't want to expose their Kafka cluster publicly, so having an intermediary can be a good way to, e.g., add a layer where you can easily filter out bad traffic. You may be able to use

Re: Kafka Connect key.converter and value.converter properties for Avro encoding

On Mon, Nov 7, 2016 at 7:14 AM, wrote: > Hi Ewen, > > Sorry but I didn't understand much of that. > > I currently have an implementation of the Converter interface that uses > Avro's > BinaryEncoder/Decoder, SpecificDatumReader/Writer. > > The main mismatch I faced is that

[GitHub] kafka pull request #2111: throw exception when the connection is error.

GitHub user huyanping opened a pull request: https://github.com/apache/kafka/pull/2111 throw exception when the connection is error. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huyanping/kafka trunk Alternatively you can

[jira] [Commented] (KAFKA-4389) kafka-server.stop.sh not work

[ https://issues.apache.org/jira/browse/KAFKA-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646319#comment-15646319 ] huxi commented on KAFKA-4389: - Seems it is a duplicate of issue 4297.

[jira] [Created] (KAFKA-4389) kafka-server.stop.sh not work

JianwenSun created KAFKA-4389: - Summary: kafka-server.stop.sh not work Key: KAFKA-4389 URL: https://issues.apache.org/jira/browse/KAFKA-4389 Project: Kafka Issue Type: Bug Components:

[jira] [Commented] (KAFKA-4386) Producer Metrics Explanation

[ https://issues.apache.org/jira/browse/KAFKA-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646087#comment-15646087 ] huxi commented on KAFKA-4386: - In my understanding, these metrics are all request-level. Take request latency

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

Hi, Rajini, A couple of other questions on the KIP. 10. For the config values stored in ZK, are those keys (s, t, k, i, etc) stored under scram-sha-256 standard? 11. Could KIP-48 (delegation token) use this KIP to send delegation tokens? In KIP-48, the client sends a HMAC as the delegation

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

For the use case, one potential use case is for schema registration. For example, in Avro, a null value corresponds to a Null schema. So, if you want to be able to keep the schema id in a delete message, the value can't be null. We could get around this issue by specializing null value during

Re: [DISCUSS] KIP-84: Support SASL/SCRAM mechanisms

I think Magnus brought up the point of bumping up the version of SaslHandshakeRequest during the discussion of KIP-35. A broker may be capable of supporting a list of SASL mechanisms X and may be enabling a list of SASL mechanisms Y. Y will be a subset of X. Now, from a client's perspective, it

Re: [DISCUSS] KIP-82 - Add Record Headers

Nacho, Thanks for the summary. #5 is not a binary decision, right? 5a) headers could be "fully" native as proposed - meaning both clients and brokers would be able to list all keys. 5b) headers could be inside the existing value field. in this case, only clients would understand the container

[jira] [Commented] (KAFKA-4353) Add semantic types to Kafka Connect

[ https://issues.apache.org/jira/browse/KAFKA-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645840#comment-15645840 ] Ewen Cheslack-Postava commented on KAFKA-4353: -- [~rhauch] Some of these make sense to me,

Re: [DISCUSS] KIP 88: DescribeGroups Protocol Update

Hi Jason, Thanks for your feedback. Yes, the intent of the KIP is to make existing offsets of the group available even when there is no active consumers in the group consuming from one or more topic partitions. Your suggestion should also work. I'm not yet sure how to obtain group's all topic

Re: [DISCUSS] KIP-82 - Add Record Headers

​Hey Roger. The original design involved: 1- a header set per message (an array of key+values) 2- a message level API to set/get headers. 3- byte[] header-values 4- int header-keys 5- headers encoded at the protocol/core level 1- I think most (not all) people would agree that having metadata

Re: [DISCUSS] KAFKA-4345: Run decktape test for each pull request

On Mon, Nov 7, 2016 at 10:30 AM Raghav Kumar Gautam wrote: > Hi Ewen, > > Thanks for the feedback. Answers are inlined. > > On Sun, Nov 6, 2016 at 8:46 PM, Ewen Cheslack-Postava > wrote: > > > Yeah, I'm all for getting these to run more frequently and on

Re: [DISCUSS] KIP-82 - Add Record Headers

Radai, If the broker must parse headers, then I agree that the serialization probably should not be configurable. However, the if the broker sees metadata only as bytes and clients are the only components that serialize and deserialize the headers, then pluggability seems reasonable. Cheers,

Re: [DISCUSS] KIP-82 - Add Record Headers

Magnus, Thanks for jumping in. Do you see a reason that the broker should understand the header structure you've proposed? I'm wondering if metadata should just be bytes from the broker's point of view but clients could implement a common header serde spec on top. Cheers, Roger On Mon, Nov 7,

Re: [DISCUSS] KIP-82 - Add Record Headers

Please see comments inline. On Mon, Nov 7, 2016 at 9:32 AM, Michael Pearce wrote: > Hi Roger, > > Thanks for the support. > Thanks for leading the discussion on this. It's an important topic. > > I think the key thing is to have a common key space to make an

Re: [VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

Hey Radai, Looking at the proposal, it looks like a major question is still unresolved? "This configuration parameter can either replace queued.max.requests completely, or co-exist with it (by way of either-or or respecting both bounds and not picking up new requests when either is hit)." On

[jira] [Assigned] (KAFKA-3910) Cyclic schema support in ConnectSchema and SchemaBuilder

[ https://issues.apache.org/jira/browse/KAFKA-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan reassigned KAFKA-3910: -- Assignee: Shikhar Bhushan (was: Ewen Cheslack-Postava) > Cyclic schema support in

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi, I'm +1 for adding generic message headers, but I do share the concerns previously aired on this thread and during the KIP meeting. So let me propose a slimmer alternative that does not require any sort of global header registry, does not affect broker performance or operations, and adds as

[jira] [Commented] (KAFKA-4362) Consumer can fail after reassignment of the offsets topic partition

[ https://issues.apache.org/jira/browse/KAFKA-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645583#comment-15645583 ] Joel Koshy commented on KAFKA-4362: --- In this specific issue, the coordinator is available, but has moved

[VOTE] KIP-72 - Allow putting a bound on memory consumed by Incoming requests

Hi, I would like to initiate a vote on KIP-72: https://cwiki.apache.org/confluence/display/KAFKA/KIP-72%3A+Allow+putting+a+bound+on+memory+consumed+by+Incoming+requests The kip allows specifying a limit on the amount of memory allocated for reading incoming requests into. This is useful for

[jira] [Commented] (KAFKA-4362) Consumer can fail after reassignment of the offsets topic partition

[ https://issues.apache.org/jira/browse/KAFKA-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645427#comment-15645427 ] Joel Koshy commented on KAFKA-4362: --- Sorry - missed this comment. It does not recover because the

[jira] [Commented] (KAFKA-4381) Add per partition lag metric to KafkaConsumer.

[ https://issues.apache.org/jira/browse/KAFKA-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645419#comment-15645419 ] Joel Koshy commented on KAFKA-4381: --- This is up to you but I definitely agree with Jason that it's

Build failed in Jenkins: kafka-trunk-jdk8 #1025

See Changes: [jason] MINOR: Fix regex on connector path param in ConnectorsResource -- [...truncated 14184 lines...] org.apache.kafka.streams.kstream.JoinWindowsTest > validWindows PASSED

[jira] [Commented] (KAFKA-4322) StateRestoreCallback begin and end indication

[ https://issues.apache.org/jira/browse/KAFKA-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645243#comment-15645243 ] ASF GitHub Bot commented on KAFKA-4322: --- Github user markcshelton closed the pull request at:

[jira] [Commented] (KAFKA-4322) StateRestoreCallback begin and end indication

[ https://issues.apache.org/jira/browse/KAFKA-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645244#comment-15645244 ] ASF GitHub Bot commented on KAFKA-4322: --- GitHub user markcshelton reopened a pull request:

[GitHub] kafka pull request #2105: KAFKA-4322 StateRestoreCallback begin and end indi...

Github user markcshelton closed the pull request at: https://github.com/apache/kafka/pull/2105 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #2105: KAFKA-4322 StateRestoreCallback begin and end indi...

GitHub user markcshelton reopened a pull request: https://github.com/apache/kafka/pull/2105 KAFKA-4322 StateRestoreCallback begin and end indication This adds a begin and end callback to StateRestoreCallback. The contribution is my original work and I license the work to

Jenkins build is back to normal : kafka-trunk-jdk7 #1675

See

[jira] [Updated] (KAFKA-4387) KafkaConsumer will enter an infinite loop if the polling thread is interrupted, and either commitSync or committed is called

[ https://issues.apache.org/jira/browse/KAFKA-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated KAFKA-4387: -- Status: Patch Available (was: Open) > KafkaConsumer will enter an infinite loop if the

[jira] [Commented] (KAFKA-4387) KafkaConsumer will enter an infinite loop if the polling thread is interrupted, and either commitSync or committed is called

[ https://issues.apache.org/jira/browse/KAFKA-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645113#comment-15645113 ] ASF GitHub Bot commented on KAFKA-4387: --- GitHub user srdo opened a pull request:

[GitHub] kafka pull request #2110: KAFKA-4387: Fix KafkaConsumer not responding corre...

GitHub user srdo opened a pull request: https://github.com/apache/kafka/pull/2110 KAFKA-4387: Fix KafkaConsumer not responding correctly to interrupts,… ... throw InterruptException from blocking methods when interrupted See

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Michael, Yes, changing the logic in the log cleaner makes sense. There could be some other thing worth thinking (e.g. the message size change after conversion), though. The scenario I was thinking is the following: Imagine a distributed caching system built on top of Kafka. A user is

[jira] [Commented] (KAFKA-4381) Add per partition lag metric to KafkaConsumer.

[ https://issues.apache.org/jira/browse/KAFKA-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644978#comment-15644978 ] Jiangjie Qin commented on KAFKA-4381: - Thanks Jason. I was not sure either. That said, we used to

Re: [DISCUSS] KAFKA-4345: Run decktape test for each pull request

Hi Ewen, Thanks for the feedback. Answers are inlined. On Sun, Nov 6, 2016 at 8:46 PM, Ewen Cheslack-Postava wrote: > Yeah, I'm all for getting these to run more frequently and on lighter > weight infrastructure. (By the way, I also saw the use of docker; I'd > really like

[jira] [Created] (KAFKA-4388) Connect key and value converters are listed without default values

Ewen Cheslack-Postava created KAFKA-4388: Summary: Connect key and value converters are listed without default values Key: KAFKA-4388 URL: https://issues.apache.org/jira/browse/KAFKA-4388

[jira] [Updated] (KAFKA-3994) Deadlock between consumer heartbeat expiration and offset commit.

[ https://issues.apache.org/jira/browse/KAFKA-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-3994: Priority: Critical (was: Major) > Deadlock between consumer heartbeat expiration and offset

[jira] [Commented] (KAFKA-4362) Consumer can fail after reassignment of the offsets topic partition

[ https://issues.apache.org/jira/browse/KAFKA-4362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644945#comment-15644945 ] Mayuresh Gharat commented on KAFKA-4362: [~jasong35] form the details that Joel has listed, I

[jira] [Commented] (KAFKA-4387) KafkaConsumer will enter an infinite loop if the polling thread is interrupted, and either commitSync or committed is called

[ https://issues.apache.org/jira/browse/KAFKA-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644936#comment-15644936 ] Stig Rohde Døssing commented on KAFKA-4387: --- Fine by me. I'll make a PR for it shortly >

[jira] [Commented] (KAFKA-4387) KafkaConsumer will enter an infinite loop if the polling thread is interrupted, and either commitSync or committed is called

[ https://issues.apache.org/jira/browse/KAFKA-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644913#comment-15644913 ] Jason Gustafson commented on KAFKA-4387: I think it's reasonable to have KafkaConsumer handle

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

Hi Becket, We were thinking more about having the logic that’s in the method shouldRetainMessage configurable via http://kafka.apache.org/documentation.html#brokerconfigs at a broker/topic level. And then scrap auto converting the message, and allow organisations to manage the rollout of

[jira] [Commented] (KAFKA-4381) Add per partition lag metric to KafkaConsumer.

[ https://issues.apache.org/jira/browse/KAFKA-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644874#comment-15644874 ] Jason Gustafson commented on KAFKA-4381: I'm not sure. Have we done KIPs for individual metrics

[jira] [Created] (KAFKA-4387) KafkaConsumer will enter an infinite loop if the polling thread is interrupted, and either commitSync or committed is called

Stig Rohde Døssing created KAFKA-4387: - Summary: KafkaConsumer will enter an infinite loop if the polling thread is interrupted, and either commitSync or committed is called Key: KAFKA-4387 URL:

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi Roger, Thanks for the support. I think the key thing is to have a common key space to make an ecosystem, there does have to be some level of contract for people to play nicely. Having map or as per current proposed in kip of having a numerical key space of map

Re: [DISCUSS] KIP 88: DescribeGroups Protocol Update

Hey Vahid, Thanks for the KIP. If I understand correctly, the problem is how to fetch existing offsets for a group which has no active members, right? I'm not totally clear why we need to modify the DescribeGroups API in order to achieve this since we already have the OffsetFetch API. I think the

[jira] [Updated] (KAFKA-4383) Update API design subsection to reflect the current implementation of Producer/Consumer

[ https://issues.apache.org/jira/browse/KAFKA-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-4383: - Fix Version/s: (was: 0.10.0.1) (was: 0.10.0.0) >

[jira] [Updated] (KAFKA-4382) Fix broken fragments in site docs

[ https://issues.apache.org/jira/browse/KAFKA-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-4382: - Fix Version/s: (was: 0.10.0.1) > Fix broken fragments in site docs >

[GitHub] kafka-site issue #29: Update the website repo link in code.html to point to ...

Github user becketqin commented on the issue: https://github.com/apache/kafka-site/pull/29 @ijuma That's fair. I updated the page. The thing I found confusing is that we have documentation in three places: 1) Kafka code repo 2) kafka-site repo 3) some documentation are just a wiki

[GitHub] kafka pull request #2108: MINOR: Fix regex on connector path param in Connec...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/2108 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

replica fetch error and shuabing

Hi, when I move __consumer_offsets from old broker to new broker, we encounter error as follow and it always shuabing. server.log.2016-11-07-19:[2016-11-07 19:17:15,392] ERROR Found invalid messages during fetch for partition [__consumer_offsets,10] offset 13973569 error Message found with

[GitHub] kafka pull request #2109: MINOR: fix incorrect logging in StreamThread

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/2109 MINOR: fix incorrect logging in StreamThread Fix incorrect logging when unable to create an active task. The output was: Failed to create an active task %s: It should have the taskId. You can

RE: Kafka Connect key.converter and value.converter properties for Avro encoding

Hi Ewen, Sorry but I didn't understand much of that. I currently have an implementation of the Converter interface that uses Avro's BinaryEncoder/Decoder, SpecificDatumReader/Writer. The main mismatch I faced is that I need to use org.apache.avro.Schema for serialization whereas the Converter

[jira] [Updated] (KAFKA-4384) ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

[ https://issues.apache.org/jira/browse/KAFKA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-4384: --- Fix Version/s: 0.10.1.1 > ReplicaFetcherThread stopped after ReplicaFetcherThread received a

[GitHub] kafka-site issue #29: Update the website repo link in code.html to point to ...

Github user ijuma commented on the issue: https://github.com/apache/kafka-site/pull/29 @becketqin, it's just that we point to the Apache git repo (which is the source repository) a few lines above for the code repo. We should probably be consistent and mention GitHub for both if we

[GitHub] kafka-site issue #26: add trademark symbol on all pages plus longer footer m...

Github user ijuma commented on the issue: https://github.com/apache/kafka-site/pull/26 @derrickdoo, can you please close this PR since it has been merged? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

Congratulations, Becket! Best wishes, Michael On Thu, Nov 3, 2016 at 5:13 PM, Efe Gencer wrote: > Congratulations, Becket! > > Best, > Efe > > 2016-11-03 11:22 GMT-04:00 Neha Narkhede : > > > Congratulations, Becket! Well done. > > > > On Wed, Nov 2,

[jira] [Updated] (KAFKA-4386) Producer Metrics Explanation

[ https://issues.apache.org/jira/browse/KAFKA-4386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pratik kumar updated KAFKA-4386: Description: Context : Kafka Producer 0.8.x Problem: Kafka Producer emits metrics regarding

[jira] [Created] (KAFKA-4386) Producer Metrics Explanation

Pratik kumar created KAFKA-4386: --- Summary: Producer Metrics Explanation Key: KAFKA-4386 URL: https://issues.apache.org/jira/browse/KAFKA-4386 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-4384) ReplicaFetcherThread stopped after ReplicaFetcherThread received a corrupted message

[ https://issues.apache.org/jira/browse/KAFKA-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15643648#comment-15643648 ] Jun He commented on KAFKA-4384: --- [~becket_qin] Thanks for reviewing this issue. Yes I will provide the patch